Cost calculator for on-premises AI
Many companies are evaluating the use of on-premises AI language models to avoid sharing data with cloud providers. On-premises AI makes this possible: Open-Weight models run entirely on your own hardware, in your own data center, or with a hosting partner you trust.
But what does it actually cost? Describe your use case and receive a specific hardware recommendation – including upfront costs and total monthly expenses – in just two minutes. It’s free and requires no registration.
Local AI: Hardware and operations
Longer contexts require significantly more VRAM.
Stronger quantization saves VRAM but costs response quality.
Only effective with MoE models (frontier tiers). Significantly reduces VRAM requirements but slows responses considerably. Suited for experiments and single users, not for production with many requests.
Proof of concept: software
One-time implementation effort until the use case runs verifiably in your environment. Pick the building blocks you need.
e.g. file server / NAS / file system, wiki, SharePoint, websites. Incl. chunking, embedding pipeline, and quality tuning per source.
Connection to ERP, ticketing systems, email, or other systems of record, including error handling.
Sign-in via your existing directory service, e.g. Entra ID or LDAP.
Permissions are inherited from the respective source (file system permissions, SharePoint permissions, and so on). Scales with the number of data sources.
Test dataset, quality metrics, and structured feedback from pilot users. What separates a solid PoC from a mere installation.
A custom interface instead of the standard frontend. For the PoC, the standard frontend is usually sufficient.
GDPR documentation and EU AI Act classification for pilot operation.
How the calculator works
We work with five quality levels for open-weight models, with 4-bit quantization by default (customizable in expert mode).
- The Compact level refers to models with 7 to 8 billion parameters, such as Qwen3-8B or Llama 3.1 8B, which are sufficient for classification, extraction, and simple text tasks.
- Solid corresponds to the 30B class, such as Qwen3-32B or Gemma 3 27B, for code assistance and more demanding text tasks.
- Large covers dense models with 70 billion or more parameters, such as Llama 3.3 70B, representing the highest quality level of classical model architectures.
- Above these are two Frontier tiers featuring a mixture-of-experts architecture, in which only a portion of the model actively computes per query. “Frontier Compact” refers to models with 200 to 300 billion parameters, such as Qwen3-235B or DeepSeek V4 Flash, which come remarkably close to the performance of the largest models while being significantly more cost-effective to operate. Frontier encompasses the largest freely available models with over 700 billion parameters, which match the performance of commercial cloud models in many tasks.
In addition to the model weights, we factor in the memory requirements for the KV cache. This is the area of the graphics memory where the model stores the conversation history and documents it has read. It grows with the context length and the number of concurrent requests and is often underestimated in practice.
The recommendation selects the smallest configuration whose graphics memory is sufficient for the model, KV cache, and operational reserve. The range starts with a single RTX 4090 or RTX 5090 in a workstation (24 or 32 GB of graphics memory), continues with the RTX 6000 Ada (48 GB) in a server, and extends to systems with one, two, four, or eight RTX PRO 6000 Blackwell cards, each with 96 GB. This covers everything from a workstation costing around 7,000 euros (including setup) to a GPU server with 768 GB of graphics memory, on which even the largest open-weight models run entirely locally. If even that isn’t enough, you’ll be in the realm of custom-designed clusters, for which we do not quote flat rates.
For the Frontier tiers, Expert Mode also offers “Expert Offloading”: This offloads part of the model to RAM, allowing for smaller graphics cards but significantly slowing down responses. It’s an option for experiments and individual users, but we do not recommend it for production use.
When training language models, the GPU is almost always the bottleneck. We therefore size the CPU and RAM according to a rule of thumb: RAM must be at least twice the amount of graphics memory so that models can be buffered while loading and switching – ranging from 64 GB in workstations to 1.5 TB in the largest GPU server. A modern multi-core CPU with at least 16 cores and sufficient PCIe lanes for the graphics cards is sufficient. These components are included in the system prices. Important to note: ECC RAM, in particular, has become significantly more expensive due to the current memory shortage. Our system prices therefore include a corresponding surcharge and are updated regularly.
The software is free: Open-weight models and inference servers such as vLLM or Ollama are open source, and there are no license fees. The purchase price includes a one-time setup fee for installing, configuring, and securing the system. The monthly operating fee covers updates, monitoring, and maintenance. In addition, there are electricity costs, which we calculate based on the actual workload of your scenario: The system’s base load runs around the clock, while GPU load occurs only when requests are being processed. To determine the total monthly cost, we amortize the purchase price over 36 months.
Not included is the actual project work: integrating the system with your systems and processes, applying RAG pipelines to your documents, or evaluating which model best solves your specific task. That is the part we can develop together with you.
Honest answer: not always. If you only have a few requests per day and your data is allowed to leave the cloud, an API is usually the more cost-effective option. On-premises AI really shines when one of these conditions applies: your data cannot leave the premises, you need to operate without an internet connection, or usage is so intensive that usage-based API costs exceed the cost of your own hardware. This happens faster than many people realize, especially with agent-based applications and high-volume embedded processes. On top of that, you get predictable costs instead of variable monthly bills, and you’re not at the mercy of price changes or model discontinuations by providers.
Case Studies and Use Cases
FAQ on Local AI and AI Costs
Not for the workstation configurations. They’re located in the office, run off a standard outlet, and are quiet during normal use, though they’re audible under full load. The server configurations (2 to 4 height units) belong in a server cabinet with cooling, which most companies have anyway. Power connections and cooling only become planning considerations for large multi-GPU systems; we’ll clarify that before placing the order.
Honest answer: The largest cloud models are still ahead at the top. For most specific business tasks—such as classification, extraction, document search, or code assistance—the difference in day-to-day use is minor or imperceptible, provided the model is a good fit for the task. The largest open-weight models match the performance of commercial cloud models in many tasks. The key factor is the fit between the model and the task, and we evaluate this in the potential analysis using your data.
Then swap out the model, not the hardware. Open-Weight models are interchangeable; switching to a new one involves just a download and a test run—no need to buy new hardware. Technology is working in your favor: The quality per gigabyte of graphics memory has been increasing for years, so the same machine can run increasingly better models over its lifetime.
Yes, and that’s the key to cost-effectiveness. A server can host multiple models or use a single model for different purposes—for example, as an assistant for employees during the day and for batch processing of documents at night. Utilization increases, while the cost per use case decreases. Configuring the server is therefore often just the beginning—not the end—of its use.
The calculator estimates the purchase price and spreads the cost over 36 months. Leasing through hardware partners is an option and helps preserve cash flow, but it has little impact on the total cost. Renting dedicated GPUs from European data centers is a middle ground for cases where data may not be transferred to U.S. providers but can be stored in an external data center. We’ll discuss which option best suits your needs on a case-by-case basis.
Unfortunately, there’s no such thing as automatic GDPR compliance. But the biggest structural hurdles are eliminated: no transfers to third countries, no U.S. data processors—your data never leaves the premises. The due diligence requirements you’re familiar with from other internal systems—such as access policies and deletion rules—apply here as well. For your data protection officer, this is a much simpler starting point than any cloud connection.