Calculating AI Costs: A Comparison of Cloud and On-Premises AI

Token prices keep falling, yet corporate AI bills keep rising. This contradiction is prompting decision-makers to take a closer look at their own AI costs. That raises a question: use AI from a cloud provider, or build on-premises infrastructure? This article shows what the total cost of ownership (TCO) looks like on both sides and names the key questions to consider when making this decision.

Three trends are converging.

1. Consumption is growing faster than prices are falling.
   Modern systems with multi-stage reasoning, tool usage, and agents multiply the number of tokens per interaction. Before generating a response, reasoning models create an internal chain of thought that typically runs five to twenty times times the length of the visible and is billed as normal output. The price per unit is falling, but the number of units is rising.
2. Flat-rate pricing is giving way to usage-based billing.
   Major cloud providers are switching their flat-rate plans to usage-based models. A plan then includes a monthly quota; anything above that is billed based on token usage. From that point on, the intensity of usage determines the bill total, and this is difficult to budget for. For example, token-hungry agentic tools spread faster than expected, leading to annual budgets that were sometimes exhausted in just a few months.
3. Providers make the decisions; companies have to go along with them.
   Prices for basic plans are rising, and AI features are being bundled in; for example, Copilot Chat is now part of the basic plan for Microsoft 365 suites. This means that even organizations that do not use the feature are paying the AI surcharge. Another development: models are being discontinued, sometimes with transition periods of just a few weeks, and processes that rely on them are migrating according to the provider’s schedule. AI is becoming an infrastructure issue with a time horizon of several years.

Cloud AI is billed in two ways, and both respond differently to growth. Per-user, per-month licensing is typically used for workforce assistance: text creation, research, or correspondence. Costs depend mainly on the number of users and the pricing tier, and hardly at all on the intensity of use.

Usage-based billing is charged per token, for example, in automated processes such as document processing, data extraction, or internal search. Here, costs depend mainly on volume, document size, and model selection, and hardly at all on the number of employees. Most companies pay for both, often adding usage-based charges on top of the license fees. And underlying all of this is a third layer that appears in no vendor quote.

The Licensing Tier

The cost of the license depends on the per-user price, the pricing tier, and the environment. Licenses are billed monthly per person, usually on an annual commitment. typically with an annual commitment. Compliance requirements often force an upgrade from the Business tier to the Enterprise tier, which can more than double the price for the same set of features. For add-ins within an office suite, their base license cost is added to the total.

The Usage Tier

With usage-based billing, the choice of model determines the cost. A reasoning model in full mode can easily cost ten to a hundred times as much as a standard model for the same task. There are two reasons for this: token prices are higher, and the model requires thinking tokens before it answers. Those tokens that are billed without being visible. In practice, the model is usually determined by a default setting in the code. In addition to the model selection, the number of tokens processed by the model determines the cost. Keep in mind: When transitioning from pilot operation to production, consumption multiplies.

Invisible Billing Items: The Layer Below the Waterline

None of these items show up above the waterline but play a crucial role.

• Implementation
  A rollout for 50 to 250 users takes four to eight weeks and requires a permissions audit and a pilot group.
• Training
  Article 4 of the EU AI Act has applied since 2025. Employees who use AI for work therefore need appropriate training.
• Compliance
  Conditions change during ongoing operations. In April 2026, Microsoft activated a feature that processes Copilot requests outside the EU during peak loads. Such changes create audit overhead that isn’t reflected on any invoice.
• Administration
  Added to this is the ongoing management of licenses, permissions, governance, and contract renewals.

With on-premises AI, costs are distributed differently. The initial setup is an investment; after that, costs are tied to operation rather than usage. The bill therefore consists of the following items.

Cost Item 1: Hardware

Local AI requires its own servers, and the number of servers determines the costs. The most common misconception here is equating the number of users with the number of servers. Out of a hundred employees, usually only a few dozen work with AI simultaneously during peak times, and a single server can handle many requests in parallel. The key factors, therefore, are the number of concurrent requests and the graphics card memory. Those without a server room can use AI hosting in Germany, where the hardware sits in a provider's data center.

Cost Item 2: Operations

Operating costs include electricity, cooling, emergency power, maintenance, and replacement parts. As a rule of thumb, maintenance and replacement parts amount to a low double-digit percentage of the hardware’s value per year. The open-source stack does not require a license. In return, the company itself handles what is otherwise included in the API price: updates, model maintenance, monitoring, and security.

Cost Item 3: Personnel

There is a one-time labor cost for the implementation project: sizing, installation, the chat interface, and integration with the company’s own documents. After that, ongoing operations are added, realistically, a fraction of an in-house position or a maintenance contract with a service provider. It’s rare to need a dedicated specialist just for AI; that’s a standard found in large corporations. For small and medium-sized businesses, a maintenance contract is usually sufficient.

Cost Item 4: Utilization

A server costs the same whether it’s running or idle. At 10 percent utilization, each request costs about ten times as much as at full load, because you’re paying for the unused capacity as well. A sustained utilization rate of 50 to 60 percent is therefore considered the minimum threshold for in-house operation. A server with low utilization quickly becomes expensive. That’s why load analysis should be the first step in any AI project.

Both options start with one-off costs: hardware and implementation on one side, rollout and training on the other. The difference lies in what follows. The cloud keeps billing as a subscription, while in-house operation drops back to depreciation, electricity, maintenance, and staff time.

How soon the gap opens depends on four factors: the utilization of your own hardware, the number of users, the required license tier, and the volume processed by the automated workflows.

Three patterns emerge consistently. Small groups using AI purely for assistance are usually cheaper off in the cloud. For a typical mid-sized company with mixed usage, both options land close together. Predictable sustained workloads, large volumes of documents, or requirements regarding data location are factors that favor on-premises operation or AI hosting in Germany. To help you make a decision, here are five questions to close with, so you can assess your situation.

The order is intentional. The first question can override all the others.

1. Can your data leave the premises?
   The exclusion question.Professional confidentiality rules, classified information, GDPR requirements, or strict customer requirements determine the location in advance; the rest is a matter of scaling. If the answer is “Yes” or “Partially,” we move on.
2. How many people will use AI, and what type of license do they need?
   Small groups with only chat needs and no special requirements favor the cloud. Starting at around 50 users, or as soon as an enterprise-level solution becomes necessary for compliance reasons, you enter the zone where it makes sense to run the numbers.
3. What does your load profile look like?
   Predictable sustained loads belong on your own hardware. Sporadic use with high peaks and long pauses remains the domain of the cloud. Individual use cases that absolutely require the most capable frontier model available should remain in the cloud, regardless of the rest.
4. What resources do you already have in-house?
   A server room, an IT team, or a dedicated service provider significantly lower the barrier to entry. If these are lacking,an operating model belongs in the calculation: managed hosting or basic AI hosting in Germany through a service provider.
5. Are we talking about an experiment or infrastructure?
   Anyone still figuring out what AI should do in their company starts in the cloud. Planning AI as infrastructure over three to five years calls for the same approach as any other infrastructure decision. One limitation remains: The more deeply processes are integrated into a cloud provider’s infrastructure, the more expensive a future migration will be.