Local AI Technology at makandra

We rely on Ollama as the runtime environment for our language models. It has become the de facto standard for getting started quickly with on-premises AI, letting us run open-source models locally without setting up complex infrastructure. Models can be swapped out at any time if a newer one suits the use case better – without locking us into a specific provider.

For scenarios with very high workloads and thousands of concurrent users, we are currently evaluating vLLM as an alternative. For most of our projects – internal AI features with manageable workload profiles – Ollama is the right choice.

We currently rely on models from Qwen, Mistral (Ministral), and Google (Gemma). Depending on the task, we use different types of models: vision models for processing documents and images, embedding models for semantic search, and text models for generation and analysis – in a range of sizes, matched to performance requirements and available hardware.

Since the open-source model landscape is evolving rapidly, we continuously evaluate new releases. One practical advantage of on-premises AI is that the model in use can be swapped out at any time when a better one becomes available – without changing the rest of the application.

Our AI applications are built on open-source language models and run entirely on our own infrastructure – either on-site at the customer's premises or on dedicated servers in German data centers. Today's open-source models are powerful enough for the majority of enterprise applications.

Data never leaves the customer's own infrastructure, there is no dependency on external API providers, and costs stay predictable. When a better model becomes available, it can be deployed without changing the architecture – an advantage that carries real weight in a market moving at this pace.