DEFINITION
Large or small language model?
The choice that decides both cost and sovereignty.
Plenty has been written about where you run AI and how you add knowledge to it. Which kind of model you choose is rarely discussed, yet that is where cost, speed and sovereignty all meet.
The choice between a large and a small language model comes down to four things: what it costs per request, how fast it answers, how good it is on hard questions, and whether it runs inside your own environment. Large commercial models are strongest on rare, difficult work; small models you run yourself are cheaper and faster on bounded work at volume, and keep data inside your walls. In practice you do not choose once for the whole organisation but per task. W69 AI Consultancy in Amstelveen sets up that choice and the router in front of it.
These figures are not research but the structure of the choice itself: four trade-offs (cost, speed, quality, sovereignty), two places to run, and the fact that a self-hosted model sends no requests outside.
The question that falls between the cracks
Plenty has been written about where you run AI and about how you add knowledge to it. Which kind of model you choose, and why, is rarely discussed. Yet that is where cost, speed and sovereignty all meet.
One model for everything is expensive
Most organisations send every request to the same large model, from summarising an email to a difficult policy judgement. You then pay for the heaviest reasoning while nine out of ten questions do not need it.
Bigger is not always better
For bounded, repeated work (classifying, extracting, summarising to a fixed pattern) a small model often performs just as well, but faster and at a fraction of the cost. The difference shows up on hard, rare questions.
Sovereignty does not stop at the data
You can keep your data neatly inside Europe and still send every question to a model outside your control. Data sovereignty without model sovereignty is half a story, and precisely the point a buyer will press on.
Large, small, or both
Four tasks, one router, two models. The choice belongs at task level, not made once for the whole organisation.
How to make the choice
Not on instinct and not on somebody else’s benchmark, but on your own tasks.
Sort your tasks by difficulty
List the work you want AI to do and how often it occurs. Routine work at volume is the candidate for a small model. Rare, hard work is not. This sorting does more for your costs than any negotiation.
Establish what may not leave
Which data may not leave your environment, legally or contractually. That is not a technical question but a governance one, and the answer determines which tasks have to run on your own model regardless.
Measure with your own tasks
Record a fixed set of realistic tasks with the answer you expect and run it across both kinds of model. A public leaderboard says nothing about your work. Your own set does, and you need it anyway to know whether things still work.
Put a router in front
Let an intermediate layer decide per task which model handles it. Then you do not have to choose between cheap and good, and you can switch models without your applications noticing.
What people ask about this
A language model with considerably fewer parameters than the large commercial ones, which lets it run on modest hardware, including your own server. Its weights are often public, so you can download, adapt and run it without a single request leaving your network. It is not a stripped-down version of a large model but a different trade-off: less breadth of knowledge, far lower cost per request.
Look at how often a task occurs and how hard it is. Bounded work that recurs thousands of times a day (classifying, extracting, summarising to a fixed pattern) is the candidate for a small model. Rare, difficult questions where a mistake is expensive belong with a large model. In practice you need both, with a router in front.
That depends what you mean. On confidentiality a model you run yourself is stronger, because nothing leaves your environment. On behaviour a small model is more vulnerable: it usually has less built-in resistance to misleading input. So you move the risk rather than remove it. Both need boundaries and oversight.
Unlike paying per request, the costs sit up front and are fixed: hardware or rented compute, plus someone to keep it running. That pays off at volume and not for occasional use. The rule of thumb: with few requests, paying per use is cheaper; with many similar requests, that tips over.
The moment you substantially modify a model or release it under your own name, the AI Act can make you a provider yourself, with documentation and information duties attached. With self-hosted and adapted models that is a real question. See the article on GPAI for the distinction between the two roles.
Are you sending everything to the same model?
The AI Navigator™ maps which work you hand to AI, how often it occurs and what that means for your costs and your sovereignty.