The most consequential number in enterprise AI right now is not a benchmark score. It is a price.
According to Stanford’s 2025 AI Index, the cost of querying a model that performs at GPT-3.5 level fell from $20.00 per million tokens in November 2022 to $0.07 by October 2024, a more than 280-fold collapse in roughly eighteen months.
Most executives miss the detail behind that figure. The model that hit the $0.07 price point was not a scaled-down offering from a frontier lab. It was an eight-billion-parameter small model.
The cheapest way to buy yesterday’s frontier intelligence is not to rent the frontier. It is to buy something small that learned from it.
That is the case for distillation in one sentence. It also explains the broader shift toward small, task-specific models, which Gartner predicts enterprises will use at three times the volume of general-purpose large language models by 2027.
An Old Idea Whose Economics Finally Arrived
Distillation is not a fad, it is a decade-old research lineage.
In 2015, Geoffrey Hinton, Oriol Vinyals, and Jeff Dean formalized the “teacher-student” method: train a large, cumbersome model, then transfer its knowledge, including the subtle probability judgments it makes between answers, into a much smaller model that is cheap to deploy.
The canonical proof arrived in 2019 with DistilBERT, which retained 97% of BERT’s language-understanding performance with 40 percent fewer parameters while running 60% faster.
That is not a compromise. For most enterprise workloads, such as classification, extraction, summarization, and routing, it is the correct engineering answer.
What changed recently is that the students got remarkably good.
Microsoft’s Phi-3-mini, at 3.8 billion parameters, scores 69% on MMLU, rivals models ten times its size, and runs on a phone. Meta’s Llama 3 8B posts near-parity with GPT-3.5 Turbo on the same benchmark, as an open-weight model you can host yourself. The definitive at-scale proof already ships in millions of pockets. Apple’s on-device foundation model, at roughly three billion parameters, powers Apple Intelligence across iOS and is engineered with aggressive compression to run locally on consumer hardware.
The question of whether small models can do real work has been answered in production.
The pricing follows the physics. When OpenAI launched GPT-4o-mini in July 2024, it listed at $0.15 per million input tokens against $2.50 for full GPT-4o, roughly sixteen times cheaper at launch, while scoring 82% on MMLU. Those models have since been superseded by newer families, but the pattern persists. Every major provider maintains the same tiered structure, because every provider knows most enterprise queries never needed the flagship.
The Honest Counterargument
A credible strategy piece must confront an uncomfortable fact: enterprises are not yet voting with their wallets for small models.
A mid-2025 survey of 150 technical decision-makers by Menlo Ventures, a venture capital firm and an investor in AI companies including Anthropic, found that closed-source frontier models still power 87% of enterprise workloads. Open-source share actually fell as spend doubled to $8.4 billion in six months.
There is no contradiction here, but there is a distinction worth being precise about. Gartner’s prediction concerns usage volume, not spend, and it describes where workloads go once they mature.
The current pattern is rational sequencing. Enterprises prototype on frontier models because they are the fastest way to discover whether a use case works at all.
The strategic error is not starting on the frontier. The error is staying there, paying flagship prices indefinitely for tasks a distilled model handles at a fortieth of the cost.
The frontier is where you learn. Small is where you run.
Control Is the Second Dividend
Cost gets the headlines, but for regulated enterprises the deeper argument is control.
A small model you host yourself keeps your data inside your perimeter and satisfies data residency obligations that no API contract can fully guarantee. It also benefits from an explicit carve-out in the EU AI Act: Article 53(2) exempts providers of free and open-source models with publicly available weights from the Act’s core technical documentation obligations under Article 53(1)(a) and (b), though the exemption does not extend to models designated as posing systemic risk.
Regulation, unusually, is blowing in the direction of the open, self-hosted small model.
The market has already demonstrated how disruptive this combination can be.
In January 2025, DeepSeek released its R1 reasoning model: open weight, MIT licensed, and priced roughly 96% below OpenAI’s comparable o1. Nvidia lost $589 billion in market capitalization in a single trading day, the largest loss in U.S. stock market history.
Two caveats belong in any board conversation about that episode.
First, DeepSeek’s widely quoted $5.6 million training cost refers, by the company’s own technical report, to the rental equivalent of the final training run only, excluding all prior research, experiments, and hardware.
Second, OpenAI’s suspicion that DeepSeek distilled its outputs remains an allegation rather than an established fact.
The structural lesson stands regardless. Capability now flows downhill from large models to small ones faster than pricing power can hold.
That lesson cuts both ways, and every CIO must draw the legal line clearly.
OpenAI, Anthropic, and Google all prohibit using their outputs to train competing models. Distilling from a proprietary API is a contract violation. Distilling from open-weight teachers whose licenses expressly allow it is permitted. Meta’s Llama 3.1 and later allow it, though the original Llama 3 license prohibited using outputs to improve other models, and DeepSeek’s MIT-licensed R1 allows it as well, subject in Meta’s case to the requirement that derivative models carry the Llama name.
There is a legally safe path to this strategy. Take that one.
What Small Models Cannot Do
Candor also requires naming the limits. Benchmark parity is not task parity. MMLU scores are self-reported by model developers and increasingly saturated, and peer-reviewed work shows that distilled models degrade more sharply than their teachers on out-of-distribution inputs, the messy, shifting, real-world data that defines enterprise environments.
A distilled model is a specialist. Point it at the task it was trained for and it excels. Drag it outside that distribution and the 97% can erode quickly.
This is why the answer is never to replace the frontier model with a small one. The answer is a portfolio.
The Prescription: Right-Size by Design
Gartner’s own guidance points to a “composite approach” of multiple models and workflow steps, routed by task.
The operating model that follows is straightforward.
Route the bulk of high-volume, well-defined work, such as classification, extraction, internal search, and first-draft generation, to small or distilled models, self-hosted where data sensitivity demands it.
Reserve frontier models for what genuinely needs them: novel reasoning, low-volume but high-stakes analysis, and the discovery phase of new use cases.
Measure the routing ratio quarterly. If 90% of your token spend still flows to flagship models a year into production, you are subsidizing convenience rather than capability.
Build the exit before you need it. A distilled model you own is the most reversible asset in your AI portfolio. No vendor deprecation calendar, repricing letter, or terms of service change can take it from you. In an era when AI commitments accumulate obligations faster than returns, the small model you control may be the only line on the AI ledger that behaves the way an asset should.
The era of reflexively reaching for the largest model is ending, not because big models failed, but because their best capabilities keep escaping into smaller, cheaper vessels.
The enterprises that win the next phase will not be the ones with the biggest models. They will be the ones that learned to stop asking which model is best and start asking a harder question: what is enough?
