Updated June 2026

Small models keep getting better, and it isn’t magic. A handful of techniques do most of the work, and knowing them at a plain-language level changes how you read model announcements and pick what to run.

Quantization: store the same brain in fewer bits

A model’s weights are billions of numbers, normally stored at 16-bit precision. Quantization stores them at lower precision (8-bit, 4-bit, sometimes less), shrinking memory and speeding up inference. The remarkable part is how little it costs: weights are redundant enough that 4-bit versions of a model typically perform within a hair of the original. Below 4-bit the damage gets noticeable; subtle reasoning degrades first, surface fluency last, which is exactly the failure mode that’s easy to miss without an eval.

Quantization changes nothing about what the model knows; it’s the same model, compressed after training. When you grab a model from Ollama or Hugging Face you’re choosing a quantization level whether you realize it or not (the Q4_K_M-style suffixes on GGUF files). Practical default: 4-bit is the sweet spot for running locally; take 8-bit when you have memory to spare and the task is reasoning-heavy.

Distillation: a small model trained by a big one

Distillation is a training technique: a large “teacher” model generates outputs (or richer signals, like its full probability distributions), and a smaller “student” model is trained to reproduce them. The student doesn’t just learn right answers; it learns the teacher’s behavior: how it phrases, reasons, and handles edge cases. This is the main reason modern small models punch so far above their parameter count. Many of the best small open models are openly distilled from larger siblings, and frontier labs’ own cheap tiers benefit from the same idea.

The part most people miss: a budget version of this is available to you. Use a frontier model to generate a few thousand high-quality labeled examples for your narrow task, then fine-tune a small model on them. For a well-bounded task (your domain’s classification, extraction, or routing) the small model often matches the frontier model that taught it, at a fraction of the cost per call. The recipe is: frontier model writes the curriculum, small model attends the class, your eval grades the final exam. Check your provider’s terms first; some restrict using outputs to train competing models.

The other two worth knowing

Mixture of experts (MoE). Instead of every parameter firing on every token, the model is built from many “expert” sub-networks and a router activates only a few per token. That’s how a model can advertise a huge total parameter count while running with the cost of a much smaller one. When a spec sheet says “200B parameters, 20B active,” the active number is the one that predicts speed and serving cost.

Pruning. Deleting weights or whole layers that contribute least, then briefly retraining to heal the damage. Less prominent than the others in practice, but part of the same toolbox.

Why you should care even if you never train anything

These techniques are why the model-selection advice in Managing Token Costs keeps getting more attractive: the cheap tier of any given year tends to match the frontier tier from one or two years before, largely on the back of distillation. They’re also why “how many parameters” stopped being a useful quality measure on its own; a distilled, well-trained 8B routinely beats a lazy 30B from an earlier generation. The only durable measure is the one you own: your eval, run against the actual candidates.