Updated June 2026

Everything else on this site assumes a model behind an API: Claude, GPT, Gemini, whichever. There’s a parallel world where the model runs on your own hardware, and it’s worth knowing when to live there.

Why run local

  • Data control. Nothing leaves your machine. For medical records, legal documents, proprietary code, or anything under a compliance regime that forbids third-party processing, local isn’t an optimization, it’s the requirement.
  • Cost model inversion. APIs charge per token; local charges per GPU. For high-volume, low-difficulty work (classification, extraction, embedding at scale), a model you already paid for is effectively free per call. The crossover math is worth doing before assuming either direction.
  • Offline and edge. No network, no rate limits, no provider outage taking your feature down.
  • Learning. Nothing demystifies LLMs like watching one run on your own machine, swapping quantizations, and feeling temperature settings change the output. The API hides everything this exposes.

Why not

Be honest about the trade. Open-weight models are genuinely good and improving fast, but the frontier API models are better at the hard stuff: long agentic runs, subtle code review, complex reasoning. You also become the ops team: serving, updating, monitoring, and capacity planning are now your problem. The right mental model is the cheap-model tier from Managing Token Costs taken one step further: local for the work it passes your eval on, API for the work it doesn’t.

The stack, by effort level

  • Zero effort: Ollama or LM Studio. Install, ollama run <model>, done. Both expose an OpenAI-compatible API on localhost, which means nearly any LLM library or tool can point at your local model by changing a base URL.
  • The layer underneath: llama.cpp. The engine most easy tools wrap. Go here when you want control over quantization, offloading, and performance details.
  • Serving for real: vLLM. When local stops meaning “my laptop” and starts meaning “our inference server,” vLLM-class servers handle batching and throughput properly.

Models and hardware, the honest version

The open-weight families to know: Llama (Meta), Mistral, Qwen (Alibaba), Gemma (Google), DeepSeek, and OpenAI’s gpt-oss. New versions ship constantly; check current leaderboards rather than trusting any note’s snapshot, including this one.

The binding constraint is memory: VRAM on a GPU, or unified memory on Apple Silicon (which is quietly one of the best consumer platforms for this). Quantization makes it workable: a 4-bit quantized model needs very roughly half a gigabyte per billion parameters, plus room for context. In practice that means a 7-8B model runs comfortably on an ordinary laptop, a 30B-class model wants 24GB+, and 70B-class models want a serious GPU or a high-memory Mac. Small models punch above their weight on narrow tasks and below it on open-ended ones; quantization costs some quality, mildly at 4-bit, noticeably below that.

What transfers and what doesn’t

Almost everything on this site applies unchanged: prompt structure, personas, injection caution, eval discipline (especially evals, since you must verify a small model handles your task before trusting it). What doesn’t transfer: provider-side conveniences like managed prompt caching, batch discounts, and server-side tools. Local trades those conveniences for control. Pick per task, not per ideology.