Updated June 2026

The grounding note for everything else here. No math, just the handful of mechanics that explain most LLM behavior you’ll ever see.

It’s next-token prediction all the way down

A language model does one thing: given a sequence of text, it predicts what comes next, as a probability distribution over its vocabulary. Generate a token, append it, predict again, repeat. Every capability you’ve seen (reasoning, coding, conversation) is this one operation, executed well, at scale. There’s no separate “logic module” or “fact database”; there are weights that, having compressed an enormous amount of text, make stunningly good predictions.

This is why the failure modes look the way they do. The model isn’t retrieving facts and occasionally fumbling one; it’s always generating plausible continuations, which usually coincide with truth and sometimes don’t. Hallucination isn’t a bug in the mechanism. It is the mechanism, on a bad day.

Tokens, not words, and definitely not letters

Models read and write tokens: chunks of a few characters, learned statistically from text. “The” is one token; “antidisestablishmentarianism” is several; code and non-English text fragment into more. Two practical consequences. First, anything character-level is unnatural for the model: counting letters in a word, reversing strings, precise character math; it literally doesn’t see letters, it sees chunks. (Modern models often get these right anyway, but the workaround is reasoning its way around the blind spot, not seeing letters. The blind spot is structural.) Second, tokens are the billing and capacity unit, so everything in Managing Token Costs is denominated in them.

Arithmetic is shaky for a related reason: numbers fragment into tokens arbitrarily, and next-token prediction is a poor calculator. The fix isn’t a smarter prompt; it’s giving the model a calculator (a code tool) and letting it use it.

Temperature, and why answers vary

The model produces a probability distribution; sampling picks from it. Temperature controls how adventurous the pick is: low temperature concentrates on the top choices, high temperature spreads the odds. This is why the same question gets different answers on different runs, and why regenerating is a legitimate strategy. (Many reasoning-focused models no longer expose temperature as an API parameter at all, but the sampling variance underneath is still there.) It’s also why inconsistency across runs is a useful suspicion signal: facts the model actually knows tend to survive resampling; confabulations vary.

The assistant is a trained behavior

Pretraining produces a text predictor. Post-training (instruction tuning, RLHF, and similar) shapes it into something that answers questions, follows instructions, refuses some things, and apologizes too much. The assistant personality is learned behavior layered onto the predictor, which is why personas work: you’re steering behavior that was always a trained choice, not a fixed property.

Nothing persists, nothing is learned

Weights are frozen at inference time. The model doesn’t remember yesterday’s conversation, doesn’t learn from your corrections, and doesn’t know anything that happened after its training data ends. Within a conversation, “memory” is just the transcript being re-read on every single turn (with all the cost and degradation implications that follow). Across conversations, memory is whatever your tooling writes to disk and pastes back in.

Hold these five facts and most surprising LLM behavior stops being surprising: it predicts tokens, it sees chunks rather than letters, sampling adds variance, the helpfulness is trained, and nothing persists unless you persist it.