Updated June 2026

The most common architecture question in this space, and the one with the most overbuilt answers. Three options, in the order you should try them: paste it, retrieve it, or train on it.

Option 1: paste it into the context

If your data fits in the context window with room to spare, put it there and stop. A prompt with the relevant documents pasted in is the simplest, most reliable, and most debuggable architecture that exists: the model sees exactly what you sent, you can read exactly what it saw, and “transforming shown text” is the most reliable thing models do.

People skip this step because it feels too dumb to be the answer. With million-token context windows and prompt caching making re-sent context cheap, “just paste the manual” covers a startling fraction of real use cases: a policy handbook, a codebase’s core modules, a product catalog. The limits are real (attention dilutes before capacity runs out, so curation still matters) but the threshold for needing more machinery is much higher than the ecosystem’s enthusiasm for that machinery suggests.

Option 2: retrieve, then paste (RAG)

When the corpus doesn’t fit, or changes constantly, or only slivers are ever relevant, add a retrieval step: index the documents, search for what’s relevant to the query, paste the results into the prompt. That’s all RAG is: search bolted onto option 1.

The thing nobody tells you until you’ve built one: the quality of a RAG system lives almost entirely in the retrieval, not the generation. When answers are bad, it’s nearly always because the right passage never reached the prompt, not because the model fumbled it. So diagnose retrieval first (what was actually retrieved for the failing query?), invest in search quality (hybrid keyword-plus-embedding search beats either alone), and resist tuning prompts to compensate for a search problem. There’s a prompt for reviewing a RAG design that walks the failure modes.

And reconsider chunking pressure: with large context windows, retrieving whole documents (then letting the model find the needle) often beats retrieving paragraph shards, which lose surrounding context.

Option 3: fine-tuning, which is rarely the answer to this question

Fine-tuning is for behavior, not knowledge. It excels at teaching format, style, and narrow skills: always emit our schema, classify in our domain’s vocabulary, match our tone. It’s a poor mechanism for injecting facts: the knowledge lands fuzzy, can’t be updated without retraining, can’t cite a source, and the model will fill gaps with confident interpolation. If the question is “how does the model know about our data,” the answer is context (options 1 and 2), essentially always.

Where fine-tuning genuinely earns its keep is the distillation pattern: high volume on a narrow task, where a small tuned model matches a frontier one at a fraction of the cost. That’s a cost decision, not a knowledge one.

The decision in one paragraph

Fits in context? Paste it. Doesn’t fit, or changes hourly? Retrieve then paste, and spend your effort on search quality. Need the model to act differently rather than know more, at serious volume? Fine-tune. And at every step, the eval decides whether the added machinery earned its complexity, because each step up this ladder is a real increase in things that can silently break.