RAG or fine-tuning? Answer two questions

RAG changes what a model knows. Fine-tuning changes how it behaves. If your complaint is "it doesn't know our policies", that is retrieval. If your complaint is "it won't follow our format", that is fine-tuning. Almost every business problem we are asked about is the first kind, and the second kind is usually solvable without fine-tuning at all.

Two questions get you to an answer in about ten minutes.

Question one: is the information already written down somewhere?

If yes, you have a retrieval problem.

The instinct to "train the model on our data" is understandable and almost always wrong for this case. Fine-tuning bakes information into weights, which means:

• Updating a policy means retraining.
• There is no citation, so nobody can check an answer against the source.
• The model will still confidently answer questions the training data never covered.

Retrieval inverts all three. The document is the source of truth, updating it updates the system immediately, and every answer can point at the passage it came from. That last property is usually what gets a system approved internally: reviewers do not have to trust it, they can check it.

Question two: is the problem the shape of the output?

If the model knows the right things but says them wrong, whether that is the wrong format, the wrong register, or ignoring a house convention, that is a behaviour problem.

Before reaching for fine-tuning, work through the cheaper options in order:

1. A better prompt with worked examples. Several examples of correct output in the prompt solves a surprising share of format problems.
2. Structured output. If you need JSON matching a schema, constrain the output to that schema rather than asking nicely and validating afterwards.
3. A validation and repair step. Check the output; if it is malformed, feed the error back once. Deterministic, debuggable, and cheap.

If you have done all three and the behaviour is still wrong across hundreds of examples, fine-tuning is a reasonable next step. In our experience that is a small minority of cases, and it is usually about tone or a genuinely idiosyncratic output format rather than knowledge.

The case for doing both

They are not alternatives. A mature system often uses retrieval for knowledge and a fine-tuned or heavily-prompted model for consistent output shape. The mistake is reaching for fine-tuning first, because it is the expensive, slow-to-update half of that pair.

What this looks like as a decision

| Symptom | Likely fix | | --- | --- | | "It doesn't know our products / policies / history" | Retrieval | | "It's out of date the moment something changes" | Retrieval | | "We can't tell where an answer came from" | Retrieval, with citations | | "It ignores our output format" | Prompt examples, then structured output | | "It's too verbose / wrong register" | Prompt, then fine-tuning if it persists | | "It's slow and expensive at our volume" | A smaller model, distillation, or caching, not fine-tuning for its own sake |

The question nobody asks first

Whichever you pick: how will you know it worked?

The honest answer is a labelled set of real questions with known-good answers, scored before and after. Without it, "RAG or fine-tuning" is an aesthetic argument. With it, you can try the cheap option, measure, and only escalate if the number says you need to.

That is the order we work in, and it is why we build the eval set before the system rather than after.