RAG vs fine-tuning for production AI: how to choose

RAG vs fine-tuning for production AI: RAG supplies current facts, fine-tuning changes behavior. How to choose, and what RAG needs in production.

By Published 4 min read

Three rising steps: instructions first, then retrieval of your current facts, then fine-tuning to change behavior.

Choosing between RAG and fine-tuning for production AI starts with the problem and not the technique. RAG gives a model the content it needs at request time. Fine-tuning trains the model so that it behaves differently. If answers must reflect facts that change, RAG can be the starting point. If outputs must follow one form or style, fine-tuning can help. RAG in production brings its own work: retrieval quality, freshness and permissions.

RAG changes what the model can see when it answers. Fine-tuning changes how the model behaves. Pick by the problem you have.

This article is for technical owners and operators who are moving an AI system from pilot to production and have to decide how to give the model your knowledge and behavior. It compares the two approaches, says when to combine them, and lists what each needs in production.

Key takeaways

  • RAG is for current facts. It retrieves content at request time and gives it to the model.
  • Fine-tuning is for behavior. It trains the model on examples of the form or style you want.
  • Start light. Clear instructions first, then retrieval, then fine-tuning if a gap remains.
  • Test each on your own cases. Score retrieval and behavior separately.
  • Plan the production work. Permissions, freshness, versions and a test set kept after go-live.

What is the difference between RAG and fine-tuning?

Retrieval-augmented generation, or RAG, combines a pre-trained model with a store of content it can look up. The original RAG paper describes models that combine pre-trained parametric memory and non-parametric memory for language generation. In plain terms: the model writes the answer, and the retrieval step supplies the facts it writes from.

Fine-tuning continues the training of a model on examples you provide, so that its behavior shifts toward them.

A table comparing RAG and fine-tuning on what each changes, what each suits, how each is updated, how each relates to sources, and the production work each needs.

How do you choose between them?

Start from the problem.

Three problems and the approach for each: answers that must reflect changing facts need RAG, outputs that must follow one form need fine-tuning, and both needs are tested separately.

  • Answers must reflect facts that change. Use RAG. You update the content, and the next answer can reflect it.
  • Outputs must follow one form or style. Consider fine-tuning, with examples of the form you want.
  • Both. Use both, and test them separately.

Before either, check the cheaper step. Clear instructions with a few examples can close some gaps. Move to retrieval or training when a gap remains on cases from your own operation.

How do the two compare on cost and effort?

Both take effort, in different places. With RAG, the work is building the retrieval and keeping it working, keeping the content up to date, and setting access rules so people only retrieve what they may see. With fine-tuning, the work is preparing and approving examples, running the training, training again when the base model or your needs change, and checking for regressions each time.

Neither is a one-time job. Which one costs more depends on your content, how often it changes and how often the behavior has to shift, so estimate both from a small test on your own data before you commit.

How do you run RAG in production?

RAG in production is a retrieval system with a model on top, and the retrieval is where problems can start. Plan for five things.

  1. Retrieval quality. The model can only use what it is given. Score whether the right content came back, apart from whether the final answer read well.
  2. Freshness. Decide how content gets in, who owns it, and how fast a change should show up in answers.
  3. Permissions. Retrieval should respect who may see what. A system that retrieves content the user is not allowed to read has a permissions problem, whatever the model says.
  4. A record. Keep what was retrieved for each answer, so a wrong answer can be traced to its source. That is the case for LLM observability.
  5. A test set. Build it from your own cases, and re-run it after any change to the content, the retrieval or the model.

What does fine-tuning need in production?

  • Training data you can account for. Know where the examples came from, who approved them and what they contain.
  • Versions. Keep track of which tuned model is running, and be able to go back.
  • Regression checks. A tuned model can improve on the target behavior and get worse elsewhere. Run your test set on every new version.
  • A plan for change. When the base model or your needs change, you may have to prepare examples and train again.

Can you combine them?

Yes. They solve different problems: retrieval supplies the facts and tuning shapes the behavior. When you combine them, the failure can come from either side, so score the retrieval and the output separately. The scoring itself is explained in what LLM evals are, and the testing of agents in how to evaluate AI agents.

What goes wrong in production?

  • The test set is missing. The choice is made on a few examples that looked good.
  • Retrieval is judged by the final answer only. A good-looking answer can hide a weak retrieval.
  • Permissions come last. The system retrieves content its users should not see.
  • A tuned model changes with no regression check. Behavior shifts in places nobody tested.

What should you do next?

Write the problem in one sentence: facts that change, a form to follow, or both. Choose the lightest approach that addresses it, build a test set from your own cases, and put the production work for your choice on the page that defines production, as in define production for an AI system.

Our forward-deployed engineers work directly with your business and technical teams to take a system from pilot to production. Read about forward-deployed engineering.

Bring us the problem, not the specification. Book a call. One call is enough to know whether we can help.

Questions

What is the difference between RAG and fine-tuning?
RAG retrieves relevant content at request time and gives it to the model. Fine-tuning trains the model further on examples, which changes how it behaves. One changes what the model sees, the other changes the model.
Which should I use first?
Start with clear instructions, then add RAG if answers must reflect your current facts. Consider fine-tuning when outputs must follow a consistent form or style that instructions do not achieve.
Can I use both?
Yes. They solve different problems. Test the retrieval and the tuned behavior separately so you can tell which one moved a score.
What does RAG in production need?
Good retrieval, content that stays current, permissions so people only retrieve what they may see, a record of what was retrieved, and a test set built from your own cases.
How do I update knowledge in each approach?
With RAG you change the content it retrieves. With fine-tuning you prepare examples and train again, then check that nothing else got worse.
How do I evaluate the choice?
Score each option on cases from your own operation, and keep the test set after go-live so you can re-run it when anything changes.

Bring us the problem.

Book a call