RAG in production: what no tutorial tells you

A RAG demo usually looks impressive.

You load a PDF, split it into chunks, create embeddings, ask a question, and the LLM returns a pretty good answer.

Then you try the same thing with real data.

That is where things get interesting.

In my work, I have built AI pipelines that process real business documents from the maritime sector. Those documents look nothing like the examples used in tutorials.

You encounter scanned pages, OCR errors, tables, information that continues onto the next page, domain-specific terminology, and the same concept expressed in five different ways. Often, the information you need is not contained in a clean, self-contained paragraph.

That is when I realised that the LLM is only one piece of the system.

If OCR loses the information, the LLM cannot find it. If chunking splits the document incorrectly, essential context may be lost before retrieval even begins. If retrieval returns irrelevant evidence, even an excellent language model can produce a highly convincing but incorrect answer.

That is why I think it matters to keep two questions separate:

Did we retrieve the right information?

and

Did the model use that information correctly?

Retrieval and generation are not the same problem.

The same applies to reranking. The chunk with the highest similarity score is not necessarily the best evidence for the question you are trying to answer.

Perhaps the biggest lesson for me, however, is the importance of evaluation. You can swap embeddings, chunk sizes, prompts, retrieval strategies, and models, convincing yourself each time that the system has improved. Without a genuine evaluation set from your domain, however, you are essentially tuning by eye.

That is not enough for production.

A demo needs to show that something can work. A real AI system needs to let you understand when it works, why it works, and how it fails.

For me, that is one of the biggest differences between using AI models and building AI systems.