How RAG works

A language model only knows what was in its training data, and it will happily make up the rest. Retrieval-augmented generation (RAG) fixes that by giving the model the right pages to read just before it answers: first find the parts of your documents that matter for the question, then ask the model to answer from those parts. It’s how Emma, the R.S. Bonum assistant, answers questions about Croatian taxes from the firm’s own booklets.

It runs in two phases: preparing the documents once, ahead of time, and then a short routine for every question.

1. Split the documents into chunks

Whole documents are too long to search by meaning and too big to hand to the model, so they’re cut into chunks of a few paragraphs. Neighbouring chunks overlap a little, so a sentence that falls on a boundary still appears whole in one of them.

Chunk size is a trade-off. Small chunks are precise but lose context (“it” no longer says what “it” is); big chunks keep context but dilute the part that actually answers the question.

2. Turn every chunk into an embedding

An embedding model turns a piece of text into a list of numbers (more on embeddings), a point in a space with hundreds or thousands of dimensions. It’s trained so that texts with similar meaning land close together, even when they share no words: “opening a sole proprietorship” ends up near “registering an obrt”. All the chunks’ embeddings go into a vector index (Emma uses FAISS), a structure built for one job: given a point, quickly find the points nearest to it.

3. For each question: find the nearest chunks

The question is embedded with the same model, and the index returns the chunks closest to it. “Close” is usually cosine similarity, the angle between the two vectors: pointing the same way means similar meaning, whatever their length. Emma takes the six nearest.

The third question shows retrieval’s blind spot, and it really happened with Emma. “Who founded the firm?” is phrased nothing like the About page that answers it, so similarity search landed among other chunks, and the bot said it didn’t know. The fix was plain engineering, not a cleverer model: the About page is now always in the context, and retrieval fills in the six chunks around it.

4. Put the chunks in the prompt, and let the model answer

The retrieved chunks are pasted into the prompt, together with instructions and the question. Simplified, Emma’s looks like this:

You are the assistant on the website of R. S. Bonum ... Respond in Croatian.
Facts about the firm, such as its people, prices and contact details, must
come from the knowledge base. Do not guess them. If they are missing, say so.

Knowledge base:
  [About page]
  [chunk 1: "Paušalni obrtnici plaćaju porez ..."]
  ...
  [chunk 6]

Question: Kako funkcionira paušalno oporezivanje?

Now the model isn’t recalling facts, it’s reading them: summarising, combining and rephrasing the chunks into an answer. That’s what language models are genuinely good at, and it’s why RAG answers can be checked: Emma lists the sources each answer came from.

What RAG doesn’t fix

• If retrieval misses, the answer misses. The model can only use what it’s given; most RAG bugs are search bugs, like the About page above.
• Chunks are read out of context. A table split across two chunks, or a rule that depends on a paragraph three pages earlier, can be misread.
• The model can still add things. The instructions (“must come from the knowledge base… do not guess”) reduce that but don’t eliminate it.
• The index is a snapshot. New or changed documents have to be chunked and embedded again before the bot knows about them.

The same idea, embeddings and similarity search, turns up all over my current research, from grouping news articles into topics to spotting new ones.