Retrieval-Augmented Generation (RAG) Explained: Chunking, Embeddings and Reranking

Language models only know what was in their training data, and they cannot see your company's documents. Retrieval-augmented generation (RAG) solves this by looking up relevant text at question time and giving it to the model along with the question. The model then answers from the supplied material rather than from memory. The idea comes from research published in 2020, and it has become the standard way to build assistants over private or fast-changing knowledge.

The pipeline in two phases

Indexing (done ahead of time):

1. Collect documents: wikis, PDFs, tickets, code, web pages.
2. Clean and split them into chunks.
3. Convert each chunk to an embedding, a list of numbers representing its meaning.
4. Store chunks, embeddings and metadata in a search index or vector database.

Query (done for each question):

1. Embed the user's question with the same model.
2. Retrieve the most similar chunks.
3. Optionally rerank them for relevance.
4. Place the best chunks in the prompt with instructions to answer from them and cite them.
5. The model generates the answer.

Most quality problems in RAG are retrieval problems: if the right text is not retrieved, no model can answer well.

Chunking: the most underrated decision

Chunk size is a trade-off. Small chunks are precise but lose context; large chunks keep context but dilute the match and consume prompt space.

• Start with roughly 200 to 500 words, with some overlap (10 to 20 percent) so ideas are not cut in half.
• Split on structure (headings, paragraphs, list items, code functions) rather than fixed character counts.
• Keep metadata with each chunk: source, title, section heading, date, author, access permissions.
• Prepend the document title and section heading to each chunk's text so it makes sense out of context.
• Tables and code need special handling; splitting a table row from its header destroys meaning.

Embeddings: meaning as numbers

An embedding model maps text to a vector so that similar meanings are close together. Search finds the nearest vectors, usually by cosine similarity. Points to consider:

• Use the same model for documents and queries, and re-embed everything if you change models.
• Models differ in language coverage, domain fit and vector size; test on your own data.
• Embeddings capture topic well but can miss exact terms such as error codes, product names and IDs.

Hybrid search and reranking

Because semantic search misses exact matches, many systems combine it with keyword search (such as BM25) and merge the results. This hybrid search is usually a clear improvement for technical content.

A reranker is a second model that reads the question and each candidate chunk together and scores relevance more carefully than vector similarity can. A common pattern is to retrieve 20 to 50 candidates cheaply, rerank them, and send only the top 3 to 8 to the model. This improves accuracy and reduces prompt cost.

Prompting for grounded answers

Tell the model what to do when the context is insufficient:

Answer using only the sources below. Cite the source ID for each claim.
If the sources do not contain the answer, say you could not find it.
Do not use outside knowledge.

Sources:
[1] (Refund policy, updated 2026-08) ...
[2] (Shipping FAQ) ...

Question: ...

This reduces, but does not eliminate, hallucinations. Show citations to users so they can verify.

Evaluating a RAG system

Evaluate retrieval and generation separately.

• Retrieval: build a set of questions with known source passages. Measure how often the right chunk appears in the top results (recall at k).
• Generation: check that answers are faithful to the retrieved text and actually address the question.
• End to end: track the share of questions answered correctly, answered "I don't know" appropriately, and answered wrongly with confidence.

Change one thing at a time (chunk size, embedding model, reranker) and re-measure.

Security controls to build in

RAG gives the model access to your data, so access control matters.

• Enforce permissions at retrieval. Store each chunk's access rules as metadata and filter by the requesting user's identity in the query. Never retrieve everything and hope the model withholds it.
• Isolate tenants with separate indexes or enforced tenant filters, and test with cross-tenant queries.
• Treat retrieved text as untrusted. A document can contain instructions aimed at the model (prompt injection). Limit who can add documents and keep a record of sources.
• Protect sensitive content. Embeddings can leak information about their source, so apply the same retention and access rules as the original documents.
• Keep the index fresh. Deleting a document must also delete its chunks and embeddings, otherwise removed data can still be served.

When RAG is the right tool

RAG suits questions about changing or private knowledge where you need citations and access control. It is not ideal for teaching a model a new style or skill, where fine-tuning may fit better (comparison).