Advanced RAG
— from "roughly" to precise
Basic RAG (embed → search → paste) works in a demo and fails in production: it returns irrelevant passages, misses answers that exist, and causes hallucinations. In this guide: smart chunking, hybrid search, reranking, metadata filtering and GraphRAG — the techniques that turn "roughly" retrieval into an accurate system you can trust.
Why basic RAG fails in production
If you have read the basic RAG guide, you know the flow: you split documents into chunks, convert to embeddings, store in avector DB, and at query time you search for the similar chunks and paste them into the prompt. It works — until it does not.
Three problems break basic RAG the moment the data is real:
- Semantic search alone misses exact terms — embeddings understand meaning but do not always catch a catalog number, a file name or a rare term. The query "error E-4032" may return "general" results about errors.
- The right chunk exists but is not at the top — sometimes the answer is in a chunk ranked 15th, and the LLM only sees the top 5.
- Poor chunking tears the context apart — naive splitting cuts a sentence in the middle or separates a table from its heading, and the chunk becomes meaningless.
Every technique in this guide directly addresses one of these problems. We will go in the order of the flow.
Smart chunking — the foundation that decides everything
Chunking quality sets the quality ceiling of the whole system. A good chunk is a coherent unit of information that can be understood on its own. Strategies beyond naive split-by-length:
- Structural splitting — split by headings, paragraphs and sections, not by a blind character count. Keep tables and lists whole.
- Overlap — let chunks overlap slightly (a few sentences), so context at the boundary is not lost.
- Metadata enrichment — attach a source, section title, date and document type to each chunk. This is what enables precise filtering later.
- Contextual retrieval — a powerful technique: before embedding, add a short context sentence to each chunk explaining where it is from (e.g. "from the Q3 2026 report, revenue section"). It dramatically improves accuracy.
Hybrid Search — the best value per unit of effort
This is the upgrade with the best return, and it is relatively simple. Hybrid search combines two kinds of search and merges the results:
You combine both and merge the rankings (usually with a method like RRF — Reciprocal Rank Fusion). The result catches both meaning and exact terms — coverage neither method gives alone. Most modernvector DBs support built-in hybrid search.
Reranking — the precision stage
Fast retrieval (vector/BM25) is a "wide net" — it quickly brings many candidates, but its ranking is coarse. Reranker is a dedicated model (a cross-encoder) that reads the query and each candidate together and gives a far more accurate relevance score.
The winning pattern is two stages:
When the 5 chunks reaching the LLM are truly relevant, you can send less context — fewer tokens, a better answer. Reranking improves quality and lowers cost at the same time.
GraphRAG & metadata filtering
Metadata filtering
Remember the metadata we attached during chunking? Now it pays off. Instead of searching the entire store, filter first: "only 2026 documents", "only this department", "only product guides". This shrinks the search space, raises precision, and prevents returning stale or irrelevant information.
GraphRAG
Regular search finds single, isolated chunks. GraphRAG builds a knowledge graph — entities (people, products, concepts) and the relationships between them — and lets you answer questions that "connect the dots" across documents. A question like "which customers are affected by the change to policy X?" requires connecting information from different sources — something a single similarity search does not do well. GraphRAG is more expensive to build, so save it for cases that genuinely require reasoning about relationships.
The full pipeline — how it all connects
Build an evaluation set (questions + correct answers) and add one technique at a time, measuring the improvement. Without evals you add complexity without knowing whether it helps. Start with hybrid + reranking — they deliver most of the improvement.