Reranking — more accurate RAG

Why vector retrieval alone isn't enough

In basic RAG, you search a Vector DB for the passages whose embedding is closest to the query. That's fast and great for "coarse filtering," but not always accurate: the embedding captures general meaning, and sometimes ranks a passage that "sounds similar" but doesn't really answer the question highly. The result — irrelevant passages in the context, which lead to worse answers and even hallucinations.

Two-stage retrieval

The idea: combine two complementary stages —

1. Broad retrieval: the Vector DB returns many candidates (say 50) — fast, narrowing to a relevant subset.
2. Precise ranking (rerank): a reranker model examines each candidate against the query and returns the truly best N (say 5) — slower, but far more accurate.

You feed the LLM only the top 5 after reranking — a concise, precise context (see also Context Engineering).

How a reranker works — cross-encoder vs bi-encoder

The technical difference that explains the accuracy:

• Bi-encoder (embeddings): encodes the query and the passage separately into vectors, and compares. Very fast (you can precompute), but less accurate because there's no interaction between the query and the passage.
• Cross-encoder (reranker): feeds the query and the passage together into a model that returns a relevance score. Much more accurate because it "reads" both together — but slow, so you run it only on a few candidates.

Hence the logic of two stages: a fast bi-encoder for coarse filtering, an accurate cross-encoder for the polish.

Hybrid Search — a bonus

A complementary improvement: combine semantic search (embeddings) with keyword search (BM25). Each has an advantage — semantic captures meaning, keyword captures exact terms, names and codes. You merge both candidate lists, then run a reranker over the union. This gives the best of both worlds and is excellent for Hebrew and for technical terms.

Code example — a reranker in service

Reranking services (like Cohere Rerank) take a query and a list of documents and return them ranked. Open models also exist (BGE-reranker) via Hugging Face.

# Stage 1: broad retrieval from the Vector DB
candidates = vector_search(query, top_k=50)   # 50 candidates

# Stage 2: re-rank (example with a reranker library)
scored = reranker.rank(query=query,
                       documents=[c.text for c in candidates])
top = sorted(scored, key=lambda x: x.score, reverse=True)[:5]

# Stage 3: feed the LLM only the top 5
context = "\n\n".join(t.text for t in top)
answer = llm_answer(query, context)

Tips

• Retrieve broad, return narrow. Try top_k=30–50 on retrieval, and return 3–8 after rerank.
• Measure with evals. Check the accuracy improvement on a real eval set — not "by eye."
• Latency: a reranker adds time. Balance accuracy and speed as needed.
• Hebrew: make sure the reranker supports multilingual; hybrid helps especially for terms and names.
• Cost: a reranker is cheap relative to the LLM, and saves tokens because you feed fewer passages.