If your RAG system frequently encounters this scenario—the retrieval logs show the correct document was recalled, but it ranks seventh or twelfth and gets cut off outside the Top 3, leaving the model to force an answer from three irrelevant snippets—what you lack isn't a better vector model. You need a Rerank (re-ranking) layer.
Reranking is currently one of the most cost-effective components for optimizing RAG: without changing your corpus, swapping out your vector database, or altering chunking strategies, simply adding a single model call often yields visibly improved retrieval quality. This article clarifies its principles, applicable boundaries, and practical configuration.
The sole task of reranking is to place the "one item that can actually answer the question" at the very top.
Why Vector Retrieval Alone Can't Rank Accurately
Vector retrieval relies on a Bi-Encoder architecture: queries and documents are encoded into vectors independently, after which cosine similarity is calculated. Document vectors are pre-computed and stored in a vector database—this enables millisecond responses even across billions of data points, but it's also why ranking accuracy suffers: when encoding the document, the model has no idea what future queries might target it. Compressing text into a vector with hundreds or thousands of dimensions inevitably loses detail; similarity measures "thematic closeness," not whether the content can answer this specific question.
Cross-Encoders (reranking models), by contrast, concatenate the query and document before feeding them into the model. This allows attention mechanisms to interact directly between every word in both inputs, outputting a score representing "how relevant this content is to this problem." They can see what Bi-Encoders miss: if the question asks "How do I get a refund?" but the passage discusses "How to issue an invoice," they share a post-sales theme yet fail to answer. While vector similarity might be high, the Cross-Encoder will assign a low score.
The trade-off is equally clear: every "query-document" pair requires a forward pass through the model, making pre-computation impossible. Scanning the entire database with this approach isn't feasible, which has led engineering teams to adopt a standard two-stage retrieval pipeline:
Coarse ranking prioritizes "don't miss anything" (recall), while fine-grained ranking focuses on "get the order right" (precision). Their roles are distinct. This also answers a common question: Reranking does not replace vector retrieval; it sits directly after it.
How Much Improvement Does Adding Rerank Deliver?
Official benchmarks from reranking model providers (Cohere, BGE, Jina, etc.) and community reproductions point to the same conclusion: assuming a decent retrieval baseline is in place, adding a reranker typically boosts Top-3/Top-5 hit rates by 10%–30%. It also improves results after Hybrid Search fusion—RRF looks only at rankings without reading content, whereas Rerank actually reads it. Based on our own experience with Chinese knowledge bases: bad cases where relevant documents were retrieved but failed to rank in the top three largely disappear once a reranker is added.
However, there are two scenarios where adding a reranker yields no benefit:
- No recall from coarse ranking: If the correct document isn't even in the Top 50, the reranker has nothing to work with ("a clever cook can't make soup without rice"). First, troubleshoot retrieval issues using 《Why RAG Answers Are Inaccurate》.
- Poor chunking quality: If the chunks themselves are fragmented or chaotic, even perfect ranking leaves the model with low-quality material. Start by reviewing 《How to Chunk Documents for RAG》.
How to Choose a Reranking Model
| Solution | Form Factor | Characteristics |
|---|---|---|
| BGE-Reranker (bge-reranker-v2 series) | Open-source, self-hosted | Strong performance on Chinese; most widely used in the community; multiple sizes available |
| Jina Reranker | API / Open-source | Multilingual support; handles long documents |
| Cohere Rerank | API | Stable results; easy integration; billed by usage volume |
| Qwen-Reranker and other domestic series | Open-source / API | Worth prioritizing for evaluation in Chinese scenarios |
| Using an LLM as a reranker | API | Lets the large model score candidates individually; excellent quality but slow and expensive, suitable mainly for offline evaluation |
Selection advice: If you have GPU resources or are willing to use inference services, choose open-source self-hosted models (starting with the BGE series). For smaller workloads where simplicity is key, go straight to an API. Pay attention to three engineering parameters:
- Candidate count: Feeding 20–50 candidates from coarse ranking into Rerank offers the best cost-performance ratio; pushing beyond 100 significantly increases latency and costs with diminishing returns.
- Truncation length: Most reranking models impose input length limits, so excessively long chunks will be truncated—another reason to avoid making chunks too large.
- Latency budget: Self-hosted small models typically score batches in tens to a couple of hundred milliseconds; API latency depends on network conditions. Compared to the several seconds often required for LLM generation, this delay is generally acceptable but unsuitable for latency-sensitive scenarios like search suggestion boxes.
Self-hosted reranking models have modest hardware requirements; a single consumer-grade GPU can handle typical knowledge base traffic.
A Ready-to-Use Configuration
Take "vector + BM25 hybrid retrieval with BGE reranking" as an example; this is a reliable starting point for Chinese knowledge base scenarios:
Coarse: Hybrid Search (RRF, k=60), retrieve Top 30
Fine: bge-reranker-v2-m3, score all 30
Filter: drop anything scoring below 0.3 (calibrate the threshold on your own eval set)
Output: take the Top 5 into the context; if everything is below the threshold -> answer "not found in the knowledge base"The final line deserves emphasis: Rerank scores serve as a natural basis for refusing to answer. Because of scale issues, it is difficult to set thresholds based on vector similarity alone, whereas Cross-Encoder relevance scores offer much better discrimination. If all candidates receive low scores, it indicates the knowledge base genuinely lacks an answer; explicitly refusing to respond is superior to forcing the model to hallucinate, directly reducing hallucinations.
Validation Process
- Filter out cases from bad examples where documents were retrieved but ranked too low (verify via retrieval logs that correct documents appear within the Top 30);
- Integrate reranking and run an A/B test on this batch, comparing only whether the correct document appears in the Top 5;
- Run a full pass using the evaluation set to confirm you haven't degraded rankings for questions that were previously ranked well (a small number of such cases is normal);
- Calibrate the refusal threshold: use ten queries where the knowledge base definitely contains no answer, examine their score distribution, and set the threshold at a level that blocks them.
Rerank benefits are easy to verify: identify bad examples where documents were retrieved but failed to rank in the top three, then compare results before and after integration.
Target Audience and Alternatives
This approach suits teams that already have a RAG system where retrieval logs indicate "recall is fine, but ranking fails." There are three tiers of alternatives: First, expand Top-K to increase context length by feeding the entire Top-10 list into the model. This is simple and direct, yet longer contexts drive up costs and introduce issues like "lost in the middle" (see Are Long Context Models Really Better?). Second, upgrade the embedding model; this addresses the root cause but requires a full index rebuild. Third, employ query rewriting, which is more effective than Rerank for cases where "the question itself was poorly phrased." The ultimate architecture for most teams combines good chunking + Hybrid recall + Rerank fine-ranking, with all three layers stacked together.
Frequently Asked Questions
Q: Is Rerank the same as RRF fusion? A: No. RRF performs mathematical fusion based solely on rank without reading content; Rerank involves a model actually scoring after reading both the "query and document." You can first fuse two recall paths using RRF, then pass them to Rerank for fine-ranking, effectively chaining the two methods.
Q: Is it cost-effective to run every query through a reranking model? A: The per-query cost of self-hosted small models is negligible; API solutions are priced by vendor but typically remain far lower than large-model generation costs. In contrast, the expenses associated with poor answers and manual troubleshooting caused by skipping reranking are significantly higher.
Q: Can Rerank be used for tool retrieval in Agent scenarios? A: Yes. When dealing with many tools (dozens or hundreds), a common practice in Agent tool management is to first use vector search for coarse candidate selection, then apply reranking to identify the most relevant few before injecting them into the prompt.
Summary
Keep this division of labor in mind: retrieval ensures nothing is missed, while reranking ensures the right order. If your failure cases are concentrated on scenarios where correct content was retrieved but failed to rank within the top results, adding a Cross-Encoder reranker is the fastest optimization technique available. It also provides reliable thresholds for refusing answers. Validate this approach with 20 bad examples before deciding to roll it out fully.