When building RAG or on-site search, one design question is unavoidable: Should you use vector search, conventional BM25 keyword search, or both through hybrid search? The prevailing answer in the community is “both,” but many teams do not understand why or know how to set fusion parameters. This guide defines the boundaries of all three methods, then recommends an approach by use case. Running both has a cost, and not every system benefits enough to justify it.
Vector search retrieves by meaning, while BM25 retrieves by literal wording. Their failure modes are almost perfectly complementary.
What Each Retrieval Method Actually Does
BM25: Exact but Rigid Word Matching
BM25 is a classic term-frequency algorithm and the default scoring method in Elasticsearch. A document scores higher when it shares more terms with the query and when those terms are rarer. Its strength is exact matching. Product models, error codes, personal names, statute numbers, and API names are easy to retrieve as long as the user’s words appear in the document. Its weakness is equally direct: a paraphrase can make the match disappear. If a user searches for “expense reimbursement process” while the document says “procedure for settling business expenses,” BM25 may assign no score at all. Chinese-language systems also depend heavily on tokenization quality; a bad word split can leave BM25 effectively blind.
Vector Search: Semantically Smart but Sometimes Mistaken
Vector search uses an embedding model to map both the query and documents into vectors, then finds nearest neighbors by similarity in a vector database. It naturally understands paraphrases: “expense reimbursement process” and “procedure for settling business expenses” sit close together in vector space. Its weaknesses are precisely where BM25 is strongest:
- Proper nouns and identifiers are confused: “ERR-4032” and “ERR-4033” may produce nearly identical vectors despite referring to entirely different user problems;
- Out-of-domain vocabulary is poorly represented: An embedding model may never have encountered an internal project codename or company jargon, leaving its vector almost arbitrary;
- It always returns something: A nearest neighbor exists even when the collection contains nothing relevant. Without a threshold, downstream generation receives noise.
Hybrid Search: Run Both, Then Fuse the Rankings
Hybrid search sends the same query through BM25 and vector retrieval, then combines the two result lists into one ranking. The most common fusion method is reciprocal rank fusion (RRF). Because the scores from the two systems use incompatible scales, RRF ignores them and uses rank alone. Each document receives a combined score of Σ 1/(k + rank), with k commonly set to 60. A high rank in either list adds weight, while a document ranked highly by both rises to the top.
查询: "ERR-4032 上传失败怎么处理"
BM25 结果: 向量检索结果:
1. ERR-4032 处理指南 1. 文件上传常见问题
2. 错误码总表 2. 上传接口超时排查
3. 上传接口文档 3. ERR-4032 处理指南
RRF 融合后:
1. ERR-4032 处理指南 ← 两边都命中，稳居第一
2. 上传接口文档 / 错误码总表 / 文件上传常见问题 ...One step beyond RRF is weighted fusion, such as Weaviate’s alpha parameter or Milvus’s WeightedRanker, which adjusts the balance between semantic and keyword signals. Beyond that, a reranking model can reorder the fused candidates in a single pass; see What Is Reranking, and Why Does RAG Often Need It?.
Practical Results: Where Hybrid Helps and by How Much
We compared the three modes on a Chinese knowledge base containing technical manuals, policy documents, and support tickets. The results followed the same general direction as academic research and official evaluations from vector-database vendors:
| Query type | BM25 | Vector | Hybrid |
|---|---|---|---|
| Contains model numbers, identifiers, or proper nouns | Strong | Weak | Strong |
| Paraphrased or conversational question | Weak | Strong | Strong |
| Long sentence with several conditions | Medium | Strong | Strong |
| Out-of-collection question that should return nothing | Fairly good | Poor | Medium, with a threshold required |
In overall hit rate, hybrid search usually improves on vector search alone by several to more than ten percentage points. The gain depends on the proportion of queries that require exact matching. It is most visible in technical support, legal work, and e-commerce SKU search, while purely conversational Q&A gains less.
Before adopting hybrid search, measure how many real queries contain identifiers or proper nouns and therefore require exact matching.
Choose by Use Case
- Internal knowledge bases, technical support, and legal contracts: Hybrid search is the clear choice. These domains are dense with proper nouns and identifiers that vector retrieval alone will miss.
- Conversational consumer Q&A and chat assistants: Begin with vector search and stop there if it is sufficient. Hybrid search offers little benefit when exact matches are rare.
- Code search: Weight BM25 more heavily because identifiers are keywords, and chunk by function. Pure vector retrieval generally performs only moderately on code.
- E-commerce and product search: Use hybrid search plus structured filters, with category, price, and brand handled through metadata filters. Retrieval then focuses on semantic relevance.
- Teams already using Elasticsearch: Elasticsearch 8.x includes dense vectors and RRF. Enable hybrid search in the existing cluster instead of introducing another vector database.
There is also a hidden engineering cost. Hybrid search means maintaining two indexes at once, one inverted and one vector. Every write, deletion, and rebuild must update both, doubling the opportunities for inconsistency. Mainstream systems such as Qdrant, Weaviate, Milvus, and pgvector with pg_trgm or tsvector now provide sparse-vector or BM25 support. Prefer two index types in one database rather than stitching together separate systems yourself.
Implementation Checklist
- Sample 100 real queries from the logs and classify them manually. What proportion requires exact matching?
- If exact queries exceed 20%, adopt hybrid search immediately. Otherwise begin with vector search plus a fallback threshold.
- Start fusion with the default RRF setting of k=60 rather than tuning weights immediately.
- Build a 50-query evaluation set, run separate BM25, vector, and hybrid experiments, and confirm the gain with data.
- After launch, log empty retrievals and low-score matches. They identify gaps in the source material and tokenizer dictionary.
Choose in this order: examine the query mix, run a small comparison, and only then tune fusion parameters.
Who This Is For and Alternative Approaches
This guide is for engineering teams selecting a RAG retrieval layer or dissatisfied with an existing one. Two alternatives are worth knowing. First, query rewriting uses a model to turn a conversational question into a normalized search query and can sometimes improve results more than changing retrieval methods at lower cost. Second, for a small corpus of only tens of thousands of words, you can avoid retrieval entirely and place the full text in the context window, eliminating all retrieval-layer complexity. Retrieval is a means; accurate answers are the objective.
Frequently Asked Questions
Q: Does hybrid search add significant latency? A: The two searches run in parallel, so latency is determined by the slower path. The increase is usually on the order of 10–50 ms, which has little effect on overall RAG latency because generation dominates.
Q: What deserves special attention when using BM25 with Chinese? A: Tokenizer quality determines everything. Use a mature tokenizer such as IK or jieba and maintain a custom dictionary containing product names and project codenames. Character-level n-grams can also provide a fallback.
Q: Are sparse vectors such as SPLADE or the sparse output from BGE-M3 equivalent to BM25? A: They share the same general idea but are more powerful. A model learns term weights instead of relying on statistical frequency and can expand the query with related words. If your vector database supports sparse vectors, using them in place of BM25 is currently the stronger hybrid combination.
Conclusion
The decision reduces to three points: if queries contain many proper nouns and identifiers, hybrid search is essential; for purely conversational Q&A, vector retrieval with a threshold is enough; whichever method you choose, build an evaluation set before reaching a conclusion. The goal of the retrieval layer is not technical sophistication but clean, relevant, citable material for downstream generation.