Keep BM25 scores out of the confidence label
BM25 ranks matching documents using collection statistics. Treating its score as confidence can turn a useful search feature into an unreliable business rule.
By 
Paul. Architecture and infrastructure
A search score can order documents usefully without telling you the probability that any document answers the question. BM25 belongs in that category. If a product converts its raw score into a confidence percentage or uses one unexplained threshold to accept results, it gives a ranking mechanism a business responsibility it has not earned. The consequence can be a convincing result label that changes meaning when the document collection changes.
Looking back from November 2026, the relevant event in the week of 13 November 2024 was an explanation, not a new search algorithm release. Evan Schwartz published his BM25 essay on 19 November. His question about comparing scores is still useful because it forces a product team to say what it intends to compare.
Read the components before the number
Schwartz explains a lexical ranking built from query terms and collection statistics. Rare terms contribute differently from common ones; repeated matches have diminishing returns; document length is considered relative to the collection's average. Scores combine contributions from the terms being searched. Those ingredients describe evidence from text, rather than a calibrated probability of a correct answer.
His essay also argues that scores for the same document can be compared across queries within the same collection. That is a bounded claim in the author's reasoning. It should neither become a prohibition on every comparison nor a promise that arbitrary scores from different indexes share a common scale.
The distinction matters even before any mathematics enters the interface. “First result” describes an ordering. “Ninety percent confidence” describes an uncertainty claim. A team that changes one label into the other has changed the product's promise without adding evidence to support it.
Matching words can be exactly what the task needs
Consider an illustrative support archive containing product model numbers and repair notes. A customer searches for an exact model identifier. Lexical matching can help preserve that specific term instead of returning a discussion about a broadly similar product. This is a proposed use case, not an evaluation of an AL+ search system.
Now consider a document that repeats the identifier in every footer. Counting appearances without restraint would overreward repetition. Saturation makes further occurrences contribute less. A long manual also has more opportunities to contain a term than a short service note; length normalization adjusts that imbalance. Neither adjustment knows whether the passage actually solves the customer's problem.
Rarity introduces a collection-wide relationship. If an identifier appears in very few documents, matching it can be informative. If the archive later gains many documents about that same product, its rarity changes. The document's text may remain untouched while its relationship to the collection shifts.
This is a useful way to explain scores to a product owner. The number describes a match under particular conditions. It is not an enduring property stored inside the document, waiting to be discovered by every search system.
A fixed comparison needs more than a frozen query
Suppose the proposed support feature hides results below a chosen score. Before accepting that rule, keep the collection revision, query processing and scoring configuration fixed while inspecting examples. Otherwise a change in returned results may come from new documents, different term handling or a changed ranking setting. Those are different interventions with different remedies.
Cross-query comparison deserves particular care. A longer query can contribute several matching terms, while a short identifier query might have only one. Even within a fixed collection, the business meaning of the totals needs evaluation. Schwartz's bounded argument about comparison does not supply labels saying that a result is sufficient to answer a support request.
Define that outcome separately. For a browsing interface, success may mean placing a relevant manual near the top. For an automatic reply, success may require a passage that directly supports the statement being sent. A score that helps with the first task cannot establish the second merely because both tasks use the same search box.
A suggested assessment would use representative questions and human judgments about useful results. Include missing answers and exact identifiers as well as ordinary phrases. These are proposed evaluation cases, not experiments run for this article. Record ranking usefulness separately from the correctness of any answer generated afterwards.
The collection is part of the product contract
Adding documents is often treated as routine content maintenance. Under a threshold-based feature, it can also change behaviour. That means the owner of the archive and the owner of search need a shared release expectation. A large import should trigger a check of the decisions that depend on scores, rather than only a count of successfully indexed files.
An access-filtered application adds another design question: which collection statistics does the implementation use for the requesting user? Do not assume that every engine recomputes them in the same way. Document the actual indexing and filtering behaviour before making score comparisons across users. The point is to identify an implementation dependency, not to assert that BM25 itself enforces document permissions.
A simple ranked list may be enough
The strongest counterargument is that many products never need to expose scores. A useful ordered list, clear snippets and a link to the source can serve readers well. Building a confidence model may add cost without helping them choose. The appropriate response can be to remove the percentage, rather than replace the search engine.
Thresholds can still be useful after evaluation under defined conditions. Keep their purpose narrow, revisit them when the collection changes and provide a sensible empty-result experience. They should express a tested product choice rather than a claim that BM25 directly measures truth.
For a custom search application, begin by naming the decision made after retrieval. If a person chooses a document, optimize the ordering and explanation. If software acts on the result, require evidence for that action beyond the raw ranking score. Keep the number's meaning attached to its collection and its task.