advanced27 min studyLesson 661 in Engineering

Lexical term weighting and BM25

Declare term-frequency saturation, field-length normalization, and IDF conventions before interpreting a lexical retrieval score.

Before you begin

On this page

The question for today

What you’ll understand

Explain how term frequency and document length affect a declared BM25 formula.
Distinguish the Robertson-style IDF from Lucene’s positive variant.
Reproduce lexical comparisons with explicit analysis and field statistics.

A lexical retriever can find an exact product code that a semantic model overlooks, but its score depends on tokenization and corpus statistics rather than a universal relevance scale. BM25 combines rarity, term-frequency saturation, and a length adjustment. This lesson follows Robertson and Zaragoza’s original-author treatment and explicitly compares Lucene 9.12.1 conventions. The worked arithmetic uses a declared textbook form, not a promise of identical engine scores.

Count the tokens the index actually stores

Term frequency counts occurrences after the index analyzer has transformed the text. Case folding, stemming, stop-word handling, synonyms, and field boundaries can therefore change both the matching terms and the length being normalized. A query containing an identifier, negation, or unit deserves inspection at that analyzed representation. A search failure may come from analysis or missing indexed content before any scoring function is applied. Record the analyzer and field configuration alongside BM25 parameters when comparing a retrieval experiment.

In the declared single-field formula, increasing a matching term’s frequency gives diminishing returns. Length normalization compares a document’s analyzed field length with the average field length. With positive IDF and other quantities fixed, a longer field receives less contribution for the same term count when b is positive. This is a model of verbosity and evidence, not a statement that long documents are intrinsically irrelevant. Chunking changes term counts, lengths, document frequency, and the unit judged relevant, so it changes more than a storage implementation detail.

S(q,d)=\sum_{t\in q}\operatorname{IDF}(t)\frac{f_{t,d}(k_1+1)}{f_{t,d}+k_1(1-b+b\ell_d/\overline\ell)}

This declared textbook form counts each distinct query term once, with no query-frequency factor or field boosts. Frequencies and token lengths are counts, k1 > 0 and 0 ≤ b ≤ 1 are dimensionless, and average length is positive. Missing terms contribute zero; scores are ranking quantities rather than calibrated probabilities.↔ Scroll wide equations horizontally.

Name the IDF and implementation conventions

The Robertson-style no-feedback IDF can become negative when a term appears in more than half the documents. Lucene 9.12.1 instead documents a logarithm with an added one, keeping that term weight positive for valid counts. These alternatives can change rankings, not merely display formatting. Distinguish document frequency, the number of documents containing the term, from the total number of occurrences. In a fielded index, the relevant document count and average length need to refer to that field’s statistics.

Lucene 9.12.1’s implementation also omits the textbook numerator factor k1 plus one in its frequency component and uses encoded length norms. For fixed shared k1, that omitted positive factor alone is a ranking-preserving scale change in the simple formula, but it affects raw score comparisons or fusion weights. Norm quantization, boosts, and analysis can add further differences. Use the engine’s explanation output and pinned source when reproducing exact scores instead of mixing a textbook formula with implementation-specific numbers.

\operatorname{IDF}_{\mathrm{RSJ}}=\ln\frac{N-n_t+0.5}{n_t+0.5},\qquad\operatorname{IDF}_{\mathrm{Lucene}}=\ln\!\left(1+\frac{N-n_t+0.5}{n_t+0.5}\right)

N is the applicable positive document count and 0 ≤ n_t ≤ N is document frequency. Natural logs give dimensionless weights. The first can be negative; the second is the Lucene 9.12.1 documented convention. Neither value alone is an empirical relevance probability.↔ Scroll wide equations horizontally.

Test lexical changes on the same judged task

Hold the corpus snapshot and relevance policy fixed when tuning saturation and length parameters. Inspect exact-match, rare-name, long-description, and morphology-heavy query classes separately, because an aggregate score can hide a regression. Adding a dense branch can complement lexical failures, but cannot repair a document excluded by preprocessing or permissions. Before training a later reranker, verify that its relevant documents survive the lexical candidate cutoff. Candidate recall and final ordering are different parts of the system.

The probability lab illustrates corpus-frequency assumptions only. Reset, set occurrence probability to 0.2 and trials to 10, predict the expected count, then Step: two. Compare a fresh Reset with both probability at 0.8 and trials at 10 before Step: eight. This binomial model assumes independent identical document occurrences; it does not estimate real corpus statistics, tokenize text, or compute BM25. The actual indexed frequency is a counted property of the corpus, not a random draw created by playback.

Work it through

Length normalization with a declared score form

For one matching term, use the displayed textbook formula with IDF = 2, k1 = 1, b = 0.5, term frequency 2, and average field length 100 tokens. Compare field lengths 100 and 300; ignore all other terms and engine norm encoding.

At length 100, the normalization term is 1 × (0.5 + 0.5 × 1) = 1. The frequency factor is 2 × 2/(2 + 1) = 4/3.
At length 300, the normalization term is 1 × (0.5 + 0.5 × 3) = 2. The frequency factor is 4/(2 + 2) = 1.
Multiplying by IDF gives scores 8/3, approximately 2.667, and 2. The longer field scores lower here at the same term count; these are not exact Lucene 9.12.1 scores.

The result

The comparison isolates one length effect under an explicit convention, without interpreting either score as a probability.

The living notebook

Explore the model

Interactive lab

Interactive lab · probability

Rare and common occurrence assumptions

Reset, set p = 0.2 and n = 10, predict expected occurrences, then Step: two. Compare a fresh Reset with both p = 0.8 and n = 10 before Step: eight. This exact binomial distribution is an assumed document-occurrence model, not indexed statistics or a BM25 calculation.

Exact probabilityHighlighted outcomeMean np

Mean · np: 5
Variance · np(1−p): 2.5
Total probability: 1

Success probability p0.5Trials n10

t = 0 s

These are exact binomial probabilities for independent trials with one fixed success probability. Playback tours the distribution; it does not resample outcomes. Parameter changes pause playback and recompute the model at the current elapsed time. Reset restores both parameters and returns time to zero. Playback starts paused. Adjust either parameter, then choose Play or Step.

Look forTerm rarity depends on the declared corpus and counting unit; a toy occurrence model does not determine relevance.

Pause & reason

Check your understanding

Interactive checks need JavaScript. You can still read the explanation.

Read the checkpoint explanation

BM25 names a family with implementation conventions. Lucene’s positive IDF and frequency scale differ from some textbook forms, and the indexed analysis and length representation also matter; raw scores do not carry a universal probability meaning.

Your turn

Try it yourself

Using the same declared textbook convention, set IDF = 3, k1 = 1, b = 0, and compare term frequencies one and three. Compute both scores and explain the diminishing return.

Take a moment to make a prediction or work it out on paper. Open a hint when you need one.

Give me a hint

With b = 0 the denominator’s normalization contribution is one regardless of length.

Compare with the solution

At frequency one, the factor is 2/(1 + 1) = 1 and the score is three. At frequency three, the factor is 6/(3 + 1) = 1.5 and the score is 4.5. Tripling occurrences increases this contribution by only 50%, not threefold. The result depends on the declared convention and says nothing by itself about measured relevance or another engine’s raw score scale.

Keep the idea

Take it with you

Declare tokenization, field statistics, and BM25 formula conventions before comparing scores.
Saturation and length normalization change evidence weighting under explicit assumptions.
BM25 scores rank candidates; they are not universal relevance probabilities.

Read further

Original sources and deeper explanations to follow at your own pace.

Optional progress, saved in this browser when available. Revisit any lesson at any time.

Lesson not marked complete.