Diversity and maximal marginal relevance

Begin with a candidate pool that satisfies the task’s basic eligibility and relevance requirements. For each next choice, compare its relevance with its maximum similarity to any already selected item. The maximum penalizes a close duplicate even if it differs from other selected items. Recompute the penalty after each selection because the selected set changes. Define an initialization policy when the set is empty, such as seeding the highest-relevance item with a deterministic tie rule. Candidate truncation can limit which aspects are available before MMR begins.

Relevance and redundancy need not use the same model. A cross encoder may score query relevance while a document embedding measures inter-document similarity. That is allowed by the original construction, but the scales and failure modes differ. A paraphrase can add no new evidence despite different surface wording, while two similar passages can provide independent corroboration. Decide what redundancy means for the product: duplicate text, repeated claim, same source, or same user intent. One cosine value cannot certify all of those notions simultaneously.

A coefficient of one half does not imply equal practical influence when one input ranges from zero to one and another spans tens of units. Calibrate the scale convention or validate transforms and the coefficient together on held-out queries. A strictly increasing relevance transform can preserve a pure relevance ranking yet change MMR decisions because the criterion uses score differences and a weighted penalty. The same issue applies when swapping embedding models: a changed similarity distribution can alter diversity behavior even if each score still lies within nominal bounds.

Positive scaling exposes that dependence algebraically. Multiplying relevance by one factor and redundancy by another changes the effective tradeoff after factoring out an overall positive constant. This is a property of the selection rule, not evidence that either score is calibrated. Negative similarities also need a policy: a negative redundancy value creates a bonus in the raw formula. If the implementation clips, shifts, or otherwise transforms similarities, document that choice and validate its effect rather than presenting it as the unchanged original criterion.

Evaluate relevance and redundancy jointly with query-specific judgments. Some queries need one exact answer or several corroborating sources, while exploratory queries may benefit from covering several intents. A diversity penalty can suppress an essential closely related document or select a tangential outlier. Inspect those cases rather than optimizing an average similarity statistic alone. The candidate pool, seed choice, desired list length, and tie policy belong in a reproducible experiment. Include the downstream use when deciding whether reduced redundancy actually improves the answer.

The vector lab supplies nonnegative coefficients for a separate hand calculation. Reset, set angle to 0 degrees and magnitude to 1, predict components, then Step: (1,0). Compare a fresh Reset with both angle at 45 degrees and magnitude at 1 before Step: approximately (0.7071,0.7071). Use horizontal coefficient times relevance minus vertical coefficient times redundancy for fixed lesson candidates. The instrument does not select documents, track a selected set, or compute MMR; its components only illustrate how a tradeoff changes.