greatitself.
Continue
advanced28 min studyLesson 666 in Engineering

Learning-to-rank objectives

Choose pointwise, pairwise, or listwise supervision with explicit query groups, label gains, ties, and evaluation alignment.

Before you begin
On this page

The question for today

What you’ll understand

  • Distinguish the training instances and assumptions of three ranking-objective families.
  • Compute a pairwise logistic preference and handle tied judgments explicitly.
  • Explain why optimizing a surrogate does not automatically optimize a deployment ranking metric.

A ranking model can be trained to predict one document label, compare two documents, or model an entire query’s candidate list. Those objectives expose different signals and weight examples differently. None removes the need for a relevance policy or a held-out ranking evaluation. This lesson uses the original RankNet and ListNet papers to make the distinctions concrete, while treating their probability constructions as training models rather than universally calibrated relevance estimates.

Pointwise targets assign a scale to labels

A pointwise model treats each query-document example as an instance. Regression can match a numeric grade, while classification can predict a relevance category. The chosen target and loss determine what differences between grades mean: squaring errors on grades zero, one, and two assumes a scale that is not implied by their ordering alone. If positives are oversampled or negatives come from a particular retriever, the training distribution also differs from arbitrary corpus pairs. Evaluate the deployed ranking rather than assuming a low regression loss establishes useful top results.

Query grouping still matters even when the loss is pointwise. A query with many labeled candidates can dominate a sum unless the sampling or weights compensate. Keep related queries, duplicates, and time-linked documents from leaking across training and evaluation boundaries. An unjudged document is not automatically a zero-grade label. State how missing judgments and disputed labels enter training; otherwise a seemingly precise numeric objective can encode an accidental annotation policy instead of the intended retrieval task.

Lpoint=∑q,dwq,d(s(q,d)−yq,d)2L_{\mathrm{point}}=\sum_{q,d}w_{q,d}\bigl(s(q,d)-y_{q,d}\bigr)^2
This is one illustrative pointwise regression objective, with nonnegative example weights and declared numeric grades y. It is not the only pointwise loss, and assigning numeric grades does not prove equal semantic spacing or probability calibration.↔ Scroll wide equations horizontally.

Pairwise losses learn a within-query preference

A pairwise objective asks which of two documents should rank higher for the same query. RankNet uses a logistic probability based on their score difference. Adding a common constant to every score for that query leaves the pairwise probability unchanged, illustrating why the resulting absolute score level is not automatically meaningful across queries. Sampling pairs also changes which comparisons receive attention; a query with many candidates has many more possible pairs than a small one. Define query and pair weighting explicitly.

Tied judgments require a policy. One can omit equal-grade pairs from a strict-preference loss or assign a soft target of one half in a symmetric preference model. Arbitrarily treating a tied pair as a strict winner teaches an order unsupported by the label evidence. Ties in predicted scores also need a deterministic evaluation rule or a metric that handles them as declared. The ability to distinguish unequal grades does not guarantee correct document ordering at a particular cutoff when relevance labels are noisy or incomplete.

P(i≻j)=σ(si−sj),Lij=−tijln⁡P−(1−tij)ln⁡(1−P)P(i\succ j)=\sigma(s_i-s_j),\qquad L_{ij}=-t_{ij}\ln P-(1-t_{ij})\ln(1-P)
For one query, finite dimensionless scores, and a temperature-one logistic link, t is one for an observed strict preference or one half for a declared tie target. A tie may instead be excluded under another explicit policy. This models preference, not an individual document’s calibrated relevance probability.↔ Scroll wide equations horizontally.

Listwise normalization depends on the candidate list

A listwise method uses a query’s list as the training instance. ListNet’s top-one construction compares distributions derived from target labels and predicted scores. Equal target labels receive equal target mass under that construction; the rest of the list still changes the normalization. This is not equivalent to independent binary relevance classification. Other listwise methods and metric-weighted surrogates have different properties, so do not identify every ranking algorithm with one loss or claim that a surrogate directly maximizes every version of NDCG.

The network lab isolates the logistic preference link only. Reset, set input to 1 and weight to 1.5, predict its fixed-bias logit and output, then Step: logit one gives approximately 0.7311. Compare a fresh Reset with both input at 1 and weight at 0.5 before Step: logit zero gives 0.5. Interpret the logit as an already supplied pair-score difference for this analogy. The instrument does not learn a ranker, form query groups, evaluate a loss, or handle judgment ties automatically.

pi=esi∑jesj,ti=eyi∑jeyj,Llist=−∑itiln⁡pip_i=\frac{e^{s_i}}{\sum_j e^{s_j}},\qquad t_i=\frac{e^{y_i}}{\sum_j e^{y_j}},\qquad L_{\mathrm{list}}=-\sum_i t_i\ln p_i
This temperature-one top-one ListNet-style cross-entropy construction uses one finite candidate list for one query. Target labels are mapped through a declared exponential transform; equal labels have equal mass. Candidate membership and label scaling affect the distributions, which are not calibrated marginal relevance probabilities.↔ Scroll wide equations horizontally.

Work it through

Strict preferences and one tied pair

For one query, grades are (2,1,1) and predicted scores are (2,0,0). Use the stated pointwise squared loss without weights, and the temperature-one pairwise link. Count strict and tied unordered pairs and inspect their signals.

  1. There are three unordered pairs. The first document is strictly preferred to each other document, giving two strict pairs; the remaining pair has tied grades.

  2. Each strict pair has score difference two and preference probability σ(2), approximately 0.8808. A strict positive-pair loss is −ln σ(2), approximately 0.1269. The tied pair has probability 0.5 and can be omitted or given the declared tie target.

  3. The pointwise squared loss is (2−2)² + (0−1)² + (0−1)² = 2. This numerical target penalty is different from a pair-order penalty even though the predicted ordering respects every strict grade preference.

The result

Objective families can reward different aspects of the same scores; label and tie policies must be explicit.

The living notebook

Explore the model

Interactive lab

Interactive lab · network

A pairwise logistic-link analogy

Reset, set x = 1 and w = 1.5, predict z = wx − 0.5, then Step: z = 1 and σ(z) ≈ 0.7311. Compare a fresh Reset with both x = 1 and w = 0.5 before Step: σ(0) = 0.5. This fixed-bias unit does not train RankNet, compute a ranking loss, or construct query lists.

A pairwise logistic-link analogyOne inspectable sigmoid unit, σ(wx − 0.5). Playback traces the computation graph. There is no training, attention mechanism or hidden deep network. Current readings: Product · wx: 1.2; Logit · wx − 0.5: 0.7; Sigmoid output: 0.6682. All parameters are adjustable with the labeled controls below.One input. One weight. One sigmoid.x1.20× w1.20+ b0.70σ0.668output0.668w = 1.00b = −0.50σ(z) = 1 / (1 + exp(−z))
Computation graphSignal trace
Product · wx
1.2
Logit · wx − 0.5
0.7
Sigmoid output
0.6682
t = 0 s

One inspectable sigmoid unit, σ(wx − 0.5). Playback traces the computation graph. There is no training, attention mechanism or hidden deep network. Parameter changes pause playback and recompute the model at the current elapsed time. Reset restores both parameters and returns time to zero. Playback starts paused. Adjust either parameter, then choose Play or Step.

Look forA logistic score difference models a preference under assumptions, not an automatically calibrated document relevance probability.

Pause & reason

Check your understanding

Two documents have equal relevance grades. What is an appropriate pairwise-training treatment?

Your turn

Try it yourself

Grades are (1,1,0) and scores are (0,0,−1). Count strict and tied unordered pairs, compute the strict-pair preference probability, and explain whether a common score shift changes it.

Take a moment to make a prediction or work it out on paper. Open a hint when you need one.

Give me a hint

Compare within the same query and use score differences.

Compare with the solution

There are two strict pairs favoring either grade-one document over the grade-zero document and one tied pair between the first two. Each strict difference is one, so probability is σ(1), approximately 0.7311. Adding the same constant to all three scores leaves every difference unchanged. Absolute score level is therefore not identified by these pairwise probabilities, and a tie policy remains necessary.

Keep the idea

Take it with you

  • Pointwise targets, pair preferences, and list distributions optimize different surrogates.
  • Query weighting, sampling, and tie handling are part of the training objective.
  • A good surrogate value does not replace held-out metric and judgment validation.

Read further

Original sources and deeper explanations to follow at your own pace.

Optional progress, saved in this browser when available. Revisit any lesson at any time.

Lesson not marked complete.