Hard-negative mining and false negatives

A hard negative scores highly under the current retriever despite being irrelevant under the task’s relevance policy. Lexical overlap, a shared entity, or an answer with the wrong date can create an informative confusion. Mining from the current index exposes the model to its own mistakes, but the mining checkpoint, corpus snapshot, and candidate filters determine which mistakes it sees. Negatives mined once can become easy as training changes. Track that provenance rather than treating hardness as an intrinsic label attached permanently to a document.

Contrastive softmax training pushes the designated positive above the listed negatives. Its probability is normalized over that training candidate set, not over every document in the collection, and it is not automatically a calibrated relevance probability. The number and distribution of negatives influence the optimization signal. In-batch negatives are convenient, but another query’s positive can also answer the current query. A batch construction that silently marks all cross-query documents negative can therefore penalize relevant evidence even when every original annotation was correct.

Collections often contain only a small set of judged positives for each query. A top-ranked passage missing from that set is unjudged, not necessarily nonrelevant. Duplicates, paraphrases, alternate supporting passages, and different legitimate interpretations increase the risk. RocketQA specifically studies denoising top-retrieved hard negatives with a cross encoder. That procedure can reduce one observed error source, but a teacher can also make systematic mistakes. Validate its filtering behavior on a human-reviewed sample rather than treating the teacher’s score as ground truth.

Choose a policy appropriate to the evidence: exclude uncertain candidates, collect judgments, use validated soft labels, or support multiple positives in a declared objective. Do not simply keep the hardest items until training loss falls; a model can learn to suppress legitimate answers. Audit query classes and near-duplicate groups, and keep evaluation examples out of mining and teacher training. A decrease in contrastive loss may reflect fitting contaminated labels rather than better retrieval on the intended relevance task.

Log which retriever version mined a negative, its rank and score, its document identity, and the evidence for its label. Compare a clean easy-negative baseline with reviewed hard negatives under the same data and evaluation protocol. Monitor retrieval quality by query type, candidate recall, and annotation coverage rather than using training loss alone. If denoising removes too many candidates, inspect whether the teacher is rejecting useful difficult distinctions or discovering alternate valid answers; those cases call for different corrections.

The probability lab models a hypothetical contamination assumption only. Reset, set false-negative probability to 0.2 and number of mined items to 5, predict the chance of at least one false negative, then Step: 0.67232. Independence and a common known rate are assumptions. The instrument does not judge passages, mine documents, or estimate a real contamination rate. Correlated near duplicates can make this binomial model inappropriate, so use it to expose assumptions rather than certify a training set.