Define which comparisons can teach the intended behavior

Store the prompt and relevant conversation context, both candidate responses, their generating model and decoding configuration, and a stable record of the annotation procedure. A preference label means that a particular annotator or aggregation process favored one candidate under a rubric. It is not an absolute utility measurement or proof that the favored candidate is correct. Both answers may be wrong, or a stylistic advantage may dominate a narrow comparison. Preserve ties, inability-to-judge outcomes, and annotations needed to distinguish these cases rather than forcing every observation into an unexplained winner field.

Candidate generation determines the alternatives the learner can compare. Pairs from nearly identical outputs may reveal subtle distinctions but omit broad failure modes; pairs from unrelated quality levels may be easy yet provide little resolution near a release decision. Mixing generators can broaden coverage while introducing model-specific style clues. Track prompt family, task, language, answer length, and correctness evidence where relevant. Original RLHF work and InstructGPT demonstrate particular collection procedures, but their empirical results do not establish that any new collection inherits their coverage or generalizes to every user group.

A prompt with k distinct candidates has k(k−1)/2 unordered pair opportunities. Expanding a ranked list into all those pairs does not create that many independent prompt observations: the pairs share candidates, context, and often one annotation session. Decide whether to weight prompts, pairs, or annotator votes, because each choice changes the empirical objective. Preserve the original grouping so uncertainty analysis and train/test splits can respect it. A high pair count can be dominated by a few prompts with many candidates, masking sparse evidence for important prompt categories.

Audit coverage against a declared target, using more than an aggregate dataset size. Count represented prompt families, alternative generators, error types, and annotation populations. Deliberate oversampling can gather useful rare cases, but report it as a changed collection distribution and decide whether training or evaluation weights need adjustment. Active selection based on model uncertainty can focus annotation effort while leaving confident blind spots unqueried. Include an independent coverage-oriented sample instead of assuming uncertain pairs alone represent the deployment population. Missing comparisons cannot be manufactured by assigning larger weights to an absent category.

Split by the unit of intended generalization, such as prompt family, source conversation, or annotator population, and check near duplicates. Putting two candidate pairs from one prompt into train and test can overstate generalization to new prompts. Version rubric changes and candidate regeneration so the same identifier never silently means a different comparison. Retain a final evaluation set beyond repeated selection and tuning; a development set used to choose every collection revision is no longer untouched release evidence. Document which deployment slices remain unmeasured.

The modular lab illustrates only deterministic coverage of category labels. For each separate run, Reset, set both modulus m and additive step a=2, predict the orbit, then Play or Step. With m=6 the orbit visits three residues; with m=7 it visits all seven. This exact arithmetic is not preference sampling or an annotation scheduler. It demonstrates how a repeatable selection rule can leave categories unseen. Actual coverage needs the identified prompt and candidate records and a target definition, not a visually varied stream or a large number of repeated comparisons.