Truncate conversations without breaking their targets

Count system text, earlier turns, tool content, role markers, boundaries, and the current assistant target using the actual tokenizer. Character limits and message counts are poor substitutes. If the model's maximum length includes a prefix or special token, reserve it explicitly. Training and inference budgets may differ, but evaluation should expose whether the deployed prompt supplies the context the adapted behavior relies on. A preprocessing stage that silently truncates twice can produce a different example from the one inspected earlier.

A whole-turn policy can preserve a system instruction and the latest self-contained user-answer pair while dropping older pairs. That policy is invalid for examples whose latest question depends on removed facts, such as 'use the previous table.' Those examples need a different selection, summarization with validated semantics, or rejection. Keeping the most recent bytes is not a universal correctness criterion. Decide whether assistant turns earlier in the conversation are targets or context, and preserve the intended role structure.

A left crop can remove an opening role marker while retaining its content. A right crop can cut an assistant answer before its end token. Training on a partial answer may be intentional, but it then has a different stopping and target policy; do not invent a terminal token claiming the original answer was complete. Apply the same retained indices to IDs and aligned labels, or preferably regenerate validated spans and masks through the supported pipeline. Never leave old mask offsets attached to a new sequence.

Packing several examples introduces another boundary question. Concatenating tokens can permit attention across records unless the chosen implementation supplies isolation or the objective intentionally allows that context. Separator tokens alone do not necessarily enforce attention independence. Inspect the trainer's versioned packing behavior and loss alignment, especially for padding-free or variable-length kernels. A throughput improvement is not acceptable if a target can now condition on another record's answer in a way the intended task did not permit.

Inspect examples that barely fit, miss the limit by one token, contain a long tool response, or end in an incomplete assistant turn. Track how many records are shortened or rejected by task and source, since aggressive truncation can change the training mixture. Compare the distribution of selected target counts before and after preprocessing. A lower loss on shortened easy answers may reflect a changed task rather than better learning, so retain an unambiguous held-out evaluation format.

The function lab is a limited size-growth illustration. Reset and set both a=1, x=1, predict y=1; Reset and re-set a=1, x=2, predict y=4, then inspect. The quadratic can represent normalized full-attention score-count growth for a fixed architecture, not an actual trainer memory budget. It has no conversations, masks, packing, or truncation. Explain why reducing sequence length can save work yet still require a separate semantic check that the retained targets have the needed context.