Estimate remaining generation without discarding the long requests

Distinguish the requested output cap, tokens actually generated, tokens delivered, and the length at which the model would naturally stop under a specified decoding policy. A cap is enforceable if the engine applies it, but it is not an unbiased prediction of natural completion. Cancellation can stop delivery before generation is acknowledged, and a server error can terminate the observation for another reason. Save stop reasons and progress so these cases are not all relabeled as short successful responses.

For natural-length estimation, a request that has produced forty tokens without its stopping event before a forty-token observation cutoff provides a lower-bound observation, not a completed length of forty. Dropping it and averaging only EOS completions can bias the estimate downward when long requests are more likely to be cut off. Statistical censoring methods require their own assumptions; cancellation based on user dissatisfaction or resource pressure may be informative rather than independent. Recording a censor flag is necessary but does not automatically solve that dependence.

A request still running after k generated tokens supplies new information about its total length. The expected remaining work is conditional on L>k and relevant request features. Subtracting k from an unconditional mean can be badly wrong: surviving beyond the short-response mode can leave only the long-response mode. A predictor should state whether it estimates total length at arrival or remaining length during execution, and when updates use observed progress.

Calibrate predictions on eligible cohorts with the same tokenizer and decoding settings as serving. Inspect underestimation rates, tail coverage, and error by prompt length, tenant, domain, and stop reason. A model can have a small overall mean error while systematically underpredicting one expensive class. Use an enforced maximum for a true resource cap; a chosen prediction quantile supports a probabilistic budget only with a justified calibration claim and an explicit policy when the prediction is exceeded.

If the scheduler prioritizes predicted-short requests, its completed-request log can overrepresent easy or short work. Training the next predictor only on that log can reinforce the original bias. Preserve offered requests, admission outcomes, incomplete observations, and enough controlled follow-up to assess the observation process. A model tested only on traffic it already chose to complete has not demonstrated performance on all offered requests. Changes in temperature, stopping rules, or model weights can also shift the length distribution and invalidate old calibration.

The probability lab isolates a fictional count of long-class requests under independent identical assignments. Reset, set both p=0.5 and n=10, predict five expected long-class assignments, then Play or Step. Reset and set both p=0.8,n=10 for eight. The exact binomial bars are not sampled request traces and contain no lengths, censoring, predictor, or conditional remaining-time update. They help separate class prevalence from a single observed batch, while the actual estimator must be evaluated on the recorded cohort and stop-policy evidence.