Learning problems and trustworthy data

Write a task specification in one sentence: at request arrival, estimate the probability that this request exceeds 200 milliseconds, using only arrival-time features, for traffic from the intended deployment population. State what action follows: perhaps route to spare capacity. Classification predicts a category or event probability; regression predicts a numerical quantity. A useful output also needs an action policy, because a probability alone does not decide whether extra capacity is worth its cost.

Define labels from an auditable process. A timeout is not necessarily a measured latency, and requests cancelled by users may have missing outcomes rather than negative labels. Ask whose examples are absent, how delayed labels arrive, and whether an intervention changes the target. Predicting historical queue delay does not automatically predict delay after a new routing policy changes the queue. This is the distinction between estimating patterns in observed data and establishing the consequences of actions.

Training data fit parameters. Validation data guide decisions such as architecture, regularization, and thresholds. A final test set estimates performance after those decisions are frozen. Random row splits make sense only when rows approximate independent draws from the relevant population. For a future-traffic predictor, split by time; for new-customer generalization, separate customers. A repeated user's nearly identical records across splits can otherwise turn apparent generalization into recognition of that user.

Fit learned preprocessing on the training partition, then apply the resulting transformation unchanged elsewhere. This includes means for normalization, imputers, vocabulary construction when it is learned from task data, and supervised feature selection. Information can leak without an explicit target column: an incident-close timestamp is a future fact, and a cached aggregate may include the very interval being predicted. Keep feature provenance and event-time boundaries alongside the data, rather than treating leakage as a final checklist.

Build a baseline that a colleague can explain: always predict the historical event rate, use a simple rule, or fit a small linear model. Compare against the operational policy already in use. A complex model that improves average accuracy but doubles costly false alarms may be a regression. Report sample counts, subgroup behavior, and a sensible uncertainty interval; a difference based on ten rare incidents is much less settled than one based on ten thousand.

Data quality is part of the model's behavior. Monitor missingness, unit changes, duplicate identifiers, and shifts in the population after launch. A label definition can remain fixed while its collection mechanism changes. Before trying more capacity, inspect examples where the baseline fails, including disagreements between annotators. An error taxonomy often reveals that a task needs clearer labels, a different prediction horizon, or an explicit abstention path rather than another layer.