Linear regression as a complete learning experiment

For one feature, predict $\widehat y=wx+b$. If $x$ is payload size in megabytes and $y$ is latency in milliseconds, $w$ has units of milliseconds per megabyte and $b$ has units of milliseconds. The intercept is the predicted value at zero input, which may lie outside observed data. Do not force it to be physically meaningful without checking whether the linear relationship is sensible near zero. A line is a local statistical hypothesis, not a law of the service.

With multiple features, each weight describes a conditional change while the other model inputs stay fixed. Correlated inputs make this interpretation fragile: request size and compressed size may carry nearly the same information, so many coefficient combinations predict similarly. A predictive coefficient does not establish the effect of an intervention. Changing payload size may also change cache behavior, routing, or client type, all of which can break an observational interpretation.

A residual is prediction minus observed target. Increasing a weight changes each prediction in proportion to that example's feature value, so its gradient is an average of residual times feature. Gradient descent subtracts a small multiple of that gradient. At a stationary point the residual vector is orthogonal to every column of the design matrix: no small move along an available feature direction can lower squared error. This geometric condition is more informative than treating the optimizer as a black box.

A unique least-squares coefficient vector requires independent columns. When columns are dependent, fitted predictions may still be unique while coefficients are not. Solve least squares using stable numerical methods such as QR or singular-value decomposition rather than explicitly forming a matrix inverse. Standardizing features can improve optimization conditioning, but its mean and scale must come from training data. Rescaling does not add predictive information or repair a poorly specified task.

Plot residuals against prediction, feature values, and time. A curved pattern suggests missing nonlinearity; widening spread suggests changing noise variance; a time trend suggests drift. One extreme target can dominate squared loss. Absolute-error or robust losses express different priorities, and they should be chosen with the problem rather than used to hide inconvenient examples. A log target may help multiplicative noise but changes what errors mean when predictions are transformed back.

Evaluate on a held-out partition using units people understand, such as mean absolute latency error, alongside the training objective. Compare against predicting the training mean. A small training residual is not evidence of useful extrapolation: a line fit between one and ten megabytes may fail at a gigabyte when a storage threshold changes. Record the input range and intended use. The best lesson from a linear baseline is often exactly where its assumptions stop describing the system.