advanced26 min studyLesson 839 in Engineering

Update forecast state without rewriting history

Separate fixed-parameter filtering, parameter refits, duplicate handling, and replay of corrected observations.

On this page

What you’ll understand

  • Compute a fixed-parameter recursive state update.
  • Explain why duplicates and corrections require different handling.
  • Specify atomic forecast state with offsets and model versions.

A forecasting service can change its next prediction after a new observation without retraining its parameters. It may update a filtered state, append a lag buffer, or advance a recurrent hidden state under fixed weights. That fast path still has correctness obligations: a duplicate event must not become a second observation, a correction must not masquerade as a later point, and a recovered state must match the consumed input position. Treat the state machine as a versioned computation over accepted observations, rather than as an opaque mutable cache that happens to improve predictions.

Distinguish state updates from learning parameters

Exponential smoothing gives an inspectable example: a fixed coefficient mixes the new value with the old level. Updating that level is not the same operation as choosing a new coefficient from a longer training sample. State-space filters make a similar distinction between filtering with fixed model parameters and refitting them. The statsmodels 0.14.4 forecasting example explicitly separates extending a result under existing parameters from appending with parameter re-estimation. Name both actions in the service’s lifecycle and logs.

Version state against the exact model and transformation artifact that produced it. Hidden state computed under one recurrent network’s weights is not generally valid under another network after promotion. A new model may require replaying a warm-up history, converting a state through a justified migration, or beginning in a documented cold state. Evaluate that transition in backtests. Otherwise a deployment can introduce prediction discontinuities that never appeared in evaluation because the notebook silently initialized every model from complete history.

ℓt=αyt+(1−α)ℓt−1,y^t+h∣t=ℓt\ell_t=\alpha y_t+(1-\alpha)\ell_{t-1},\qquad \hat y_{t+h\mid t}=\ell_t
0<α≤1 is fixed here, y and level ℓ share units, and this simple level-only forecasting rule assumes no trend or seasonality. It is an illustrative update, not a universal streaming forecaster.↔ Scroll wide equations horizontally.

Handle duplicates, corrections, and late insertions

Give observations stable logical identities and monotonic version rules. If the same accepted version is redelivered, deduplicate it before advancing state. Applying the smoothing update twice treats one measurement as two sequential observations and changes the level incorrectly. A corrected value for an earlier timestamp is a replacement in the logical history. Restore a checkpoint preceding that point and replay accepted observations, or use a proven model-specific correction formula when its assumptions truly hold.

For fixed-coefficient smoothing with no intervening structural changes, a correction’s contribution decays geometrically through subsequent accepted updates. This makes a useful reference oracle. It does not apply unchanged after a parameter refit, nonlinear clipping, missing-value policy change, or insertion that shifts the sequence of observations. Late events require a declared ordering policy. Event-time reordering with replay, bounded-lateness rejection, and correction-only scoring are different products and should not be mixed implicitly in one mutable state path.

Δℓj+r=α(1−α)rΔyj,r≥0\Delta\ell_{j+r}=\alpha(1-\alpha)^r\Delta y_j,\qquad r\geq0
The correction changes only existing y_j, with the same fixed α, same pre-j state, and r subsequent accepted observations unchanged. r counts updates, not elapsed seconds. The result does not cover an inserted extra observation or changed model parameters.↔ Scroll wide equations horizontally.

Commit state and input position together

Checkpoint the model identity, feature-definition identity, state schema, accepted input position, logical revision state, and numerical state together. If state commits without its offset, replay can double-apply inputs; if the offset commits first, recovery can skip updates. Use a transactional boundary or an idempotent recovery design that makes this relationship explicit. A message broker’s delivery guarantee alone does not prove the forecasting state is correct across application failures and external storage writes.

Preserve issued forecasts as immutable decision records even when later truth corrects the historical state. New forecasts should use the corrected replayed state according to policy, while earlier forecasts remain evidence of what was actually served. Monitor state age, replay backlog, duplicate suppression, revision frequency, and model-state compatibility failures. A state checksum can detect byte changes but cannot establish that the correct logical history was consumed; validate recovery by replaying a known fixture and comparing forecast outputs and input positions.

A duplicate and an older correction

Use α=0.25 and starting level 20. Accept observation y₁=28 once, then y₂=12. A duplicate of y₁ arrives, followed by a correction replacing y₁ with 24. Compute the original state and the corrected replay state without treating the duplicate as another observation.

  1. After y₁, level is 0.25×28+0.75×20=22. After y₂, it is 0.25×12+0.75×22=19.5. The duplicate is ignored because its logical version was already accepted.

  2. For corrected replay, restore level 20 before y₁. Replacing 28 by 24 gives level 0.25×24+0.75×20=21; processing unchanged y₂=12 gives 3+0.75×21=18.75.

  3. The correction changes y₁ by −4 and has one subsequent update. The analytic difference is 0.25×0.75×(−4)=−0.75, agreeing with 18.75−19.5.

The result

Deduplication leaves the sequence length unchanged, and replacement replay changes the earlier value at its original position. Appending 24 as a new latest measurement would compute a different, incorrect history.

Explore the model

Interactive lab

Interactive lab · optimization

Recomputation is different from a continued update history

Reset, set Step size η to 0.5 and Initial position x₀ to 2. After two Steps, predict x₂=0.5. Change x₀ to 1 without Reset: the retained n=2 recomputes x₂=0.25 from the new initial value, rather than applying one new step to 0.5. For an independent run, Reset, set η=0.5 and x₀=1, then take two Steps. This quadratic model illustrates deterministic recomputation; it is not an exponential-smoothing or streaming checkpoint implementation.

Recomputation is different from a continued update historyGradient descent on f(x) = x²/2: xₙ = (1 − η)ⁿx₀. Playback advances two discrete iterations per second. Values outside |x| ≤ 4 are explicitly reported. Current readings: Position x0: 2; Objective x²/2: 2; Iteration behavior: Stable step. All parameters are adjustable with the labeled controls below.−4−202402.174.356.528.7Parameter xf(x) = x²/2n = 0
ObjectiveDiscrete iterate
Position x0
2
Objective x²/2
2
Iteration behavior
Stable step
n = 0 · t = 0 s

Gradient descent on f(x) = x²/2: xₙ = (1 − η)ⁿx₀. Playback advances two discrete iterations per second. Values outside |x| ≤ 4 are explicitly reported. Parameter changes pause playback and recompute the model at the current elapsed time. Reset restores both parameters and returns time to zero. Playback starts paused. Adjust either parameter, then choose Play or Step.

Look forA state value is meaningful only together with the history and parameters used to compute it.

Check your understanding

How should the already accepted duplicate of y₁ be handled in the example?

Try it yourself

A fixed smoother uses α=0.5. An existing observation is corrected upward by 8 units, and two subsequent accepted observations are unchanged. What is the correction to the current level? Would the same expression cover inserting an extra observation?

Take a moment to make a prediction or work it out on paper. Open a hint when you need one.

Give me a hint

Use α(1−α)ʳΔy with r=2, then inspect the stated fixed-sequence assumption.

Compare with the solution

The level correction is 0.5×0.5²×8=1 unit. The formula does not cover inserting an extra observation because insertion changes the sequence length and the weights applied to later values. Replay the intended logical sequence or derive a separate valid insertion update.

Take it with you

  • Separate changing filtered state from changing learned parameters.
  • Deduplicate redelivery and replay historical replacements correctly.
  • Recover model-compatible state and its consumed input position together.

Read further

Original sources and deeper explanations to follow at your own pace.

Optional progress, saved in this browser when available. Revisit any lesson at any time.

Lesson not marked complete.