Rescaling moves stable key groups rather than inventing new histories

Flink 2.1 uses key groups as atomic redistribution units for keyed state, with their count determined by maximum parallelism. A key's group remains stable under a compatible key-selection and hashing setup, while ownership of groups changes with actual parallelism. The maximum-parallelism choice therefore influences future scaling and snapshot compatibility; it is not merely a display setting. Keep key types, equality, hash behavior, serializers, and operator identity compatible with the saved state. Increasing worker count cannot compensate for a changed key function that sends old and new records into different histories.

Key-group redistribution differs from operator-state redistribution. The latter follows its declared list or union semantics and can have different duplication and ownership behavior. Do not describe every saved source partition assignment or broadcast structure as keyed state. A rescale plan needs the relevant state type, its restore rules, and the connector's split ownership contract. Validate a checkpoint or savepoint intended for the operation rather than assuming any snapshot mode supports every topology change under all versions. The primary documentation and tested upgrade path determine the actual supported procedure.

One group can contain a hot key or much larger values than another. Balanced counts of key groups therefore do not guarantee balanced event rates, CPU cost, resident bytes, timers, or checkpoint work. A single hot key remains a serial aggregation boundary under the ordinary keyed model. Splitting it requires an application transformation with a valid combine or sequencing argument, not merely assigning more workers. Inspect per-key or per-group skew and the state distribution before recommending a parallelism change from an aggregate utilization chart.

Physical rescale cost includes reading saved state, constructing backend structures, moving or fetching artifacts, reconnecting channels, and catching up source lag. A logical owner changes while an artifact may already be accessible in shared checkpoint storage; conversely, an unchanged logical owner may still need a full reload on a new machine. Count logical reassignment separately from network bytes. During the transition, external sink and transaction recovery rules continue to apply. A job reporting RUNNING does not by itself prove its final keyed state and visible outputs survived the rescale unchanged.

Build a small trace with known per-key totals, a consistent snapshot, a parallelism change, and additional events. Verify every key's final value and source position, including keys assigned to different owners. Include skew and state schema evolution cases. Stable operator identifiers help associate saved state with intended operators; silently dropping unmatched state can make a fast restart look successful while losing history. Record restore duration, catch-up lag, state distribution, and visible output invariants alongside throughput so the claimed scaling benefit includes correctness.

The modular lab checks coverage in a separate toy, not Flink redistribution. Reset, explicitly set BOTH m=8 and a=3, predict all eight residues before repeating, then Play or Step. Reset and re-set BOTH m=8 and a=2; predict only four. Playback uses n=floor(2t), and edits retain t/recompute from zero. The modulus remains eight in both runs; changing stride does not simulate changing worker parallelism. Use the worked owner formula for rescaling and this finite orbit only to ask whether a hypothetical traversal actually reaches every stable group.