Dictionary IDs and runs exploit different repetition

A dictionary stores distinct values once and replaces occurrences with integer IDs. With K entries, an ideal fixed-width representation needs enough bits to distinguish IDs zero through K minus one. The dictionary itself still costs bytes, and variable-length entries need lengths or other representation metadata. For short columns or nearly unique values, the dictionary plus IDs can cost more than directly encoded values. Evaluate cardinality over the actual dictionary scope rather than the entire table, since separate chunks can have different vocabularies and distributions.

Parquet dictionary pages store the values, while dictionary-encoded data pages store IDs using the specified hybrid RLE/bit-packing representation. At most one dictionary page occurs in a column chunk and it precedes its data pages. A writer may switch later pages to a different permitted encoding when dictionary growth becomes unattractive, so one chunk can contain both dictionary and plain pages. A reader must obey each page's encoding and its chunk's dictionary. The same numeric ID in another chunk need not mean the same category, which matters for joins and aggregations performed directly over IDs.

Consider IDs alternating zero, one, zero, one. The vocabulary is tiny, but adjacent equal runs are short. A block containing ten thousand zeros followed by ten thousand ones has the same two IDs and much better run structure. Hybrid encoding can choose runs for repetition and bit-packed literals for irregular stretches. Parquet's hybrid RLE is used for its supported streams, including dictionary indices and structural levels; it is not a blanket promise that arbitrary physical integer payload uses that encoding. ORC has its own type-specific RLE variants and framing rules.

Sorting can improve runs and statistics, but rearranging data has a cost and may alter useful locality on another key. Preserve row alignment across columns when reordering, and account for downstream consumers that require a particular order. Dictionary decoding can also involve indirection and cache access; reducing file bytes does not imply every downstream operator becomes faster. Some engines compare encoded IDs efficiently when dictionaries are compatible, while others materialize values. Compare the full required operation, including dictionary reconciliation or decoding, rather than benchmarking a code stream in isolation.

Inspect per-page encoding metadata, dictionary size, fallback points, and decoded equality results on representative data. Include high-cardinality and changing-cardinality groups, not only a small categorical sample. A metadata list of encodings used somewhere in a chunk does not establish that every page uses dictionary IDs. Verify null handling separately because structural validity is not an ordinary dictionary entry for a nonnull value. Compatibility tests should include mixed-encoding chunks so a reader cannot accidentally pass by assuming the writer's most common path is the only path.

The cache lab illustrates lookup working sets only. Reset, set both C=4 and s=4, predict a four-address cycle with twelve hits in sixteen requests, then Play or Step. Reset and set both C=2 and s=4, predict misses throughout that cycle, then Play or Step. Each Reset starts an empty cache followed by sixteen modeled requests; controls do not specify dictionary cardinality, encoded byte width, or actual Parquet addresses. The lesson is that a lookup table's hot working set can matter independently of how few bits its IDs occupy.