Row groups, column chunks, and pages solve different problems

A row group represents a horizontal subset of rows. For a flat schema, that subset has one column chunk for each column; for nested schemas, reason about the encoded leaf-column paths rather than assuming one chunk per top-level object. A column chunk holds that column's values and structural information for the row group in a contiguous file range. Consequently, reading two projected columns can avoid unrelated column chunks, while reading all rows of one column can require visiting its chunk in every selected row group.

Column chunks contain pages, which carry encoded data or a dictionary and have headers describing their contents. Encoding and compression operate at their defined page boundaries; a row group is a larger planning unit, not a single mandatory compression stream. File metadata records locations and schema information, allowing a reader to find the ranges it needs. A file's storage block size, object-store request size, Parquet page size, and engine vector batch size are distinct settings. Naming the layer prevents accidental tuning of a parameter that the actual bottleneck does not use.

Larger row groups can amortize metadata and scheduling overhead and give encoders a broader local population. Smaller row groups can expose more independent tasks and allow a selective predicate to discard smaller regions. Neither direction is universally better. A query that scans an entire dataset, a point lookup, and a memory-constrained training loader can prefer different shapes. Distribution matters too: if values are mixed uniformly through every group, reducing group size may still leave nearly every group relevant, while increasing footer entries and setup work.

An execution engine can stream decoded batches smaller than a row group rather than materializing every projected column at once. Yet dictionaries, decompression buffers, pending range requests, and concurrent tasks still consume memory. Conversely, an Arrow record batch is an in-memory collection of equal-length arrays; it is not a promise that its underlying Parquet row group has the same row count or byte layout. Build a peak-live-buffer budget for the chosen reader and concurrency. Do not derive memory consumption simply by multiplying the on-disk compressed file size by one guessed factor.

Start a scan investigation with a known schema, row-group count, selected columns, and metadata dump. Then observe requested byte ranges and decoded rows. A network client may coalesce nearby ranges or prefetch extra bytes, and metadata requests may dominate a small query. A logical ability to skip a column does not require a client to issue the smallest imaginable request. Keep logical payload, requested bytes, bytes delivered, decoded bytes, and output bytes as separate counters; they locate different opportunities for improvement.

Use the cache lab to isolate reuse, not to pretend it renders Parquet. Reset, set both capacity C to four and stride s to four, and predict twelve hits among the initial sixteen requests to four repeated addresses; then Play or Step. Reset and re-enter both C=2 and s=4 for a separate run, predicting no hits because the four-address cycle exceeds capacity. The cache starts empty before those sixteen simulated requests, adds two requests per second, and caps at 4,096. These synthetic addresses are not real row groups or file ranges.