Engineering / advanced
Columnar storage and analytical scans
Connect encoding, pruning, partition layout, and vectorized scans to the physical costs of analytical data systems.
Your route through engineering
Follow the path or begin with a question. Every lesson is open.
Your place, at your pace. Progress is optional and saved only in this browser when available.
Read / explore / reason
Inside this module
Module 01 advanced
Columnar storage and analytical scans
Connect encoding, pruning, partition layout, and vectorized scans to the physical costs of analytical data systems.
- 01
Row groups, column chunks, and pages solve different problems
Trace a projected scan through Parquet's physical hierarchy without confusing row partitions, column ranges, encoding units, or in-memory batches.
25 min - 02
Dictionary IDs and runs exploit different repetition
Estimate dictionary costs, distinguish cardinality from adjacency, and handle mixed dictionary/plain pages within a column chunk.
26 min - 03
Small residuals, fixed-width arithmetic, and bit packing
Derive delta residual widths, handle specified wraparound correctly, and distinguish a teaching sequence from a valid format block.
28 min - 04
Use bounds to prove a region cannot match
Apply conservative min/max pruning with correct null and ordering semantics while distinguishing overlap from evidence that matching rows exist.
27 min - 05
Project the columns the whole plan needs
Calculate required column unions, distinguish output selectivity from physical reads, and reason about page-level late materialization.
26 min - 06
Partitioning removes files; clustering sharpens what remains
Separate partition transforms, directory conventions, and file statistics while choosing layout from representative query patterns and skew.
27 min - 07
A smaller file can demand a slower scan
Compare compressed transfer, decompression, value decoding, and memory movement with explicit byte domains and overlap assumptions.
28 min - 08
Preserve nulls, empty lists, and repetition without row objects
Trace definition/repetition levels and contrast them with Arrow offsets and validity while preserving nested semantics exactly.
30 min - 09
Compact files without publishing duplicates or losing readers
Estimate metadata amortization and rewrite costs while preserving snapshot visibility, row semantics, and safe reclamation of replaced files.
29 min - 10
Evolve meaning as carefully as bytes
Separate physical carriers, logical types, schema reconciliation, and feature support when introducing new analytical file writers and readers.
29 min