Partitioning removes files; clustering sharpens what remains

A directory named day=2026-01-05 is useful only if the reader knows its partition convention and the data actually satisfies that declaration. Partition fields can be materialized metadata rather than repeated payload columns. A table format can instead record a transform such as a day derived from a timestamp or a hash bucket derived from an identifier. Translate the query through that transform conservatively: equality on a source ID may select one known hash bucket, while an arbitrary range on IDs generally does not select a contiguous range of hash buckets.

Time boundaries need particular care. A day partition derived under one timezone or instant interpretation is not automatically compatible with a query expressed under another. A range spanning midnight can touch two or more partitions even if it is short. Missing, malformed, or stale partition metadata must not silently remove files from consideration. Keep the residual row filter after pruning, because a selected partition can contain many nonmatching rows. An engine's ability to infer partition values from paths is a reader feature and convention, not an inherent guarantee of every Parquet file.

Partitioning by every high-cardinality request ID can create tiny files and expensive listings or metadata even if equality pruning looks attractive. Partitioning only by year may keep files large but retain too much data for common daily queries. Estimate the expected bytes and file count per active partition, including skewed tenants and late-arriving updates. A partition with one unusually large customer can dominate runtime while many others finish quickly. More partitions do not create parallel work proportional to their count if the data and tasks are severely unbalanced.

Within each partition, sorting or clustering by a frequent range key can tighten file or row-group min/max bounds. This complements partition pruning rather than replacing it. A daily partition might first exclude other days; a tenant or timestamp ordering can then reject irrelevant groups within that day. Sorting has a write and compaction cost, and one order cannot optimize every unrelated access pattern equally. Compare layouts with a workload of queries, maintenance, and ingestion, preserving identical answers instead of selecting the single query that makes a preferred layout look best.

A query can spend substantial time discovering files, reading manifests or footers, and scheduling tasks before useful data transfer begins. Table metadata can make candidate discovery cheaper than recursively listing a directory, but it introduces its own update and compatibility contracts. Repartitioning or compaction must publish the new file set consistently so readers do not see both old and replacement data or neither. Save old generations long enough for active snapshots and the retention protocol. A directory move alone is not an assumed atomic table transaction on an object store.

The cache lab demonstrates a narrow reuse effect when repeatedly consulting a small synthetic working set. Reset, set both C=8 and s=2, predict an eight-address period with eight hits in sixteen initial requests, then Play or Step. Reset and re-enter both C=4 and s=2, predict no hits as the same period exceeds capacity, then inspect. It starts empty before the initial sixteen requests and caps at 4,096. The addresses are not partitions or manifest entries from a real table, so the lab cannot choose partition keys or estimate listing latency.