Compact files without publishing duplicates or losing readers

Every file can introduce metadata discovery, footer reads, request setup, task scheduling, and decoder initialization. These costs can dominate very small payloads, especially across a remote storage interface. Combining files can amortize them and create more useful row groups. Yet overly large files can reduce task granularity, complicate balanced work, or weaken selective scans if clustering is lost. Choose target sizes from representative scans, available task parallelism, writer memory, and the table's partition boundaries rather than declaring one universal megabyte value optimal.

Compaction is not necessarily simple byte concatenation. Valid output requires new file metadata, page and chunk boundaries, encodings, and statistics. If it changes row order, keep every column aligned. If the table has deletes or other row-level change semantics, use the table engine's supported rewrite operation so those changes remain correctly applied. Preserve duplicates unless the table's explicit semantics call for deduplication; two identical rows can be legitimate observations. Row count alone is a useful check but cannot prove value equality, multiplicity, or correct treatment of deletes.

A table snapshot identifies the data-file set readers should use. A compactor can write new immutable output files, validate them, and propose a new metadata state that replaces selected old files with their outputs. Publication must use the table/catalog's atomic commit and concurrency checks so a concurrent writer's changes are not overwritten. Iceberg's specification separates immutable data/metadata files from the atomic metadata update. The exact commit mechanism depends on the catalog; neither listing a directory nor assuming object-store rename behaves like a local transaction establishes equivalent guarantees.

A failed or uncertain commit needs inspection through the table protocol, not an immediate deletion of anything whose name looks temporary. After success, older snapshots or active readers can still reference replaced files. Reclamation therefore follows retention, snapshot-expiration, and orphan-cleanup rules with enough safety for in-flight work. Deleting all files absent from the newest listing can destroy valid older views or an uncommitted writer's outputs. Keep identifiers and metadata evidence that distinguish unreferenced debris from files still protected by another valid table state.

Compare pre- and post-compaction query results against the same snapshot contents, including nulls, nested values, duplicates, and deleted records. Check output statistics, row groups, encodings, and compatibility before publication. Then measure scan setup, fetched bytes, decode CPU, peak memory, and end-to-end latency on a workload of queries. A sorted rewrite may improve one predicate while hurting another; a denser codec may reduce storage while increasing decode cost. Include temporary space, I/O contention with ingestion, and the expected lifetime over which benefits can accumulate.

The queue lab isolates maintenance pressure with equal-sized compaction jobs. Reset, set both arrival λ=4 and service μ=2, predict growth from the initial four jobs, then Play or Step. For a separate adequately provisioned case, Reset, re-enter both λ=2 and μ=3, predict drainage from four, then inspect. Slider edits retain elapsed time and recompute from the initial four; they do not switch the accumulated queue to new rates. This fluid model has no files, snapshots, concurrency checks, or storage-cost estimates; it only illustrates whether maintenance demand outruns completion.