Lakehouse transactions and tables

A table snapshot identifies the data and metadata needed to read one committed version. Writers can prepare immutable data files before publishing their membership in the table. In Iceberg, metadata and manifest structures describe snapshots and their files; the catalog's atomic update protocol determines the current metadata reference. A reader pins a snapshot instead of inferring membership from every object it happens to find under a path. Unreferenced objects from failed attempts are not automatically visible rows.

This does not mean all formats use one identical commit implementation. Delta Lake uses its transaction log to define committed table versions, while catalogs and storage systems impose specific requirements. Use a supported engine and compatible protocol features. Raw-file scans can ignore deletes and include obsolete files. Multi-table atomicity is a separate feature: one table's atomic commit does not by itself give a transaction over every table in a pipeline.

Optimistic concurrency lets a writer read a table version, prepare output, and then attempt a commit. If another writer changes the table, the system validates whether the operation can safely apply to the newer state or must retry. Two independent appends may be compatible under the format's rules. A rewrite or update based on removed input files can conflict with another operation that modifies those inputs. Retrying a metadata publication is different from blindly repeating a transformation computed from stale rows.

Deletes can be represented through rewritten data files or format-supported deletion metadata. Iceberg's position deletes identify rows through file paths and positions, while equality deletes identify values using declared field identities under applicable rules. Sequence and snapshot metadata determine applicability; an engine cannot ignore them and remain correct. Schema evolution also needs stable field identity so a rename does not accidentally reinterpret a different column. Verify that every reader supports the table's features before enabling new write behavior.

Small files increase metadata, planning, and request overhead even when total data bytes are modest. Compaction rewrites them into fewer files and commits a new snapshot; old files may remain necessary for retained snapshots or active readers. An object being absent from the latest snapshot is therefore insufficient evidence that it can be immediately deleted. Expiration and orphan cleanup must respect retention, reader behavior, and in-progress writes, with safety intervals that match operational reality.

Time travel needs both metadata and the referenced data objects. Premature object-lifecycle deletion can leave named snapshots unreadable. Storage versioning helps recovery without replacing format-aware retention. Measure planning time, file counts, delete-file work, and scan bytes before choosing compaction frequency. Observe maintenance and test long-running readers during rewrite and cleanup.