Change data capture

An initial snapshot establishes the existing rows at a consistent source view. The change stream must continue from a position coordinated with that snapshot, retaining all changes needed while the scan runs. Simply copying rows and then asking for changes from now creates a gap. Starting log reading before the scan without reconciling the overlap can instead duplicate or overwrite newer state. A connector's documented snapshot algorithm defines how it bridges these intervals; implement that contract rather than inventing a timestamp approximation.

PostgreSQL logical decoding translates WAL into a logical change stream through an output plugin. Replication slots retain resources for consumers but are not unlimited archives. Table identity and replica-identity configuration affect what information is available for updates and deletes. A stable primary key makes downstream addressing simpler. A key change may require removing the old projection identity.

A connector may replay changes after failure when the last downstream progress record was not durable. Use a stable event identity or a source ordering token together with the entity key, and commit projection effects with their applied progress where possible. An idempotent upsert prevents a repeated assignment from duplicating a row, but an increment operation is not idempotent. A delete also needs replay handling so a delayed older update cannot resurrect an entity after deletion.

If multiple workers process one key out of order, a last-write-wins policy needs a valid comparable source version, not the consumer's arrival clock. Source positions may be ordered only within a partition or source timeline. Preserve source transaction boundaries where a projection requires atomic visibility of related changes, or explicitly expose eventual intermediate states. Do not claim a global transaction order after splitting tables into independent topic partitions without an additional coordinating protocol.

A stopped consumer can cause the source to retain WAL, increasing disk pressure even while the application database continues operating. Monitor source retention bytes, connector lag, destination freshness, and failed records as separate signals. A configurable retention cap may protect the database at the cost of invalidating the consumer's recovery position. If required history is gone, restarting from an arbitrary current position silently loses changes; re-snapshot through a consistent rebuild procedure instead.

Projection rebuilds should have a named generation. Load a new generation, replay to a chosen freshness point, validate counts and representative records, and atomically switch readers when the destination supports it. Keep source-to-destination reconciliation independent of the transport's health check. A flowing stream can still drop fields or ignore deletes. Test duplicate changes, source failover, snapshot interruption, and schema evolution.