Change Data Capture Cheat Sheet
Explains log-based versus trigger-based CDC approaches and shows how to configure Debezium with PostgreSQL logical replication for streaming changes.
2 PagesAdvancedMar 12, 2026
CDC Approaches
The main techniques for capturing row-level changes as they happen.
- Log-based CDC- Reads the database's transaction log (WAL, binlog, redo log) to capture every insert/update/delete with minimal impact on the source. Used by Debezium and AWS DMS.
- Trigger-based CDC- Database triggers write changes to a shadow/audit table on every DML statement. Simple to set up but adds write overhead to the source database.
- Query-based (polling)- Periodically queries rows using an updated_at timestamp or incrementing ID column. Misses hard deletes and adds detection latency.
- Snapshot + streaming- CDC tools first take a consistent initial snapshot of existing data, then switch to streaming log changes going forward.
- Outbox pattern- The application writes domain events to an outbox table in the same transaction as the business data; CDC streams that table to avoid dual-write inconsistency.
Debezium Connector Config
Register a Kafka Connect source connector that streams changes from PostgreSQL.
json
{ "name": "orders-connector", "config": { "connector.class": "io.debezium.connector.postgresql.PostgresConnector", "database.hostname": "postgres", "database.port": "5432", "database.user": "debezium", "database.password": "dbz", "database.dbname": "shop", "topic.prefix": "shop", "table.include.list": "public.orders,public.customers", "plugin.name": "pgoutput" }}
PostgreSQL Logical Replication Setup
Prerequisites Debezium needs on the source PostgreSQL database.
sql
-- postgresql.conf must have:-- wal_level = logical-- Create a publication for the tables you want to captureCREATE PUBLICATION orders_pub FOR TABLE orders, customers;-- Debezium creates a replication slot automatically, or create manually:SELECT pg_create_logical_replication_slot('orders_slot', 'pgoutput');-- Grant replication privilege to the CDC userALTER ROLE debezium WITH REPLICATION;
Register the Connector
Deploy the connector via the Kafka Connect REST API and check its status.
bash
curl -X POST http://localhost:8083/connectors \ -H "Content-Type: application/json" \ -d @orders-connector.json# Check connector and task statuscurl http://localhost:8083/connectors/orders-connector/status
Key Concepts
Terminology you'll run into when operating a CDC pipeline.
- Debezium- Open-source, Kafka Connect-based CDC platform with log-based connectors for Postgres, MySQL, MongoDB, SQL Server, and Oracle.
- Topic per table- Debezium publishes one Kafka topic per captured table by default (<topic.prefix>.<schema>.<table>), with each message carrying before/after row state.
- Replication slot- A PostgreSQL server-side object that retains WAL segments until the consumer has read them — prevents data loss but risks disk bloat if the consumer stalls.
- At-least-once delivery- Most CDC pipelines guarantee at-least-once delivery; downstream consumers must handle duplicate events idempotently.
- Schema evolution- CDC connectors typically emit schema change events so consumers can adapt automatically when columns are added or dropped upstream.
Flattening the Debezium Envelope
Use the ExtractNewRecordState SMT so downstream consumers get flat records instead of the raw before/after/source envelope.
json
{ "name": "orders-connector", "config": { "connector.class": "io.debezium.connector.postgresql.PostgresConnector", "database.hostname": "postgres", "topic.prefix": "shop", "table.include.list": "public.orders", "transforms": "unwrap", "transforms.unwrap.type": "io.debezium.transforms.ExtractNewRecordState", "transforms.unwrap.drop.tombstones": "false", "transforms.unwrap.delete.handling.mode": "rewrite", "transforms.unwrap.add.fields": "op,source.ts_ms" }}
MySQL Binlog Prerequisites
Server-side configuration MySQL needs before Debezium's MySQL connector can attach.
sql
-- my.cnf settings (require a restart)-- [mysqld]-- server-id = 223344-- log_bin = mysql-bin-- binlog_format = ROW-- binlog_row_image = FULL-- expire_logs_days = 10-- Grant the CDC user replication privilegesCREATE USER 'debezium'@'%' IDENTIFIED BY 'dbz';GRANT SELECT, RELOAD, SHOW DATABASES, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'debezium'@'%';FLUSH PRIVILEGES;
Triggering an Incremental Snapshot
Re-snapshot a table without stopping the connector by inserting a signal row Debezium watches for.
sql
-- Signal table must already be included in the connector's signal.data.collection configINSERT INTO shop.debezium_signal (id, type, data)VALUES ( 'resnapshot-orders-2026-07-21', 'execute-snapshot', '{"data-collections": ["public.orders"], "type": "incremental"}');-- The connector interleaves snapshot chunks with ongoing streaming, so live-- writes to the table are never blocked while the resnapshot runs.
Consuming Deletes and Tombstones
Kafka consumer logic for correctly handling Debezium delete events and the tombstone that follows them.
python
from confluent_kafka import Consumerconsumer = Consumer({"bootstrap.servers": "kafka:9092", "group.id": "orders-sink"})consumer.subscribe(["shop.public.orders"])while True: msg = consumer.poll(1.0) if msg is None: continue if msg.value() is None: # Tombstone: log compaction marker, safe to ignore in application logic continue event = msg.value() op = event.get("op") if isinstance(event, dict) else None if op == "d" or event is None: delete_from_sink(key=msg.key()) else: upsert_into_sink(event) # handles op in ('c', 'u', 'r') idempotently
Operational Failure Modes
Issues that only show up once a CDC pipeline is running in production at scale.
- Exactly-once vs. at-least-once- True exactly-once requires idempotent sinks (upsert on primary key) or transactional Kafka producers end-to-end; most pipelines settle for at-least-once plus idempotent consumers.
- Snapshot modes- Debezium supports initial (default), initial_only, no_data (streaming only), when_needed, and incremental — choose based on whether you need historical rows or just new changes.
- Heartbeat events- Configure heartbeat.interval.ms so the connector emits periodic events on low-traffic tables, advancing the WAL/binlog position and preventing replication slot bloat during idle periods.
- Schema registry compatibility- When using Avro/Protobuf with a schema registry, set a BACKWARD or FULL compatibility mode so adding/dropping source columns doesn't break existing consumers mid-stream.
- DDL changes mid-stream- Log-based CDC connectors parse DDL from the transaction log itself; large migrations (table rewrites, column type changes) can briefly pause capture while the connector reconciles the new schema.
- Connector restart semantics- Kafka Connect stores offsets in an internal topic; a connector restart resumes from the last committed WAL/binlog position, not from the beginning, so avoid deleting offset topics casually.
Pro Tip
Monitor replication slot lag closely — an idle or crashed CDC consumer leaves the WAL replication slot open, causing WAL files to accumulate on the primary until it runs out of disk space, even though the database itself looks perfectly healthy.
Was this cheat sheet helpful?
Previous
Vector Databases (Pinecone/Weaviate) Cheat Sheet
Next
Database Security Best Practices Cheat Sheet
Explore Topics
#ChangeDataCapture#ChangeDataCaptureCheatSheet#Database#Advanced#CDCApproaches#DebeziumConnectorConfig#PostgreSQL#Logical#Databases#CheatSheet#SkillVeris
