Change Data Capture (CDC)
Change Data Capture (CDC) is a set of techniques for detecting and capturing row-level inserts, updates, and deletes in a source database as they happen, so those changes can be propagated to downstream systems in near real time.
Definition
Change Data Capture (CDC) is a set of techniques for detecting and capturing row-level inserts, updates, and deletes in a source database as they happen, so those changes can be propagated to downstream systems in near real time.
Overview
Traditional batch integration reloads entire tables on a schedule, which wastes compute and leaves downstream systems stale for hours. CDC instead watches a database's transaction log (or uses triggers, timestamps, or periodic diffing) and emits a stream of change events the moment a row is inserted, updated, or deleted, dramatically reducing latency and load. Log-based CDC is the most common production approach: tools like Debezium tail the write-ahead log of a source database and publish each change as an event, typically onto a message bus such as Kafka for downstream consumers to subscribe to. This decouples the source system from consumers and lets many downstream services react to the same stream of changes independently. CDC is a foundational building block for keeping a data warehouse or data lake in sync with operational databases without full reloads, for feeding ETL and ELT pipelines incrementally, for cache invalidation, for audit trails, and for driving database replication between heterogeneous systems. Modern data platforms like the Apache Kafka & Messaging course cover CDC as a core pattern for building event-driven architectures where every state change becomes a first-class, replayable event. Because CDC captures every intermediate state change rather than just a final snapshot, it also underpins data lineage tracking and enables patterns like event sourcing, where an application's state is reconstructed by replaying its full history of changes.
Key Concepts
- Log-based capture that reads database transaction logs without impacting source performance
- Near real-time propagation of inserts, updates, and deletes to downstream consumers
- Decouples source systems from consumers via an event stream, often built on a message bus
- Supports incremental loading into warehouses and lakes instead of full-table reloads
- Preserves ordering and, in many implementations, exactly-once or at-least-once delivery semantics
- Works across heterogeneous databases (MySQL, PostgreSQL, MongoDB, SQL Server, Oracle)
- Enables event sourcing and audit-trail patterns by capturing every intermediate state
- Can trigger downstream cache invalidation, search-index updates, or microservice notifications
Use Cases
Frequently Asked Questions
From the Blog
Career Change at 40: Is It Too Late to Switch to Tech?
A career change at 40 into tech is not too late; it simply requires a focused plan that leverages existing experience while filling specific skill gaps. This guide covers how to evaluate the move, choose a direction, and build momentum realistically.
Read More AI & TechnologyHow Delimiters and Section Order Change Prompt Accuracy
Clear boundaries between instruction, context and data reduce the two most common prompt failures: the model treating supplied data as commands, and instructions getting lost in long context. This explains why delimiters work, which ones to choose, and how section order changes what the model attends to.
Read More AI & TechnologyHow to Keep a RAG Index Fresh as Documents Change
Keep a RAG index fresh by detecting change at the source, upserting only affected chunks with stable identifiers, and propagating deletions as first-class events. This covers change detection, deterministic chunk IDs, tombstoning, reindex triggers and the monitoring that tells you when stale content is still being served.
Read More AI & TechnologyHow to Re-Embed a Corpus When You Change Models
Re-embed by writing the new vectors into a separate versioned collection, backfilling in batches while the old collection continues serving, then cutting over behind a config flag. This article covers the mixing hazard, batch and checkpoint design, dual-write during backfill, and how to validate before you switch.
Read More