Source and Sink Connectors

Analogy categoryAnalogies

Source Connectors: Getting Data Into Kafka

A source connector's job is to detect changes in an external system and publish them as records into one or more Kafka topics, and the mechanism for detecting those changes varies widely by connector: the Debezium PostgreSQL connector reads the database's write-ahead log to capture every row-level insert, update, and delete as a change-data-capture (CDC) event, while the JDBC source connector instead periodically polls a table with a query, using either an incrementing column or a timestamp column to find new rows since the last poll. CDC-based connectors capture every change with low latency and no polling overhead, while polling-based connectors are simpler to reason about but can miss rapid intermediate updates and add query load to the source database.

🏏

Cricket analogy: CDC is like a stump microphone catching every single sound event in real time during play, while polling is like a scorer walking out to check the board every over, both work, but the mic misses nothing while the scorer might miss something that changed and reverted between checks.

Sink Connectors: Getting Data Out of Kafka

A sink connector consumes records from one or more Kafka topics and writes them into a target system, and the interesting design decisions are usually about how it handles delivery semantics and target-side schema. The Elasticsearch sink connector, for instance, can be configured to use a record's key as the document ID, which makes writes idempotent (replaying the same record just overwrites the same document rather than duplicating it), while the S3 sink connector instead batches records into files, flushing based on record count, time, or size thresholds, since object storage rewards fewer, larger files over many tiny ones.

🏏

Cricket analogy: Using a record key as a document ID for idempotency is like re-entering a run correction under the same ball number, it overwrites that specific ball's record instead of adding a phantom extra run to the total.

Delivery Semantics and Idempotency

Kafka Connect provides at-least-once delivery by default, meaning a worker crash or rebalance can cause some records to be reprocessed and written to the sink again, so a well-designed sink connector must handle duplicates gracefully, typically through idempotent writes keyed on a unique identifier, as opposed to blindly appending every record. Newer connector versions increasingly support exactly-once semantics for sinks (using Kafka's transactional consumer offset commits alongside the target write) or for sources (using the same transactional producer machinery Kafka Streams uses), but this support is connector-specific and must be explicitly verified rather than assumed.

🏏

Cricket analogy: At-least-once delivery is like a scorer re-announcing a boundary over the PA system if they're unsure the first announcement was heard, better to repeat than miss it, but the scoreboard operator must be smart enough not to add the runs twice.

json

{
  "name": "orders-es-sink",
  "config": {
    "connector.class": "io.confluent.connect.elasticsearch.ElasticsearchSinkConnector",
    "connection.url": "http://elasticsearch:9200",
    "topics": "shop.public.orders",
    "key.ignore": "false",
    "schema.ignore": "true",
    "behavior.on.null.values": "delete",
    "tasks.max": "3"
  }
}

Setting key.ignore to false tells the Elasticsearch sink to use each Kafka record's key as the document _id, which is what makes reprocessing after a rebalance safe: the same record just overwrites the same document.

Polling-based source connectors (like the plain JDBC source without CDC) can miss updates that happen and revert between poll intervals, and they add repeated query load to the source database; for high-change-rate tables, a log-based CDC connector is almost always the better choice.

Source connectors detect external changes and publish them into Kafka; the two dominant strategies are log-based CDC and periodic polling.
CDC connectors like Debezium read a database's write-ahead log for low-latency, complete change capture.
Polling connectors are simpler but can miss rapid intermediate changes and add query load.
Sink connectors write Kafka records to external systems, often batching for efficiency (as with S3).
Kafka Connect defaults to at-least-once delivery, so sink connectors need idempotent writes to handle duplicates.
Using a record's key as a target-system identifier (like an Elasticsearch document ID) is a common idempotency pattern.
Exactly-once support exists in some connectors but is connector-specific and must be verified, not assumed.