What Kafka Connect Solves
Kafka Connect is a framework for reliably streaming data between Kafka and external systems, such as databases, object stores, or search indexes, without writing bespoke producer or consumer code for every integration. Instead of every team hand-rolling a JDBC poller or an S3 uploader from scratch, Connect provides a standard runtime, REST API, and plugin model, so integrations become configuration (a JSON connector config) rather than custom application code that each needs its own deployment, monitoring, and error handling.
Cricket analogy: It is like a standardized DRS review system used across every stadium instead of each ground inventing its own ad hoc replay process, so umpires everywhere follow the same protocol rather than reinventing it per venue.
Connectors, Tasks, and Workers
A connector is a logical job definition, configured via JSON, describing what to integrate (for example, 'stream the orders table from this Postgres database'). Internally, Connect splits that job into one or more tasks, the actual units of parallel work, which are then scheduled onto worker processes; in distributed mode, multiple worker JVMs form a cluster that automatically rebalances tasks if a worker joins, leaves, or crashes, giving Connect the same elastic scaling and fault tolerance that consumer groups provide for regular Kafka consumers.
Cricket analogy: A connector is like a tour itinerary defining which matches will be played, while tasks are like the individual innings assigned to specific grounds, and workers are like the venues themselves, which can be added or lose availability without cancelling the whole tour.
Source vs. Sink Connectors, and Standalone vs. Distributed Mode
Connectors come in two directions: source connectors pull data from an external system into Kafka topics, while sink connectors push data from Kafka topics out to an external system, and a single Connect cluster can run both kinds simultaneously for different integrations. Standalone mode runs a single worker process reading connector configs from local files, which is fine for local development or single-node edge deployments, but production systems almost always use distributed mode, where multiple workers coordinate through internal Kafka topics for config storage, offset tracking, and status, giving you high availability and horizontal scalability.
Cricket analogy: A source connector is like a stadium's ball-tracking sensor feeding data into the broadcast system, while a sink connector is like that broadcast system pushing highlights out to a streaming app, both running from the same production truck.
# Register a connector via the Connect REST API in distributed mode
curl -X POST http://localhost:8083/connectors \
-H "Content-Type: application/json" \
-d '{
"name": "orders-source",
"config": {
"connector.class": "io.debezium.connector.postgresql.PostgresConnector",
"database.hostname": "localhost",
"database.dbname": "shop",
"table.include.list": "public.orders",
"topic.prefix": "shop",
"tasks.max": "1"
}
}'
# Check connector and task status
curl http://localhost:8083/connectors/orders-source/status
The Connect REST API also exposes /connectors/{name}/pause, /resume, and /restart endpoints, letting you manage running connectors without redeploying anything, since connector configs themselves live inside an internal Kafka topic in distributed mode.
Standalone mode stores connector offsets in a local file, which is fine for a single always-up worker but means offsets are lost if that worker's disk is wiped, distributed mode avoids this by storing offsets in an internal, replicated Kafka topic.
- Kafka Connect turns data integration into configuration rather than custom producer/consumer code.
- A connector is a logical job; it is split into tasks, the units of parallel work.
- Workers run tasks and, in distributed mode, form a cluster that rebalances automatically on failure.
- Source connectors pull external data into Kafka; sink connectors push Kafka data to external systems.
- Standalone mode is single-process and file-based; distributed mode is clustered and stores state in internal Kafka topics.
- The REST API lets you create, pause, resume, and inspect connectors without redeploying workers.
Practice what you learned
1. What is the relationship between a connector and a task in Kafka Connect?
2. What is the main advantage of distributed mode over standalone mode in Kafka Connect?
3. Which of these is an example of a sink connector's job?
4. Where does standalone mode store connector offsets, and why does this matter?
Was this page helpful?
You May Also Like
Source and Sink Connectors
How source connectors capture external changes into Kafka and sink connectors deliver Kafka data to external systems, including delivery semantics.
Schema Registry and Avro
How the Schema Registry and Avro serialization give Kafka's opaque byte messages an enforced, evolvable structure across producers and consumers.
Kafka Streams Basics
An introduction to Kafka Streams, the client library for building stateful stream processing applications directly on top of Kafka topics.
Related Reading
Related Study Notes in DevOps
Browse all study notesNginx Study Notes
DevOps · 30 topics
DevOpsAnsible Study Notes
DevOps · 30 topics
DevOpsAdvanced Kubernetes Study Notes
Kubernetes · 30 topics
DevOpsAdvanced Bash Scripting Study Notes
Bash · 30 topics
DevOpsDocker & Kubernetes Study Notes
YAML · 40 topics
DevOpsCI/CD Tools & Pipelines Study Notes
YAML · 37 topics