What is Kafka Connect and when would you use it?
Learn what Kafka Connect is, how source and sink connectors work, standalone vs distributed mode, and when to use it for data integration pipelines.
Expected Interview Answer
Kafka Connect is a framework and runtime for streaming data between Apache Kafka and external systems using reusable, configuration-driven connectors instead of hand-written producer or consumer code.
It runs source connectors that pull data from systems like databases, files, or APIs into Kafka topics, and sink connectors that push data from topics into targets like Elasticsearch, S3, or a data warehouse. Connect handles scaling, offset tracking, fault tolerance, and delivery guarantees for you, and can run in standalone mode for testing or distributed mode across a cluster of workers. You use it whenever you need reliable, repeatable integration pipelines without writing and maintaining custom glue code.
- No custom producer or consumer code to maintain
- Built-in scaling and fault tolerance via distributed workers
- Automatic offset management and restart from last position
- Large ecosystem of ready-made connectors
- Single Message Transforms for lightweight in-flight tweaks
AI Mentor Explanation
Kafka Connect is like the standardized boundary rope and sight-screen kit a ground crew rolls out for every match: instead of each groundsman improvising fittings, they clip in a pre-approved connector that just works. A source connector is the crew feeding balls into the nets, a sink connector stores used balls back in the kit bag, and the framework tracks exactly which over you reached so play resumes cleanly after rain.
Step-by-Step Explanation
Step 1
Choose the connector
Pick an existing source or sink connector for your system (JDBC, Debezium, S3, Elasticsearch) rather than writing code.
Step 2
Configure it
Supply a JSON or properties config: class, topics, connection details, key/value converters, and any transforms.
Step 3
Pick a mode
Use standalone mode for local testing, or distributed mode where connectors and tasks spread across a worker cluster.
Step 4
Deploy via REST
POST the config to the Connect REST API; Connect creates tasks that do the actual data movement and rebalances them across workers.
Step 5
Let it manage offsets
Connect stores source offsets and consumer positions so it can restart from the last committed point after failures.
Step 6
Monitor and scale
Track task status and lag, and add workers or increase task count to scale throughput horizontally.
What Interviewer Expects
- Clear distinction between source and sink connectors
- Understanding of standalone vs distributed mode
- Awareness that Connect handles offsets, scaling, and fault tolerance
- Knowledge of converters and Single Message Transforms
- A concrete use case such as CDC from a database into Kafka
Common Mistakes
- Confusing Kafka Connect with Kafka Streams (integration vs processing)
- Writing custom producers or consumers when a connector already exists
- Ignoring converter configuration and hitting serialization errors
- Running standalone mode in production instead of distributed mode
- Assuming Connect can do complex joins or aggregations
Best Answer (HR Friendly)
“Kafka Connect is a ready-made tool that moves data in and out of Kafka using pre-built connectors, so teams do not have to write and maintain custom code. You just configure a connector to pull from a database or push to storage, and it handles the scaling and reliability for you.”
Code Example
{
"name": "orders-source",
"config": {
"connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector",
"connection.url": "jdbc:postgresql://db:5432/shop",
"mode": "incrementing",
"incrementing.column.name": "id",
"topic.prefix": "pg-",
"tasks.max": "3"
}
}Follow-up Questions
- What is the difference between standalone and distributed mode?
- How does Kafka Connect achieve fault tolerance and exactly-once semantics?
- What are Single Message Transforms and when should you use them?
- How would you implement change data capture with Debezium and Connect?
- What role do converters play in a connector configuration?
MCQ Practice
1. Which connector type moves data FROM an external system INTO Kafka?
A source connector reads from an external system and writes into Kafka topics; a sink connector does the reverse.
2. In which mode do connectors and tasks distribute across a cluster of workers?
Distributed mode spreads tasks across worker nodes and provides fault tolerance and rebalancing; standalone runs in a single process.
3. What handles small in-flight modifications like renaming a field in Kafka Connect?
Single Message Transforms (SMTs) apply lightweight per-record changes such as masking, routing, or field renaming inside the connector.
Flash Cards
What is Kafka Connect? — A configuration-driven framework for streaming data between Kafka and external systems via reusable connectors.
Source vs sink connector? — Source pulls external data into Kafka topics; sink pushes topic data out to external targets.
Standalone vs distributed mode? — Standalone runs in one process for testing; distributed spreads tasks across workers with fault tolerance.
What is an SMT? — A Single Message Transform applies lightweight per-record changes like masking or renaming inside a connector.