Sqoop and Flume

Analogy categoryAnalogies

Moving Data In and Out of Hadoop

Sqoop and Flume solve two very different data-movement problems in the Hadoop ecosystem: Sqoop bulk-transfers structured data between relational databases and HDFS, Hive, or HBase using JDBC and a MapReduce job that runs in parallel across multiple mappers, while Flume continuously ingests unstructured or semi-structured streaming data, most commonly application log files, into HDFS or HBase using a lightweight agent-based pipeline of sources, channels, and sinks. Choosing between them is a matter of shape and cadence, a one-time or scheduled batch pull from a table with a known schema calls for Sqoop, while a continuous stream of events with no fixed end calls for Flume.

🏏

Cricket analogy: Sqoop is like scheduling a bulk data transfer of an entire season's completed scorecards from a board's archive into a stats platform, while Flume is like a live feed continuously streaming ball-by-ball commentary data as a match unfolds in real time.

Sqoop: Bulk Transfer Between RDBMS and HDFS

A Sqoop import connects to a source database over JDBC using a supplied connection string, username, and password, then splits the source table into roughly equal ranges using a numeric split-by column (the primary key by default) so that multiple mapper tasks, controlled by the -m flag, can pull rows in parallel, each writing its own output file into the destination HDFS directory or directly into a Hive table via --hive-import; Sqoop can also run in the reverse direction with sqoop export, pushing data from HDFS back into a relational table, useful for publishing an aggregate computed in Hive back to an operational reporting database.

🏏

Cricket analogy: Splitting a table by a numeric split-by column is like dividing a full ODI's ball-by-ball data among four separate scorers, each covering a contiguous block of overs in parallel, rather than one scorer transcribing all 300 balls sequentially.

bash

sqoop import \
  --connect jdbc:mysql://db.internal:3306/orders \
  --username analytics_ro --password-file /user/etl/.mysql.pass \
  --table customer_orders \
  --split-by order_id \
  -m 8 \
  --hive-import --hive-table warehouse.customer_orders \
  --as-parquetfile

sqoop job --create incr_orders -- import \
  --connect jdbc:mysql://db.internal:3306/orders \
  --table customer_orders \
  --incremental append \
  --check-column order_id \
  --last-value 0

Sqoop Incremental Imports

Because re-importing an entire table every night doesn't scale as a source grows, Sqoop supports incremental imports in two modes: append, which only pulls rows whose split-by key (typically an auto-incrementing ID) is greater than the value stored from the last run, and lastmodified, which pulls rows whose timestamp column has changed since the last run, capturing updates as well as new inserts; Sqoop persists this last-value bookmark automatically when run as a saved job via sqoop job --create, so a scheduled cron invocation only needs to trigger the saved job rather than manually tracking watermark state itself.

🏏

Cricket analogy: Incremental append is like a stats feed pulling only matches with a match_id higher than the last one it already ingested, rather than re-downloading the entire tournament archive every single night, saving the pipeline from redundant work.

Sqoop's --hive-import writes directly into a Hive table but still runs as a two-step process under the hood: import to a staging directory in HDFS, then load into the Hive table, so a failed load step can leave orphaned staging files that need manual cleanup.

Flume: Streaming Log Ingestion

A Flume deployment is built from agents, each a single JVM process combining a Source that consumes incoming data such as tailing a log file or listening on a network port, a Channel that buffers events durably (a file channel persists to local disk for durability, a memory channel is faster but loses buffered events on a crash), and a Sink that writes events onward to a destination like HDFS or HBase, and agents can be chained together so one agent's sink feeds the next agent's source, forming a multi-hop pipeline that fans in log data from many application servers toward a central HDFS cluster. Flume's transactional handoff between source, channel, and sink guarantees at-least-once delivery, so a downstream consumer must tolerate occasional duplicate events, typically handled by making the final write idempotent.

🏏

Cricket analogy: A Flume agent is like a single relay station in a chain carrying live match data from a stadium to a central broadcast hub, each station buffering incoming footage on local storage before forwarding it, so no signal is lost even if one hop briefly stalls.

properties

agent1.sources = weblog
agent1.channels = fileChannel
agent1.sinks = hdfsSink

agent1.sources.weblog.type = exec
agent1.sources.weblog.command = tail -F /var/log/app/access.log

agent1.channels.fileChannel.type = file
agent1.channels.fileChannel.checkpointDir = /flume/checkpoint
agent1.channels.fileChannel.dataDirs = /flume/data

agent1.sinks.hdfsSink.type = hdfs
agent1.sinks.hdfsSink.hdfs.path = /raw/weblogs/%y-%m-%d
agent1.sinks.hdfsSink.hdfs.fileType = DataStream
agent1.sinks.hdfsSink.hdfs.rollInterval = 300

agent1.sources.weblog.channels = fileChannel
agent1.sinks.hdfsSink.channel = fileChannel

Both Sqoop and Flume are considered mature-but-legacy tools in newer Hadoop deployments: Sqoop's last major release predates its move to the Apache Attic, and many teams now use Debezium or Kafka Connect for change-data-capture and streaming ingestion instead. They remain common in existing production pipelines, so understanding them is still essential for maintaining legacy Hadoop clusters.

Sqoop bulk-transfers structured data between relational databases and HDFS/Hive/HBase over JDBC, using parallel mappers.
The --split-by column (default: primary key) determines how Sqoop divides a source table across parallel mapper tasks.
sqoop export reverses the direction, pushing HDFS/Hive data back into a relational table.
Incremental imports (append or lastmodified) avoid re-pulling an entire table every run by tracking a last-value watermark.
Flume agents combine a Source, a durable Channel (file or memory), and a Sink to continuously move streaming data.
Flume's transactional handoff guarantees at-least-once delivery, so downstream consumers must handle possible duplicates.
Multiple Flume agents can be chained into a multi-hop pipeline that fans in logs from many servers toward central HDFS.