What Is a DataStream?
A DataStream in Apache Flink represents a stream of data records of a specific type, flowing through a pipeline of operators. Unlike a static collection, a DataStream can be unbounded (a continuous stream, e.g. from a Kafka topic) or bounded (a finite stream, e.g. reading a file), and Flink's runtime treats both uniformly through the same DataStream API, applying identical checkpointing, scheduling, and fault-tolerance mechanisms regardless of boundedness.
Cricket analogy: A DataStream is like ball-by-ball commentary of a Test match — Sachin Tendulkar's innings arrives delivery by delivery rather than as one final scorecard, and Flink processes each ball the moment it happens instead of waiting for the innings to end.
Creating and Executing a DataStream Job
Every Flink DataStream program begins by obtaining a StreamExecutionEnvironment, which acts as the context for defining sources, transformations, and sinks. The program you write is only a logical dataflow graph until env.execute() is called, at which point Flink translates it into a JobGraph, submits it to the cluster, and only then does data actually begin flowing through the operators.
Cricket analogy: Writing DataStream code without calling execute() is like a captain setting a batting order and field placements before the toss — nothing actually happens on the pitch until the umpire signals play to begin.
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
DataStream<String> lines = env.socketTextStream("localhost", 9999);
DataStream<Integer> lineLengths = lines
.map(line -> line.length())
.returns(Types.INT);
lineLengths.print();
// Nothing above has actually run any data through the pipeline yet.
// The dataflow graph is only submitted and executed here:
env.execute("Line Length Job");Parallelism and Task Slots
Each operator in a DataStream job can run with its own parallelism, meaning Flink splits the operator into multiple parallel subtasks that each process a portion of the stream's partitions. These subtasks are scheduled onto task slots provided by TaskManagers, and Flink automatically chains adjacent operators with compatible parallelism into a single task to avoid serialization and network overhead between them.
Cricket analogy: Operator parallelism is like the IPL splitting group-stage matches across Wankhede, Chinnaswamy, and Eden Gardens simultaneously — each venue is a subtask handling its own slice of fixtures instead of one stadium hosting every match sequentially.
By default, Flink chains operators like map and filter that have the same parallelism into a single task to eliminate serialization and thread-switching costs. You can override this with .disableChaining() on an operator to isolate it, or .startNewChain() to begin a fresh chain boundary — useful when profiling to see per-operator metrics.
Time Characteristics
Flink DataStream jobs can reason about time using event time (the timestamp embedded in the record when it actually occurred) or processing time (the wall-clock time when the record happens to be handled by the operator). Event time requires watermarks to track progress and correctly handle out-of-order arrivals, while processing time is simpler to reason about but non-deterministic and unsuitable for reproducible analytics or exactly reconstructing history.
Cricket analogy: Event time is like recording exactly when Virat Kohli's six actually left the bat according to the stadium clock, while processing time is like recording whenever the TV broadcast happens to show the replay — the two can diverge if there's a delay.
Relying on processing time in production jobs makes results non-deterministic and non-reproducible: rerunning the same job over the same input can yield different aggregation boundaries because results depend on when records happen to arrive, not when they actually occurred. Prefer event time with watermarks whenever correctness matters, such as billing or fraud detection.
- A DataStream represents a sequence of records that may be unbounded or bounded, processed uniformly by Flink's runtime.
- Programs are lazily evaluated: nothing executes until StreamExecutionEnvironment.execute() is called.
- Operators run as parallel subtasks distributed across TaskManager task slots.
- Flink chains compatible adjacent operators into a single task to avoid unnecessary serialization overhead.
- Event time uses timestamps embedded in records and requires watermarks for correctness with out-of-order data.
- Processing time is simpler but non-deterministic, making it unsuitable for reproducible, correctness-critical pipelines.
- Choosing the right time characteristic is foundational to correct windowing and aggregation later in a Flink job.
Practice what you learned
1. What happens when you call methods like .map() on a DataStream before env.execute() is invoked?
2. Why does Flink chain compatible adjacent operators into a single task?
3. What is required to correctly handle out-of-order records when using event time?
4. Which statement about processing time is correct?
5. What are task slots used for in a Flink cluster?
Was this page helpful?
You May Also Like
Transformations: Map, Filter, FlatMap
How to use Flink's core stateless transformation operators — map, filter, and flatMap — to reshape and clean streaming data.
Keyed Streams and keyBy
Partitioning a DataStream by key using keyBy() to enable per-key state and correct windowed aggregations.
Connectors: Sources and Sinks
An overview of Flink's connector ecosystem for reading from and writing to external systems like Kafka, files, and databases.
Related Reading
Related Study Notes in Programming
Browse all study notesApache Spark Study Notes
Programming · 30 topics
ProgrammingHadoop Study Notes
Programming · 30 topics
ProgrammingSnowflake Study Notes
Programming · 30 topics
ProgrammingApache Airflow Study Notes
Programming · 30 topics
Programmingdbt (Data Build Tool) Study Notes
Programming · 30 topics
ProgrammingPowerShell Study Notes
Programming · 30 topics