The Map Transformation
The map() transformation applies a one-to-one function to every element of a DataStream, transforming a value of type A into exactly one value of type B. It is the workhorse for tasks like parsing a raw JSON string into a strongly-typed POJO, converting units, or extracting a single field, and because it always emits exactly one output per input, Flink can reason precisely about downstream record counts.
Cricket analogy: map() is like a scorer converting every raw ball-by-ball note into a formatted entry in the official scorebook — one input event produces exactly one transformed output, never more or fewer.
Filtering Elements
The filter() transformation evaluates a boolean predicate against each element and keeps only those elements for which the predicate returns true, discarding the rest. It is commonly used to remove malformed records, exclude records outside a valid range, or narrow a stream to a subset relevant to downstream logic, and unlike map(), the number of output records is always less than or equal to the number of inputs.
Cricket analogy: filter() is like a selection committee keeping only players with a batting average above 40 for the World Cup squad, discarding everyone else from the wider pool of candidates.
DataStream<String> rawLogs = env.socketTextStream("localhost", 9999);
DataStream<String> cleaned = rawLogs
.filter(line -> !line.isBlank()) // drop empty lines
.flatMap((String line, Collector<String> out) -> {
for (String word : line.split("\\s+")) {
if (!word.isEmpty()) {
out.collect(word.toLowerCase()); // one-to-many emission
}
}
})
.returns(Types.STRING)
.map(word -> word.trim()); // one-to-one cleanup
cleaned.print();
env.execute("Tokenize And Clean");FlatMap for One-to-Many
The flatMap() transformation is the most general of the three: for each input element it can emit zero, one, or many output elements by calling collect() on the provided Collector any number of times. This makes it ideal for tokenizing a sentence into individual words, splitting a nested record into multiple flat records, or combining filtering and mapping logic in a single operator by simply choosing not to call collect() for records that should be dropped.
Cricket analogy: flatMap() is like a highlights editor taking one full over of footage and producing anywhere from zero clips (a maiden over) to six clips (one per boundary) depending on what happened each ball.
flatMap() is implemented via the FlatMapFunction interface, whose flatMap(T value, Collector<O> out) method receives an output Collector rather than returning a value directly. Calling out.collect() zero times effectively acts as a filter, calling it once acts like map(), and calling it multiple times performs a one-to-many expansion — making flatMap() strictly more general than the other two.
Choosing Between Them
In practice, choose map() when the transformation is a pure one-to-one conversion with no possibility of dropping records, choose filter() when you only need to include or exclude records without changing their shape, and reach for flatMap() when the cardinality of output records genuinely varies per input, such as splitting, exploding, or conditionally emitting. Using flatMap() everywhere works functionally but obscures intent and can make the dataflow graph harder to reason about during debugging.
Cricket analogy: Choosing the right operator is like a captain deciding between a specialist bowler for a single job (map), a fielder who simply stops certain balls (filter), and an all-rounder who can bowl, field, and bat depending on the situation (flatMap).
map(), filter(), and flatMap() are stateless by default, meaning each call only has access to the current record, not previous ones. If your logic needs to remember information across records — such as deduplicating or counting — you need a RichFlatMapFunction or a keyed stream with managed state, not a plain lambda; otherwise your transformation will silently behave incorrectly under parallelism.
- map() performs a strict one-to-one transformation from input type A to output type B.
- filter() keeps only elements matching a boolean predicate, always producing fewer or equal outputs.
- flatMap() is the most general: it can emit zero, one, or many outputs per input via a Collector.
- flatMap() can subsume both map() and filter() behavior, but using the specific operator communicates intent.
- All three are stateless by default and only see the current record being processed.
- Use RichFlatMapFunction with managed state when logic needs to remember information across records.
- Choosing the right transformation improves both readability and the clarity of the resulting dataflow graph.
Practice what you learned
1. What is the defining characteristic of the map() transformation?
2. How does flatMap() emit output records?
3. Which statement correctly describes filter()'s output cardinality relative to its input?
4. Why might using flatMap() everywhere, even for simple one-to-one transformations, be discouraged?
5. What must you use if a transformation needs to remember information across multiple records?
Was this page helpful?
You May Also Like
DataStream Basics
An introduction to Flink's DataStream API, the core abstraction for processing unbounded and bounded streams of data in real time.
Keyed Streams and keyBy
Partitioning a DataStream by key using keyBy() to enable per-key state and correct windowed aggregations.
Serialization in Flink
How Flink serializes records for network transfer, state storage, and checkpointing, and why type information matters for performance.
Related Reading
Related Study Notes in Programming
Browse all study notesApache Spark Study Notes
Programming · 30 topics
ProgrammingHadoop Study Notes
Programming · 30 topics
ProgrammingSnowflake Study Notes
Programming · 30 topics
ProgrammingApache Airflow Study Notes
Programming · 30 topics
Programmingdbt (Data Build Tool) Study Notes
Programming · 30 topics
ProgrammingPowerShell Study Notes
Programming · 30 topics