Setting Up a Minimal Project
A typical Flink project starts from the official Maven archetype (org.apache.flink:flink-quickstart-java), which scaffolds a pom.xml with the correct flink-streaming-java dependency and a StreamingJob.java stub containing an empty main method. Dependencies for the core API are marked provided in the pom since the cluster already supplies flink-dist at runtime, while connector dependencies like flink-connector-kafka must be bundled into your job JAR using the Maven Shade plugin so they are available on every TaskManager.
Cricket analogy: The Maven archetype is like a franchise's standard training kit issued to every new signing, whether you play for Chennai Super Kings or Delhi Capitals, ensuring everyone starts with the same base gear before adding personal touches.
Writing the Job
A minimal job reads from a source, applies one or more transformations, and writes to a sink, then calls env.execute() to trigger execution. In this example, we read comma-separated sensor readings from a socket, parse them into a typed record, key by sensor ID, compute a rolling maximum temperature per sensor using a tumbling window, and print the results. Note that env.execute() must be the final call; nothing runs before it because the DataStream API only builds the logical plan until execution is explicitly triggered.
Cricket analogy: Building the job step by step is like a bowler's run-up: approach (source), gather (transformation), release (window aggregation), and follow-through (sink) — the ball is only actually delivered at the very end, like env.execute().
public class SensorMaxTemperatureJob {
public static void main(String[] args) throws Exception {
final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
DataStream<String> raw = env.socketTextStream("localhost", 9999);
DataStream<SensorReading> readings = raw.map(line -> {
String[] parts = line.split(",");
return new SensorReading(parts[0], Double.parseDouble(parts[1]), Long.parseLong(parts[2]));
}).returns(SensorReading.class);
DataStream<SensorReading> maxPerWindow = readings
.keyBy(reading -> reading.sensorId)
.window(TumblingProcessingTimeWindows.of(Time.seconds(10)))
.max("temperature");
maxPerWindow.print();
env.execute("Sensor Max Temperature Job");
}
}Running and Verifying the Output
To test this job locally, open a terminal and run nc -lk 9999 to start a simple socket listener, then type comma-separated lines like sensor-1,23.5,1699999999 to simulate readings. Running the job from your IDE or via ./bin/flink run submits it to a local or standalone cluster, and results appear in the TaskManager's stdout log (visible in the Web UI's TaskManager logs tab) since print() writes to the sink's local task output rather than your terminal when run on a cluster.
Cricket analogy: Feeding lines into the socket is like a bowling machine operator manually feeding balls one at a time during a practice session, each ball simulating a live delivery for the batsman to react to.
When running via an IDE with the local MiniCluster (the default for getExecutionEnvironment() outside a real cluster), print() output does go directly to your IDE console, which makes local development and debugging fast before you package and deploy the JAR to a real cluster.
Tumbling processing-time windows like Time.seconds(10) group events by wall-clock arrival time, not by any timestamp embedded in the event, which means results can vary between runs if data arrives at different real-world speeds. For deterministic, replayable results, use event-time windows with watermarks instead.
- A minimal Flink job scaffolds from the flink-quickstart-java Maven archetype.
- Core Flink dependencies are marked provided; connector JARs are shaded into the job JAR.
- A job reads a source, applies transformations, writes to a sink, then calls env.execute().
- Nothing executes until env.execute() is called, since the API only builds a logical plan before that.
- keyBy plus a window function like TumblingProcessingTimeWindows enables per-key rolling aggregation.
- print() output appears in TaskManager logs on a real cluster, but directly in the IDE console with a local MiniCluster.
- Processing-time windows are non-deterministic across runs; event-time windows with watermarks give replayable results.
Practice what you learned
1. What must every Flink DataStream job call to actually trigger execution?
2. Why are core Flink dependencies like flink-streaming-java marked as 'provided' in the pom.xml?
3. What combination enables computing a rolling maximum per sensor ID in the example job?
4. Where does print() output appear when a job runs on a real standalone or distributed cluster?
5. What is a key limitation of tumbling processing-time windows compared to event-time windows?
Was this page helpful?
You May Also Like
Installing and Running Flink
A practical guide to installing Apache Flink locally, starting a standalone cluster, and submitting your first job through the CLI and Web UI.
The Flink Execution Model
How Flink transforms a program into a JobGraph and ExecutionGraph, schedules parallel tasks, and moves data between operators via streams and partitions.
Flink Architecture
A tour of Flink's distributed architecture: JobManager, TaskManagers, slots, and how a job is deployed and coordinated across a cluster.
Related Reading
Related Study Notes in Programming
Browse all study notesApache Spark Study Notes
Programming · 30 topics
ProgrammingHadoop Study Notes
Programming · 30 topics
ProgrammingSnowflake Study Notes
Programming · 30 topics
ProgrammingApache Airflow Study Notes
Programming · 30 topics
Programmingdbt (Data Build Tool) Study Notes
Programming · 30 topics
ProgrammingPowerShell Study Notes
Programming · 30 topics