Flink Quick Reference

This quick reference collects the Flink APIs, configuration keys, and CLI commands you reach for most often once you already understand the underlying concepts. It's meant as a fast lookup, not a tutorial — pair it with the deeper best-practices and architecture guides when you need the full reasoning behind a setting.

The DataStream API's core building blocks are map/flatMap (transform each record), keyBy (logically partition by key), window (group records within a time or count boundary), and process (custom low-level logic via ProcessFunction/KeyedProcessFunction with timers). The Table API and Flink SQL sit on top of the same runtime and offer a declarative, relational way to express the same joins, aggregations, and windowing, and can be freely converted to and from DataStreams via StreamTableEnvironment.

Common flink-conf.yaml and job-level settings: parallelism.default sets the cluster-wide default parallelism when an operator doesn't specify its own; state.backend selects hashmap or rocksdb; execution.checkpointing.interval controls how often checkpoints fire in milliseconds; and taskmanager.memory.process.size sets the total memory budget for a TaskManager process, which Flink then subdivides into managed memory, network buffers, and JVM heap/off-heap regions automatically.

Session mode runs a long-lived Flink cluster that accepts multiple jobs submitted over time, sharing JobManager and TaskManager resources — convenient for development and short-lived ad-hoc jobs. Per-job mode (largely superseded by application mode in newer Flink versions) spins up a dedicated cluster for a single job's lifetime. Application mode runs the job's main() method inside the JobManager itself rather than on the client, producing a cluster dedicated to exactly one application and avoiding the need to ship the job's dependencies to a submitting client — the recommended default for production.