Apache Flink

Definition

Apache Flink is an open-source distributed stream-processing framework for stateful computations over both unbounded (streaming) and bounded (batch) data at scale.

Overview

Flink began as a research project called Stratosphere at TU Berlin around 2010, and was contributed to the Apache Software Foundation in 2014, graduating to a top-level project the following year. Its defining characteristic is that it treats streaming as the primary computation model, processing events one at a time (or in small pipelined batches) rather than relying on micro-batching. Flink handles out-of-order and late-arriving data through event-time processing and watermarks, and guarantees exactly-once state consistency through a distributed checkpointing mechanism, which is critical for use cases like financial transaction processing where correctness matters as much as speed. Applications can be written against the lower-level DataStream API or the higher-level Table API/Flink SQL for more declarative pipeline definitions. Flink is frequently deployed on Kubernetes and paired with Apache Kafka as its primary ingestion source, and is often discussed alongside Apache Spark's Structured Streaming as an alternative approach to the same class of real-time processing problems, with Flink generally favored when very low latency and precise event-time semantics are the top priority.

Key Features

• True event-at-a-time stream processing rather than micro-batching
• Unified APIs for both streaming and batch workloads
• Event-time processing with watermarks for out-of-order data
• Exactly-once state consistency via distributed checkpointing
• High-throughput, low-latency execution engine
• Complex event processing (CEP) library for pattern detection
• Native deployment support on Kubernetes and YARN
• Flink SQL and Table API for declarative pipeline definitions

Use Cases

Frequently Asked Questions

Related in DevOps