Exactly-Once Semantics

Analogy categoryAnalogies

Delivery Guarantees: At-Most-Once, At-Least-Once, Exactly-Once

Streaming systems describe their failure-recovery guarantee along a spectrum: at-most-once means a failure can silently drop records, at-least-once means a failure can cause records to be reprocessed and therefore double-counted or duplicated downstream, and exactly-once means that, despite failures, each input record affects the final result as if it were processed exactly one time — no drops, no duplicates. Achieving exactly-once end-to-end requires two separate pieces working together: exactly-once state updates inside Flink (which checkpointing provides) and exactly-once effects at the sink, which requires the sink itself to support transactional or idempotent writes.

🏏

Cricket analogy: A DRS review must reach exactly one final decision — out or not out — never silently skipping the review (at-most-once) nor applying the same wicket twice to the scorecard (at-least-once duplication), which is the exactly-once ideal.

Exactly-Once Within Flink: Checkpointing Provides It for Free

Inside the boundary of a Flink job, exactly-once state consistency comes automatically from the checkpointing mechanism described earlier: because a checkpoint captures a consistent cut of both operator state and stream position simultaneously, recovering from any checkpoint and replaying records from the recorded source offsets reproduces exactly the same state as if no failure had occurred — Flink deduplicates any reprocessing internally by discarding the (now-stale) in-progress computation and resuming cleanly from the snapshot. This internal guarantee is enabled simply by setting CheckpointingMode.EXACTLY_ONCE (the default when checkpointing is enabled), as opposed to CheckpointingMode.AT_LEAST_ONCE, which skips barrier alignment for lower latency at the cost of potential duplicate processing within the job graph itself.

🏏

Cricket analogy: When rain interrupts play mid-over, the match resumes from the exact recorded ball count rather than replaying the whole over, ensuring every legitimate ball is counted exactly once toward the final total, like checkpoint-based internal consistency.

End-to-End Exactly-Once: The Two-Phase Commit Sink

Internal exactly-once alone doesn't guarantee exactly-once effects on external systems, because a sink might have already emitted output for a record before the job crashes and replays that record after restart. Flink's TwoPhaseCommitSinkFunction (and its modern replacement, the unified Sink API's SupportsCommitter) solves this by tying external transactions to the checkpoint lifecycle: the sink opens a transaction before writing (pre-commit), buffers writes within that transaction without making them externally visible, and only commits the transaction when notified that the corresponding checkpoint has been globally acknowledged as complete — meaning a replay after failure re-executes the same records into a transaction that either never got pre-committed (so is safely discarded) or was already committed (so the replay's records land in a fresh transaction with no duplicate effect).

🏏

Cricket analogy: A DRS decision is only finalized and shown on the big screen after the third umpire formally confirms it, keeping the tentative review invisible to players until that confirmation, mirroring a transaction only becoming externally visible on commit.

java

KafkaSink<String> sink = KafkaSink.<String>builder()
    .setBootstrapServers("broker1:9092,broker2:9092")
    .setRecordSerializer(KafkaRecordSerializationSchema.builder()
        .setTopic("orders-out")
        .setValueSerializationSchema(new SimpleStringSchema())
        .build())
    .setDeliveryGuarantee(DeliveryGuarantee.EXACTLY_ONCE)
    .setTransactionalIdPrefix("orders-pipeline-tx")
    .build();

Where Exactly-Once Can Still Break

Exactly-once guarantees only extend as far as every component in the pipeline supports them: a source must be replayable with recorded offsets (Kafka and Kinesis qualify; a non-replayable socket source does not), and a sink must support transactions or be naturally idempotent (a Kafka sink with EXACTLY_ONCE delivery guarantee qualifies, but a sink that calls a non-idempotent external REST API to increment a counter does not, no matter how the checkpoint is configured). Transactional Kafka sinks also introduce an operational subtlety: transactions left open past transaction.max.timeout.ms on the Kafka broker are aborted, so checkpoint interval and Kafka's transaction timeout must be tuned together, or a slow checkpoint can silently cause data loss when its pending transaction expires.

🏏

Cricket analogy: A chain is only as strong as its weakest link: even with perfect ball-by-ball scoring, if the final printed scorecard uses a faulty printer that occasionally skips a line, the official record still ends up wrong, mirroring how a non-idempotent sink breaks end-to-end exactly-once.

Setting DeliveryGuarantee.EXACTLY_ONCE on a Flink sink is not a magic switch — it only produces true exactly-once behavior when the underlying external system actually supports transactions or idempotent writes; wrapping a plain non-transactional database UPSERT in an exactly-once-labeled sink still risks duplicate side effects unless the write itself is naturally idempotent (e.g., keyed UPSERT by primary key).

A common and simpler alternative to two-phase commit sinks is idempotent writes: if the sink write is naturally idempotent (upsert by deterministic key, or a database with a unique constraint on an embedded record ID), at-least-once delivery combined with an idempotent write achieves the same effective outcome as exactly-once without transactional overhead.

Exactly-once means each record affects the final result as if processed exactly one time, despite failures — no silent drops, no duplicated effects.
Flink's checkpointing mechanism provides exactly-once state consistency inside the job automatically under CheckpointingMode.EXACTLY_ONCE.
True end-to-end exactly-once additionally requires transactional or idempotent sinks, since internal consistency alone doesn't prevent duplicate external effects.
TwoPhaseCommitSinkFunction (and the modern Sink API's committer) ties external transaction commit to checkpoint completion notification.
Sources must be replayable from recorded offsets for exactly-once to hold; non-replayable sources cap the guarantee at at-most-once.
Kafka transactional sinks require coordinating checkpoint interval with transaction.max.timeout.ms to avoid silently aborted transactions.
Idempotent writes are a simpler alternative to two-phase commit when the target system supports deterministic upserts.