Late-arriving data is a fundamental challenge of stream processing: events may arrive minutes or hours after generation, due to network delays, device reconnection, or intermediate system buffering. A streaming pipeline ignoring late data produces incorrect aggregations — a bowler's economy computed from only the deliveries that arrived on time, silently missing late ones. Watermarks are the mechanism Spark Structured Streaming uses to handle this: a bounded delay tolerance that allows late events to update previously computed results while discarding events that arrive too late.
The watermark is a threshold defined relative to event time: `withWatermark('event_ts', '10 minutes')` tells Spark that events may arrive up to 10 minutes late, and any event with a timestamp more than 10 minutes behind the current maximum observed timestamp should be considered too late and dropped. Spark computes the current watermark as `max(observed_event_time) - delay_threshold` and advances it monotonically. Windows whose end time is earlier than the current watermark are finalised, emitted in `append` mode, and their state memory released.