Spark Structured Streaming is a fault-tolerant, exactly-once stream processing engine built on the Spark SQL execution engine. It models a live data stream as an unbounded table that grows continuously — each new batch of arriving data appends rows to this table — and allows users to express streaming queries using the same DataFrame and SQL API used for batch processing. This unified API means a Spark engineer who knows batch DataFrame transformations can write streaming pipelines immediately, with the framework handling state management, fault recovery, and output consistency automatically.
The key design insight of Structured Streaming is that a streaming query is a continuously running batch job: Spark periodically polls the source for new data, processes it using the same Catalyst-optimised plan as a batch query, and writes results to the output sink. The trigger interval controls how frequently this micro-batch runs — from 500ms to hours. This micro-batch model sacrifices true event-at-a-time latency (minimum ~100ms) for the simplicity and correctness guarantees of the batch execution model, making Structured Streaming the right choice for most analytics streaming workloads.