Apache Storm

Definition

Apache Storm is an open-source, real-time distributed stream computation system designed to process high-velocity, unbounded data streams with low latency, arranging processing logic as a topology of spouts and bolts.

Overview

Storm was created by Nathan Marz at the startup BackType, which was acquired by Twitter in 2011; Twitter open-sourced Storm shortly after, and it was donated to the Apache Software Foundation, graduating to a top-level project in 2014. A Storm application is defined as a topology: a directed graph where spouts pull data in from an external source (commonly Apache Kafka) and bolts perform processing, filtering, aggregation, or writing to a destination. Unlike micro-batch systems, Storm processes records individually as they arrive, giving it very low processing latency, and by default guarantees at-least-once delivery, with an optional Trident API layered on top for exactly-once, stateful processing. Storm competes with newer stream-processing engines such as Apache Spark Structured Streaming and Apache Flink; in recent years many teams evaluating new stream-processing stacks have gravitated toward Flink or Spark for their broader ecosystems and unified batch/streaming APIs, so Storm today is more often encountered in existing systems than chosen for brand-new projects. Where ultra-low latency, record-by-record processing is the priority, Storm remains a proven, battle-tested option.

Key Features

• Topology model of spouts (data sources) and bolts (processing units)
• True record-at-a-time stream processing for low latency
• At-least-once processing guarantees by default
• Trident API for stateful, exactly-once processing
• Horizontal scalability across worker nodes
• Language-agnostic component definition via Thrift
• Fault tolerance through supervisor and worker process monitoring

Use Cases

Frequently Asked Questions

Related in DevOps