Apache Storm
By the Apache Software Foundation
Apache Storm is an open-source, real-time distributed stream computation system designed to process high-velocity, unbounded data streams with low latency, arranging processing logic as a topology of spouts and bolts.
Definition
Apache Storm is an open-source, real-time distributed stream computation system designed to process high-velocity, unbounded data streams with low latency, arranging processing logic as a topology of spouts and bolts.
Overview
Storm was created by Nathan Marz at the startup BackType, which was acquired by Twitter in 2011; Twitter open-sourced Storm shortly after, and it was donated to the Apache Software Foundation, graduating to a top-level project in 2014. A Storm application is defined as a topology: a directed graph where spouts pull data in from an external source (commonly Apache Kafka) and bolts perform processing, filtering, aggregation, or writing to a destination. Unlike micro-batch systems, Storm processes records individually as they arrive, giving it very low processing latency, and by default guarantees at-least-once delivery, with an optional Trident API layered on top for exactly-once, stateful processing. Storm competes with newer stream-processing engines such as Apache Spark Structured Streaming and Apache Flink; in recent years many teams evaluating new stream-processing stacks have gravitated toward Flink or Spark for their broader ecosystems and unified batch/streaming APIs, so Storm today is more often encountered in existing systems than chosen for brand-new projects. Where ultra-low latency, record-by-record processing is the priority, Storm remains a proven, battle-tested option.
Key Features
- Topology model of spouts (data sources) and bolts (processing units)
- True record-at-a-time stream processing for low latency
- At-least-once processing guarantees by default
- Trident API for stateful, exactly-once processing
- Horizontal scalability across worker nodes
- Language-agnostic component definition via Thrift
- Fault tolerance through supervisor and worker process monitoring
Use Cases
Frequently Asked Questions
From the Blog
Introduction to Apache Spark for Beginners
Apache Spark is a fast, distributed engine for processing huge datasets across many machines. Learn what it is, how it works, and how to run your first job.
Read More AI & TechnologyWhat Is PySpark? Python's Gateway to Big Data
PySpark is the Python API for Apache Spark, letting developers process massive datasets across many machines using familiar Python syntax. This guide covers what PySpark does, its core components, and when to reach for it.
Read More