Celery
Celery is an open-source distributed task queue for Python that lets applications run work asynchronously and on a schedule — offloading slow or delayed jobs to background worker processes instead of blocking the main request-response…
Definition
Celery is an open-source distributed task queue for Python that lets applications run work asynchronously and on a schedule — offloading slow or delayed jobs to background worker processes instead of blocking the main request-response cycle.
Overview
Web applications frequently need to do things that shouldn't happen inline with a user's request — sending emails, resizing images, generating reports, or calling slow third-party APIs. Celery solves this by letting a Python application publish "tasks" to a message broker, most commonly RabbitMQ or Redis, which one or more separate worker processes then pick up and execute independently of the web server. A typical setup pairs Celery with a web framework such as Django or Flask: the app enqueues a task (e.g. "send_welcome_email"), Celery workers consume it from the broker, and results — success, failure, retries — can optionally be stored in a results backend for the app to check later. Celery also supports periodic tasks (a built-in scheduler called Celery Beat), task chaining and grouping for building multi-step pipelines, rate limiting, and automatic retries with backoff for flaky operations. Because it decouples slow work from request handling, Celery is a long-standing default choice in the Python ecosystem for background job processing, and it scales horizontally simply by adding more worker processes or machines listening on the same queue.
Key Features
- Distributed task queue built around a message-broker architecture (RabbitMQ, Redis, or others)
- Asynchronous execution of background jobs outside the request-response cycle
- Celery Beat scheduler for periodic and cron-like recurring tasks
- Task chaining, grouping, and chords for building multi-step pipelines
- Automatic retries with configurable backoff for failed tasks
- Result backends for tracking task status and return values
- Horizontal scaling by adding more worker processes or machines
- Deep integration with Python web frameworks like Django and Flask
Use Cases
History
Celery is an open-source distributed task queue for Python, used to run work asynchronously in the background and to schedule periodic jobs. It was created by Ask Solem and first released on April 24, 2009. Initially built to give Django applications a robust background-task mechanism, it quickly evolved into a general-purpose task queue usable from any Python project. Celery distributes "tasks" across one or more worker processes via message brokers such as RabbitMQ or Redis, supporting concurrency through multiprocessing, eventlet, or gevent. It became Python's de-facto standard for asynchronous processing, running at large scale in production systems that handle millions of tasks per day.
Sources
- Celery — official documentation · as of 2026-07-17
- Celery on GitHub — celery/celery · as of 2026-07-17