Getting Started with Celery: Asynchronous Task Queue for Python

Celery is a distributed task queue or job queue built on asynchronous message passing. It allows Python applications to process background tasks either immediately or at a scheduled time. You can configure Celery to run tasks synchronously or asynchronously, depending on your use case.

Asynchronous message passing ensures that messages are stored between the sender and receiver. This enables the sender to continue other operations after sending a message and allows multiple messages to be queued before being processed.

Main Components of Celery

• Tasks
• Task Broker
• Result Backend
• Worker

Tasks are standard Python functions that Celery executes in its workers. They are identified with a special decorator to distinguish them from regular functions.

The Task Broker is the system responsible for delivering messages between your application and Celery. It handles the transmission of tasks for execution. Supported brokers include Redis®, RabbitMQ, and Amazon SQS.

The Result Backend stores the results of executed tasks. Celery retrieves this information after completion. It supports various backends such as Redis®, RabbitMQ (AMQP), and SQLAlchemy.

A Worker is a Celery process that continuously runs in the background, waiting for tasks to arrive in the broker. Typically, multiple workers operate simultaneously to ensure concurrent task execution.

Task Life Cycle in Celery

Celery’s task execution process consists of three main stages:

• Task Registration
• Task Execution
• Result Storage

Your application sends a task to the broker, which is then picked up by a worker for execution. Once processed, the result is stored in the result backend.

Applications of Celery

Celery can be used in various ways depending on your project’s needs. Below are the most common applications:

• Periodic Execution – Tasks that run at fixed intervals, such as sending a monthly newsletter.
• Third-Party Execution – Tasks that interact with external systems, like sending emails via SMTP.
• Long-Running Execution – Tasks requiring significant time, such as compressing or processing large files.

Creating Your First Celery Program

This section demonstrates how to integrate Celery tasks into a simple Python program.

Prerequisites

To follow this guide, ensure you have:

• Python 3.7 or later
• Redis® server installed

Installing Celery

Celery can be installed via Python’s package manager (pip).

Install the latest version of Celery:

pip install celery

Install dependencies for Redis® support:

pip install celery[redis]

Writing Your First Celery Task

Here’s a simple example of converting a regular Python function into a Celery task.

Import and initialize the Celery object:

from celery import Celery

app = Celery(
    'tasks',
    broker="redis://localhost:6379/0",
    backend="redis://localhost:6379/0"
)

If your Redis® configuration differs, modify the connection string accordingly:
redis://username:password@hostname:port/db

Create a simple Celery task:

@app.task
def multiply(a, b):
    import time
    time.sleep(5)
    return a * b

Final version of tasks.py:

from celery import Celery

app = Celery(
    'tasks',
    broker="redis://localhost:6379/0",
    backend="redis://localhost:6379/0"
)

@app.task
def multiply(a, b):
   import time
   time.sleep(5)
   return a * b

Managing Celery Workers

Workers execute tasks that are queued by Celery.

Start a Celery worker:

celery -A tasks worker -n worker1 -P prefork -l INFO

Arguments explained:

• -A: Specifies the application.
• -n: Names the worker.
• -P: Defines the pool type.
• -l: Sets log level.

To run the worker in the background, add the -D flag.

Stop all Celery workers:

ps auxww | awk '/celery(.*)worker/ {print $2}' | xargs kill -9

Executing Tasks in Celery

After starting the worker, open a Python shell in the same directory as tasks.py.

Import and execute the task:

from tasks import multiply
task1 = multiply.delay(512, 100)

Check task status:

task1.state

Retrieve result:

task1.get()

Types of Celery Workers

Choosing the right worker type affects efficiency and performance.

• Solo – Executes tasks sequentially.
• Prefork – Uses multiple processes for parallel task execution.
• Eventlet – Uses green threads for concurrent I/O-bound tasks.
• Gevent – Similar to Eventlet, ideal for network-bound workloads.

Start worker with Solo pool:

celery -A tasks worker --pool=solo --loglevel=info

Start worker with Prefork pool:

celery -A tasks worker --pool=prefork --concurrency=4 --loglevel=info

Start worker with Eventlet or Gevent pool:

celery -A tasks worker --pool=eventlet --concurrency=500 --loglevel=info
celery -A tasks worker --pool=gevent --concurrency=500 --loglevel=info

Note: Install dependencies for Eventlet or Gevent separately:
pip install celery[eventlet] or pip install celery[gevent]

Conclusion

With this guide, you learned how to integrate Celery into Python applications. Celery powers background task execution for many SaaS applications, offering scalable and efficient asynchronous processing for diverse workloads.