Background Jobs in Python: Celery vs Simpler Queues

Long-running work should not sit inside an HTTP request. Sending a transactional email, generating a PDF, processing an upload, or embedding a batch of documents for a search or RAG pipeline can take seconds or minutes. If that work runs in the request-response cycle, users wait, workers time out, and a retry from the client can start the same job twice.

Python gives you several ways to move that work into a background job. The useful question is not “which tool is best,” but which one matches the shape of your app: a small FastAPI service, a growing backend with Redis, or a distributed system that needs retries, schedules, and multiple queues.

This guide compares four options you will actually see in Python backends: FastAPI BackgroundTasks, RQ, ARQ, and Celery. It stays on the job-queue choice itself, not general async syntax or concurrency models.

What a background job needs to get right

A queue is more than “run this later.” For anything you care about, the setup should give you:

• Durability. If the API process restarts, the job is still waiting in a broker such as Redis or RabbitMQ.
• A separate worker. Heavy work runs in another process, so the web process stays free for requests.
• Retries with limits. Transient failures (a mail provider timeout, a locked row, a brief Redis blip) can be retried without blocking the user.
• Idempotency. The same job can safely run twice. That matters because retries and at-least-once delivery are normal.
• Visibility. You can see queued, running, failed, and completed jobs, and inspect why a job failed.

FastAPI BackgroundTasks only covers the “run this after the response” part. RQ, ARQ, and Celery add a broker and a worker. That is the line between a convenience helper and a real job queue.

Option comparison

If you only need to send a welcome email and can accept lost work on a process restart, start with BackgroundTasks. If the job must survive a deploy, move to RQ or ARQ. Reach for Celery when you need more than a simple Redis-backed queue: several worker pools, a scheduler, or chained workflows.

FastAPI BackgroundTasks: useful, but not a queue

FastAPI can run a function after it sends the response. You do not need Redis, a worker service, or a new deployment unit. That is perfect for short, non-critical work such as writing an audit log or sending a welcome email.

from fastapi import BackgroundTasks, FastAPI

app = FastAPI()

def send_welcome_email(email: str) -> None:
    # Keep this short. It still runs inside the API process.
    print(f"Sent welcome email to {email}")

@app.post("/register")
async def register(email: str, background_tasks: BackgroundTasks):
    background_tasks.add_task(send_welcome_email, email)
    return {"message": "Registration successful. Email will be sent shortly."}

The limitation is the same as the convenience: the job lives in memory in the API process. If the process is killed, restarted, or replaced during a deploy before the function finishes, the job is gone. There is no built-in retry dashboard, no dead-letter queue, and no way to scale that work independently of your web workers.

Do not use BackgroundTasks for payments, invoice generation, video processing, document embedding, or anything you cannot afford to lose. Also avoid long or blocking work here. It still shares the process with your API, so a slow job can hurt request latency.

RQ: the simple Redis-backed queue

RQ (Redis Queue) is a small, readable library for enqueueing Python callables and running them in a separate worker process. Jobs are stored in Redis, so they survive an API restart. The mental model is easy to teach and easy to operate: enqueue a function, run rq worker, and inspect failures with RQ’s tools.

from redis import Redis
from rq import Queue

queue = Queue("reports", connection=Redis())

def generate_report(user_id: int) -> str:
    # Long-running CPU or I/O work belongs here, not in the request handler.
    return f"report-{user_id}.pdf"

# In your API process, after the request is validated:
job = queue.enqueue(generate_report, 42)

Start a worker in another process with rq worker reports. That separation is the main upgrade over BackgroundTasks: you can restart the API without dropping queued jobs, and you can run more workers when the report backlog grows.

RQ is a strong default when your tasks are ordinary synchronous Python. That is common in backends that already use a sync database driver, SDK, or report library. The trade-off is that the worker itself is sync. If your FastAPI app is fully async and the jobs are mostly async I/O (HTTP calls, async database drivers, embedding APIs), ARQ is usually a better fit than forcing that work into RQ.

ARQ: a Redis queue that fits asyncio and FastAPI

ARQ is a lightweight job queue built on Redis and asyncio. Jobs and workers are async functions, which matches a modern FastAPI service. You still get durability and a dedicated worker, without bringing in Celery’s broader feature set.

# worker.py
from arq.connections import RedisSettings

async def download_and_embed_pdf(ctx, url: str) -> str:
    # Long-running RAG or document work in a worker, not in the API request.
    return f"Processed {url}"

class WorkerSettings:
    functions = [download_and_embed_pdf]
    redis_settings = RedisSettings(host="localhost", port=6379)

# Inside an async FastAPI route, after you accept the request:
from arq import create_pool
from arq.connections import RedisSettings

redis = await create_pool(RedisSettings(host="localhost", port=6379))
await redis.enqueue_job(
    "download_and_embed_pdf",
    "https://example.com/doc.pdf",
)

Run the worker with arq worker.WorkerSettings. Keep the API process responsible for validation, auth, and enqueueing. Keep the worker responsible for the slow work. That split stays clean as the service grows.

ARQ also supports deferred and scheduled jobs, which is enough for many product features: “send this reminder in an hour,” “retry this embedding later,” or “run this cleanup on a timetable.” If you later need multi-step workflows, routing across many queues, or a broker other than Redis, that is usually the point where Celery starts to earn its complexity.

Celery: when the simpler queues are not enough

Celery is the option people jump to first, and often earlier than they need to. It is a full distributed task queue, not just a Redis worker. That power is useful, but it also means more moving parts: a broker, a result backend, worker processes, and, in many production setups, a scheduler.

Use Celery when at least one of these is true:

• You need several queues and worker pools, for example a fast “email” queue that must not wait behind a slow “video” or “embedding” queue.
• You need workflow primitives: chaining steps, grouping tasks, or coordinating a set of jobs before the next step runs.
• You need a first-class scheduler (Celery Beat) for recurring work.
• You want a battle-tested retry, routing, and monitoring ecosystem, or you may later use RabbitMQ instead of Redis.

# tasks.py
from celery import Celery

app = Celery(
    "jobs",
    broker="redis://localhost:6379/0",
    backend="redis://localhost:6379/1",
)

@app.task(
    bind=True,
    autoretry_for=(Exception,),
    retry_backoff=True,
    retry_kwargs={"max_retries": 5},
)
def generate_report(self, user_id: int) -> str:
    return f"report-{user_id}.pdf"

# From the API process, after validation:
from tasks import generate_report

generate_report.delay(42)

The FastAPI (or other web) process should only enqueue the job. The Celery worker runs in its own process or container. That is the same shape as RQ and ARQ; Celery just gives you more control over routing, retries, and orchestration once the backlog is no longer a single queue of similar tasks.

The cost is operational. You will need to define queues, supervise workers, think about task serialization, and decide what belongs in the result backend. For a small FastAPI service with a handful of Redis-backed jobs, that overhead is usually not worth it. For a backend that already has mixed workloads and a need to isolate them, Celery is still the most complete Python option.

How to choose

• Choose FastAPI BackgroundTasks when the work is short, non-critical, and acceptable to lose on a restart. Think log writes or a best-effort notification.
• Choose RQ when you want a durable Redis queue, your jobs are mostly synchronous, and you want the smallest production setup.
• Choose ARQ when the API and the jobs are already async, especially in FastAPI services that call other APIs, async database drivers, or embedding endpoints.
• Choose Celery when you need multiple queues, workflows, scheduled tasks, or a broker choice beyond a simple Redis worker.

A practical path for many FastAPI products is to start with BackgroundTasks for the truly disposable work, add ARQ (or RQ) as soon as a job must be durable, and only introduce Celery when the queue itself becomes a system: mixed priorities, several worker types, or scheduled pipelines.

Do not pick Celery only because it is the most famous name. And do not stay on BackgroundTasks because it is already in the framework. The wrong choice usually shows up as lost jobs after a deploy, or as a week spent operating a platform you did not need yet.

Production habits that matter more than the library

Once you have a real broker and worker, the same habits apply to RQ, ARQ, and Celery:

• Keep handlers thin. Validate and persist the request in the API, then enqueue an ID, not a huge payload.
• Make jobs idempotent. Use a unique job key, a processed-flag in the database, or an upsert so a retry cannot bill twice or embed the same document twice.
• Retry only transient errors. A typo in an email address or a missing file should fail clearly. A timeout can retry with backoff and a max attempt count.
• Separate slow and fast queues. A 20-minute export should not sit in front of password-reset emails.
• Watch the backlog. Queue depth, failed jobs, and worker heartbeats tell you about production health earlier than user complaints.
• Run workers as supervised processes. Treat them like the API: deploy them, restart them cleanly, and give them their own resource limits.

Those practices matter more than micro-differences between libraries. A small RQ or ARQ setup with idempotent jobs and a failed-job view will beat a large Celery install that retries unsafe work and shares one queue for every task type.

Summary

Use FastAPI BackgroundTasks for short, non-critical work that can disappear when the process restarts. Step up to RQ or ARQ when the job must live in Redis and run in a worker. Reserve Celery for the cases that actually need a distributed task platform: multiple queues, workflows, scheduling, or a more demanding backend.

If you are choosing a queue for a FastAPI or Python backend, or you already have jobs that disappear during deploys, get in touch. I can help you pick a setup that fits the app you have now, and keep a path open for the one you will need next.