Introduction

This post is to help with the following two tutorials (for setting up Celery with Django):

• https://levelup.gitconnected.com/django-celery-going-deeper-with-background-tasks-in-python-fa44958cf7cd
• https://testdriven.io/courses/django-celery/docker/

If you’re using or want to use Windows with Celery, it is better to run it in a container (or another way that you’re actually using Linux). Since Celery on Windows is not supported and they will not help you if you are going to still do this.

Assumption

• You have Docker and docker compose up and running locally.

Overview

• Django, Python webframework.
• Redis, in memory key-value store, with message broker capabilities.
• Celery, used to schedule jobs in a distributed system, allows both synchronous and asynchronous jobs.

Environment setup

Docker-compose can be used to run everything in a Linux container no matter the host platform.

version: "3.8"
services:
  django_server:
    build:
      context: .
      dockerfile: Dockerfile
    command: python manage.py runserver 0.0.0.0:8000
    ports:
      - "8000:8000"
    volumes:
      - ./:/app
    depends_on:
      - redis
    working_dir: /app


  # No longer being developed.
  # flower:
  #   build:
  #     context: .
  #     dockerfile: Dockerfile
  #   command: celery -A django_celery_example flower --port=5555
  #   ports:
  #     - "5555:5555"
  #   volumes:
  #     - ./:/app
  #   depends_on:
  #     - redis
  #   working_dir: /app

  worker:
    build:
      context: .
      dockerfile: Dockerfile
    command: celery -A django_celery_example worker --loglevel=info
    volumes:
      - ./:/app
    depends_on:
      - redis
    working_dir: /app

  redis:
    image: redis:alpine
    ports:
      - "6379:6379"

And the Dockerfile:

FROM python:alpine
# Currently = 3.11.1-alpine3.17

WORKDIR /app

# Fix stdout not updating...
ENV PYTHONUNBUFFERED=1

COPY requirements.txt /requirements.txt
RUN pip install -r /requirements.txt && rm -f /requirements.txt && mkdir -p /app

Exploration

Some changes you could do with the tutorials:

• Return JSON (and sent data to the API via Postman or CURL)
• Running multiple jobs
• Monitoring
  • Locally (e.g. Flower, Clearly)
  • Or in production (Morpheus and Kibana/Grafana, or some of the cloud platform solutions)