Run a dedicated scheduler worker

🛠 This page is for engineering teams self-hosting their own Lightdash instance. If you want to set up scheduled deliveries, go to the Scheduled deliveries guide.

By default, the Lightdash backend runs scheduled jobs in-process. In production, run a dedicated scheduler worker so a heavy dashboard export can’t starve the API. Run one through the Helm chart’s built-in worker, or as a standalone process you start yourself.

Helm chart

Set scheduler.enabled: true and the chart deploys a dedicated scheduler worker:

scheduler:
  enabled: true
  replicas: 1            # scale up if deliveries queue at peak times
  concurrency: 3         # jobs per replica (sets SCHEDULER_CONCURRENCY)
  resources:
    requests:
      cpu: 500m
      memory: 1425Mi
      ephemeral-storage: 1Gi

When scheduler.enabled: true, the chart automatically sets SCHEDULER_ENABLED: "false" on the backend — don’t set it yourself. Tune delivery throughput with SCHEDULER_JOB_TIMEOUT, SCHEDULER_SCREENSHOT_TIMEOUT, and scheduler.tasks.include / scheduler.tasks.exclude to shard task types across worker groups. See the scheduler environment variables for the full reference.

Standalone worker

For infrastructure that doesn’t use the Helm chart, run the scheduler worker as a separate process.

1. Disable the scheduler worker in the main server

Set the following environment variable value:

SCHEDULER_ENABLED=false

2. Run the standalone scheduler worker

Command to start scheduler worker:

yarn workspace backend scheduler

Note that it expects the same environment variables as the main server.

Example docker-compose.yml

version: "3.8"
x-environment: &commonEnvironment
  - PGHOST=${PGHOST:-db}
  - PGPORT=${PGPORT:-5432}
  - PGUSER=${PGUSER:-postgres}
  - PGPASSWORD=${PGPASSWORD}
  - PGDATABASE=${PGDATABASE:-postgres}
  - SECURE_COOKIES=${SECURE_COOKIES:-false}
  - TRUST_PROXY=${TRUST_PROXY:-false}
  - LIGHTDASH_SECRET=${LIGHTDASH_SECRET}
  - PORT=${PORT:-8080}
  - SITE_URL=${SITE_URL}
  - EMAIL_SMTP_HOST=${EMAIL_SMTP_HOST}
  - EMAIL_SMTP_PORT=${EMAIL_SMTP_PORT}
  - EMAIL_SMTP_SECURE=${EMAIL_SMTP_SECURE}
  - EMAIL_SMTP_USER=${EMAIL_SMTP_USER}
  - EMAIL_SMTP_PASSWORD=${EMAIL_SMTP_PASSWORD}
  - EMAIL_SMTP_ALLOW_INVALID_CERT=${EMAIL_SMTP_ALLOW_INVALID_CERT}
  - EMAIL_SMTP_SENDER_NAME=${EMAIL_SMTP_SENDER_NAME}
  - EMAIL_SMTP_SENDER_EMAIL=${EMAIL_SMTP_SENDER_EMAIL}
  - HEADLESS_BROWSER_HOST=headless-browser
  - HEADLESS_BROWSER_PORT=3000
services:
  headless-browser:
    image: browserless/chrome
    restart: always
    ports:
      - "3001:3000"

  db:
    image: postgres
    restart: always
    environment:
      POSTGRES_PASSWORD: ${PGPASSWORD}
      POSTGRES_USER: ${PGUSER:-postgres}
      POSTGRES_DB: ${PGDATABASE:-postgres}
    volumes:
      - db-data:/var/lib/postgresql/data

  lightdash:
    image: lightdash/lightdash:latest
    depends_on:
      - db
    environment:
       <<: *commonEnvironment
       SCHEDULER_ENABLED: 'false'
    volumes:
      - "${DBT_PROJECT_DIR}:/usr/app/dbt"
    ports:
      - ${PORT:-8080}:${PORT:-8080}

  scheduler:
    image: lightdash/lightdash:latest
    entrypoint: ["yarn", "workspace", "backend", "scheduler"]
    depends_on:
      - db
    environment: *commonEnvironment

volumes:
  db-data:

​Helm chart

​Standalone worker

​1. Disable the scheduler worker in the main server

​2. Run the standalone scheduler worker

​Example docker-compose.yml

Helm chart

Standalone worker

1. Disable the scheduler worker in the main server

2. Run the standalone scheduler worker

Example docker-compose.yml