/api-queue-bullmq

name: api-queue-bullmq
description: Job queues, background processing, and task scheduling with BullMQ v5

BullMQ Patterns

> **Quick Guide:** Use BullMQ (v5.x) for background job processing, task scheduling, and workflow orchestration on top of Redis. Core classes: `Queue` (adds jobs), `Worker` (processes jobs), `QueueEvents` (global event listener), `FlowProducer` (parent-child job trees). Always pass a `connection` object to every constructor (required in v5). Set `maxRetriesPerRequest: null` on ioredis connections for Workers. Use `upsertJobScheduler` for repeatable/cron jobs (replaces deprecated repeatable API). QueueScheduler was removed in v4 -- its responsibilities are now handled by Workers automatically.

---

<critical_requirements>

CRITICAL: Before Using This Skill

**(You MUST pass a `connection` object to every Queue, Worker, QueueEvents, and FlowProducer constructor -- BullMQ v5 throws if connection is missing)**

**(You MUST set `maxRetriesPerRequest: null` on ioredis connections used by Workers -- BullMQ requires infinite retries and throws without this setting)**

**(You MUST call `await worker.close()` on SIGTERM/SIGINT for graceful shutdown -- without it, in-progress jobs become stalled)**

**(You MUST use `upsertJobScheduler` for repeatable/cron jobs -- the old `repeat` option on `queue.add` is deprecated since v5.16.0)**

</critical_requirements>

---

Examples

• [Core Patterns](examples/core.md) -- Queue setup, Worker processing, job options, connection factory, graceful shutdown, typed jobs
• [Advanced Patterns](examples/advanced.md) -- FlowProducer, rate limiting, job scheduling, QueueEvents, concurrency, sandboxed processors

**Additional resources:**

• [reference.md](reference.md) -- Decision frameworks, job option reference, anti-patterns, production checklist

---

**Auto-detection:** BullMQ, bullmq, Queue, Worker, QueueEvents, FlowProducer, job queue, background job, worker process, job scheduler, upsertJobScheduler, rate limiter, job priority, job delay, sandboxed processor, repeatable job, cron job, flow producer, parent child jobs

**When to use:**

• Background processing (email sending, image processing, PDF generation)
• Scheduled/cron jobs (nightly reports, periodic cleanup)
• Workflow orchestration with parent-child job dependencies (FlowProducer)
• Rate-limited API consumption (throttling outbound requests)
• Priority-based job processing (urgent jobs before bulk operations)
• Distributing CPU-intensive work across multiple workers or machines

**Key patterns covered:**

• Queue and Worker setup with typed job data and return values
• Connection factory with `maxRetriesPerRequest: null` for Workers
• Job options: delay, priority, attempts, backoff, removeOnComplete/Fail
• Graceful shutdown with `worker.close()` on process signals
• FlowProducer for parent-child job trees with dependency tracking
• Job Schedulers for repeatable/cron jobs (`upsertJobScheduler`)
• Rate limiting (global limiter and manual `Worker.RateLimitError`)
• Concurrency control (local per-worker and global)
• QueueEvents for global event monitoring across all workers
• Sandboxed processors for CPU-intensive work

**When NOT to use:**

• Simple in-process timers or `setTimeout` (no persistence needed)
• Real-time pub/sub messaging without persistence (use your pub/sub solution)
• Data that must be processed synchronously within a request-response cycle
• Queues that don't need persistence, retries, or scheduling

---

<philosophy>

Philosophy

BullMQ is a **Redis-backed job queue** for Node.js that provides reliable background processing with at-least-once delivery guarantees. The core principle: **separate job production from job consumption** so your application stays responsive while work happens asynchronously.

**Core principles:**

1. **Producers and consumers are decoupled** -- Any process can add jobs to a queue; any Worker can process them. This enables horizontal scaling by adding more Workers. 2. **Jobs are persistent** -- Jobs survive process restarts because they live in Redis. A crashed Worker's jobs are picked up by other Workers (or the same Worker after restart). 3. **At-least-once delivery** -- BullMQ guarantees every job is processed at least once. Use idempotent processors to handle the (rare) case of duplicate processing after a stall. 4. **Fail gracefully with retries** -- Configure `attempts` and `backoff` strategies so transient failures resolve automatically. Permanently failed jobs move to the failed set for inspection. 5. **Each Queue/Worker/QueueEvents needs its own connection** -- BullMQ manages connection state internally. Never share a single ioredis instance across multiple BullMQ classes (except Queues acting only as producers).

</philosophy>

---

<patterns>

Core Patterns

Pattern 1: Connection Factory

BullMQ v5 requires an explicit Redis connection on every constructor. Workers need `maxRetriesPerRequest: null` so ioredis retries indefinitely instead of giving up.

import Redis from "ioredis";

function createBullMQConnection(): Redis {
  const url = process.env.REDIS_URL;
  if (!url) throw new Error("REDIS_URL is required");
  return new Redis(url, { maxRetriesPerRequest: null });
}
export { createBullMQConnection };

**Why good:** `maxRetriesPerRequest: null` satisfies BullMQ's requirement, factory ensures consistent config, environment variable keeps credentials out of code

// Bad -- missing maxRetriesPerRequest
const redis = new Redis("redis://localhost:6379");
const worker = new Worker("emails", processor, { connection: redis });
// BullMQ throws: "maxRetriesPerRequest must be null"

**Why bad:** BullMQ requires infinite retries on Worker connections and will throw at startup without `null`

See [examples/core.md](examples/core.md) for the full connection factory with producer vs consumer separation.

---

Pattern 2: Queue and Worker Setup

Queue adds jobs; Worker proce