/task-scheduling

name: task-scheduling
description: This skill should be used when the user designs a "task scheduler", "job scheduler", "job queue", "cron at scale", "distributed cron", "delayed / scheduled / recurring tasks", a "worker pool", reaches for "Celery / Sidekiq / Airflow", or wrestles with "task leasing", visibility timeouts, job priorities, fairness, or duplicate task execution. Use it whenever work must run later, on a schedule, or be reliably leased to a pool of workers, even if the user doesn't say "scheduler". (For plain queue transport and delivery guarantees, that is `messaging-streaming`.)

Task Scheduling

Decide *when* work runs and *which worker* runs it: fire jobs on a schedule (cron/delayed/recurring), hand each job to exactly one worker via a lease, and make sure it completes once despite crashes and retries. This sits *on top of* `messaging-streaming` queues — the queue is the transport; this skill adds the scheduling, leasing, priorities, and task-level idempotency. Getting it wrong shows up as jobs that never run, run twice (double charge, double email), or pile up until a worker fleet falls permanently behind.

When to reach for this

Work must run **later** (send a reminder in 24h), **on a schedule** (nightly rollups, hourly cron), or **repeatedly** (poll every 5 min); a slow operation is already off the request path (→ `messaging-streaming`) and now needs reliable allocation to a pool of workers; jobs need **priorities** (paid before free) or **fairness** (no single tenant starves others); or a job must complete **exactly once** even though the worker holding it can crash mid-flight.

When NOT to

The caller needs the result inline — that's a synchronous call, not a scheduled job. A single fire-and-forget async step with no schedule, priority, or exactly-once need — a plain queue + idempotent consumer (`messaging-streaming`) is simpler; don't add a scheduler on top. One periodic job on one box — OS `cron` is fine until you have multiple schedulers or need history and retries. A long-running multi-step saga with rollback — reach for a durable workflow engine instead of hand-rolling state across jobs. Don't stand up Airflow/Celery "because we'll have batch jobs eventually" (YAGNI): it's a stateful control plane to operate and monitor.

Clarify first

• **Trigger type** — scheduled (cron/at a time), delayed (run after N seconds),

recurring (every N), or event-driven (a queue message arrives)? This decides whether a scheduler is even in scope.

• **Exactly-once vs at-least-once** — is a duplicate run harmful (money, email)

or harmless (idempotent recompute)? Drives the leasing + dedup design.

• **Latency budget vs throughput** — must a delayed job fire within seconds of

its time, or is "within a few minutes" fine? Tight timing is more expensive.

• **Priority / fairness** — do some jobs jump the line, and must one tenant or

job class be prevented from starving the rest? (→ `back-of-the-envelope` for arrival vs. service rate.)

• **Job duration & variance** — seconds or hours? Sets the visibility-timeout /

lease length and whether long jobs need heartbeats.

• **Idempotency key** — what identifies a task as the same task on retry?

(The key contract is owned by `api-design`.)

The options

**Scheduling trigger**

• **OS cron / single scheduler** — one process fires jobs on a crontab. *Use

when* one node, a handful of jobs, no HA requirement.

• **Distributed scheduler (HA cron)** — a leader-elected scheduler enqueues due

jobs into a queue; followers stand by. *Use when* the schedule must survive a node loss and must not double-fire.

• **Delay queue / timer** — jobs carry a "not before" time; the queue holds them

until due (delivery delay, sorted-set scoring, or a timer wheel). *Use when* per-job delays vary and you don't want a cron tick.

• **Workflow/orchestration DAG** — declared task dependencies with backfill and

history (Airflow-style). *Use when* batch pipelines have dependencies and you need a run history and reruns.

**Worker allocation**

• **Pull (worker leasing)** — workers poll the queue, lease a job for a

**visibility timeout**, and ack/delete on success. *Use when* you want back-pressure for free and elastic, self-balancing workers. The default.

• **Push (dispatcher assigns)** — a coordinator routes jobs to specific workers.

*Use when* affinity/locality matters (a job must run where its data is).

**Priority & fairness**

• **Priority queues** — separate high/low queues drained in order. *Use when*

some classes must run first.

• **Weighted / fair scheduling** — round-robin or weighted draw across per-tenant

queues. *Use when* one tenant's burst must not starve others.

Trade-offs

| Option | What it solves | What it worsens | Change it when | |---|---|---|---| | OS cron / single scheduler | Trivial; zero infra | SPOF — node dies, schedule stops; no retry/history | You need HA or missed-run recovery → distributed scheduler | | Distributed scheduler (HA cron) | Survives node loss; no double-fire (leader-elected) | Needs leader election (→ `consistency-coordination`); more moving parts | One box and one job is enough → OS cron | | Delay queue / timer | Per-job delays without a cron tick; precise-ish timing | Far-future jobs sit in the queue; timer accuracy bounded by poll interval | Delays are uniform/periodic → cron; dependencies exist → DAG | | Workflow DAG (Airflow-style) | Dependencies, backfill, run history, reruns | Heavy control plane; scheduler latency; overkill for single jobs | Jobs are independent one-shots → plain queue + scheduler | | Pull (worker leasing) | Self-balancing, elastic, natural back-pressure | At-least-once: lease expiry on a slow job re-runs it (need idempotency) | A job must run on a specific node (data locality) → push | | Push (dispatcher) | Affinity/locality; central control | Dispatcher is a bottleneck/SPOF; must track worker health | No locality need → pull is simpler | | Priority