Design a Distributed Job Scheduler (High-Level Design)
High-level design of a distributed job scheduler (cron-as-a-service): jobs indexed by nextRunTime, a leader scheduler that enqueues due jobs, stateless workers, and the reliability story — leader election, at-least-once + idempotency, visibility timeouts, retries and sharding.
🔴 High-Level Design · Case Study · Super-senior
"Design a distributed job scheduler" (think cron-as-a-service, or the engine behind AWS Lambda schedules, CI runners, or email-send jobs) is a favourite because it forces you to reason about reliability: run every job on time, exactly the right number of times, even when workers crash. It's a masterclass in queues, retries, and idempotency. Let's design it.
Step 1 — Requirements
- Functional: schedule a job to run once at time T or on a recurring schedule (cron); execute it; retry on failure; let users view status/history; cancel.
- Non-functional: run jobs on time (low delay), scale to millions of jobs, at-least-once execution (never silently drop a job), no thundering herd, survive worker/scheduler crashes.
Step 2 — The classes / data model
🟠A Job row: id, payload, schedule (cron or runAt), nextRunTime, status, retryCount. The key field is nextRunTime — the whole system is "find jobs whose nextRunTime ≤ now and run them." Index on nextRunTime so that query is cheap even with millions of rows.
Step 3 — Architecture
- Scheduler polls the DB every few seconds for jobs with
nextRunTime ≤ now, and enqueues them onto a message queue. - Workers (stateless, horizontally scalable) pull jobs off the queue and execute the payload.
- On success the job's nextRunTime is advanced (for recurring jobs) or it's marked done; on failure it's retried with backoff.
Decoupling scheduling from execution via a queue is the core move — it absorbs bursts and lets you scale workers independently.
Step 4 — Reliability: the hard part
🔴 This is where the question is really scored:
- Don't double-schedule: if two scheduler instances both pick the same due job, it runs twice. Fix with leader election (only one active scheduler via consensus/a lock), or have workers atomically claim a job (compare-and-set status PENDING→RUNNING) so only one wins.
- At-least-once + idempotency: queues guarantee at-least-once, so a job can run twice on retry. Make execution idempotent (dedup on job-run ID) so re-running is safe — turning "at-least-once" into effectively "exactly-once."
- Crash recovery / visibility timeout: if a worker dies mid-job, the queue re-delivers after a visibility timeout so the job isn't lost. Store a heartbeat/lease so stuck jobs are reclaimed.
- Retries & dead-letter: retry N times with exponential backoff, then move to a dead-letter queue for inspection.
Step 5 — Scaling
🔴 A single scheduler polling one table won't scale to billions of jobs. Shard jobs by ID or by time bucket, with a scheduler per shard. Avoid the thundering herd when thousands of jobs share the same run time by jittering execution. For very-far-future jobs, a time-bucketed design (hierarchical timing wheels, or a "next N minutes" hot table) keeps the due-query tiny. Partition workers by job type so a slow job class doesn't starve others.
The interview-ready walkthrough
"Jobs live in a store indexed by nextRunTime. A scheduler (single leader, or with atomic job-claiming) polls for due jobs and enqueues them; stateless workers pull from the queue and execute, then advance nextRunTime or mark done. Reliability is the crux: leader election / atomic claim to avoid double-runs, at-least-once delivery + idempotent execution for effectively-once, visibility timeouts so crashed jobs re-run, and retries with backoff into a dead-letter queue. I'd shard by time bucket and jitter to avoid thundering herds." That covers the pipeline and every failure mode they'll poke at."
What to read next
- Message Queues · Retries & Circuit Breakers — the execution backbone
- Consensus (Raft/Paxos) — leader election for the scheduler
- Design a Notification System — a close cousin (queue + workers)
- ← The complete System Design guide (hub)