Job Scheduler Design — Preventing Duplicate Execution

Every production system you've ever used is secretly running a job scheduler behind the scenes. GitHub Actions triggers your CI pipeline. Netflix re-encodes video in background workers. Your bank sends monthly statements at 2 AM. Uber's surge-pricing model re-trains on fresh data every few minutes. None of these happen because someone clicked a button — they happen because a scheduler decided it was time, found a free worker, handed the job over, and made sure it finished. Scheduling is the silent engine of the internet.

The problem a job scheduler solves is deceptively simple: 'run this thing at this time.' But the moment you add scale, reliability, and fairness requirements, the surface area explodes. What happens when the machine running the scheduler dies? What if a job takes ten times longer than expected? How do you stop one noisy tenant from starving everyone else? What if the same job fires twice because of a clock drift? These aren't hypotheticals — they're Tuesday in any company running infrastructure at scale.

By the end of this article you'll be able to walk into a system design interview and confidently sketch a distributed job scheduler from first principles. You'll understand how to choose between push and pull delivery, how to build a reliable delay queue using sorted sets, how to design idempotent workers, how to handle retries with exponential back-off, and how to reason about exactly-once execution guarantees — and why that last one is almost always a lie.

Core Architecture: From Delay Queues to Distributed Execution

A job scheduler isn't just a timer; it's a state machine. To design one that won't lose data, you need four distinct components: an API for job submission, a Metadata Store (Postgres or DynamoDB) to track job states, a Delay Queue (Redis Sorted Sets are the industry standard here) to handle the 'waiting' room, and a Worker Pool to do the heavy lifting.

In a distributed setup, multiple Schedulers poll the Delay Queue. When a job's execution time matches the current timestamp, the Scheduler moves the job from the 'Delayed' set to an 'Active' queue (like RabbitMQ or Kafka). Workers then pull from this queue. To prevent 'Zombie Jobs' (tasks that are assigned but never finish because a worker crashed), we implement a visibility timeout—if the worker doesn't send an 'ACK' within 5 minutes, the job is re-queued.

In production, ensure that the scheduler's polling interval is less than the visibility timeout to avoid latency. Set the visibility timeout based on the 99th percentile job execution time plus a buffer.

package io.thecodeforge.scheduler;

import java.util.UUID;
import java.time.Instant;

/**
 * Production-grade Job Submission Logic.
 * Note the use of Idempotency Keys to prevent duplicate submissions.
 */
public class JobController {
    
    public JobResponse scheduleJob(JobRequest request) {
        String jobId = UUID.randomUUID().toString();
        long scheduledTime = Instant.parse(request.getStartTime()).getEpochSecond();
        
        // In the Forge, we use Redis ZSET for delay management
        // ZADD delay_queue <scheduledTime> <jobId>
        System.out.println("Job [" + jobId + "] added to Redis ZSet at priority: " + scheduledTime);
        
        return new JobResponse(jobId, "ACCEPTED");
    }

    public static void main(String[] args) {
        JobController forgeScheduler = new JobController();
        JobRequest emailJob = new JobRequest("2026-03-15T14:00:00Z", "SEND_EMAIL");
        forgeScheduler.scheduleJob(emailJob);
    }
}

class JobRequest { 
    private String startTime; 
    private String type; 
    public JobRequest(String t, String type) { this.startTime = t; this.type = type; }
    public String getStartTime() { return startTime; }
}

class JobResponse { 
    private String id; 
    private String status; 
    public JobResponse(String id, String s) { this.id = id; this.status = s; }
}

Handling Scale: Partitioning and Fault Tolerance

When you have 10 million jobs per second, a single Redis instance becomes a bottleneck. We solve this by sharding the Delay Queue based on a Job_ID hash. Furthermore, we use a distributed locking mechanism (like Redlock or Zookeeper) to ensure that only one Scheduler instance processes a specific time-slice of the queue at once, preventing duplicate job firing.

For worker reliability, we utilize a 'DLQ' (Dead Letter Queue). If a job fails 3 times after exponential back-off, we move it to the DLQ for manual intervention rather than retrying forever and wasting CPU cycles.

In practice, use Redis Cluster for horizontal scaling of the delay queue. For sharding, assign each scheduler a hash slot range to avoid lock contention.

version: '3.8'
services:
  redis-delay-queue:
    image: redis:7.2
    container_name: forge-delay-node
    command: ["redis-server", "--save", "60", "1"] # Persist for reliability
    ports:
      - "6379:6379"

  worker-service:
    image: io.thecodeforge/scheduler-worker:latest
    environment:
      - REDIS_HOST=redis-delay-queue
      - MAX_RETRIES=3
    deploy:
      replicas: 5 # Horizontal scaling in action

Worker Lifecycle and Idempotency Design

Workers must be designed to tolerate failure mid-execution. The job status in the metadata store (e.g., 'PROCESSING') is updated before work begins, and a heartbeat mechanism refreshes a 'leases' record in Redis. If the heartbeat stops, another worker can pick up the task after the lease expires. Idempotency is achieved by including a deterministic job ID in all external side effects (e.g., DB insert or API call). The worker checks if the effect already occurred before performing it again. This is especially critical for financial operations where duplicate charges are unacceptable.

A common pattern is to use a 'dedup' table keyed by job ID. On each execution attempt, the worker runs a conditional insert. If the insert fails (unique constraint violation), the work has already been done, and the worker can safely skip the job.

package io.thecodeforge.scheduler;

import redis.clients.jedis.Jedis;
import java.util.UUID;

public class Worker {
    private static final String DEDUP_PREFIX = "dedup:";
    private static final String LEASE_PREFIX = "lease:";
    private static final int LEASE_TTL_SECONDS = 120;

    public boolean tryAcquireLease(String jobId) {
        try (Jedis jedis = new Jedis("redis-delay-queue")) {
            String result = jedis.set(LEASE_PREFIX + jobId, workerId, "NX", "EX", LEASE_TTL_SECONDS);
            return "OK".equals(result);
        }
    }

    public boolean isAlreadyProcessed(String jobId) {
        try (Jedis jedis = new Jedis("redis-delay-queue")) {
            return jedis.exists(DEDUP_PREFIX + jobId);
        }
    }

    public void markProcessed(String jobId) {
        try (Jedis jedis = new Jedis("redis-delay-queue")) {
            jedis.setex(DEDUP_PREFIX + jobId, 86400, "1"); // expire after 1 day
        }
    }

    public void processJob(Job job) {
        if (!tryAcquireLease(job.getId())) {
            return; // another worker has the lease
        }
        if (isAlreadyProcessed(job.getId())) {
            return; // job already executed successfully
        }
        // execute the actual task
        try {
            performWork(job);
            markProcessed(job.getId());
        } catch (Exception e) {
            // job fails – lease will expire allowing retry
            throw e;
        }
    }

    private void performWork(Job job) { 
        // actual work here 
    }
}

class Job { 
    private String id; 
    public String getId() { return id; } 
}

Handling Retries, Backoff and Dead Letter Queues

When a worker fails to complete a job, the scheduler must retry. Simple immediate retries can cause thundering herd. The standard approach is exponential backoff with jitter. For each job, track the retry count in the metadata store. After each failure, the scheduler increments the count and reschedules the job at time = now + (base_delay * 2^attempt) + random_jitter. After a maximum number of retries, the job moves to a Dead Letter Queue (DLQ) – a separate queue or table that requires manual inspection. Monitoring the DLQ depth is essential to detect systemic failures.

Choose the base delay and maximum retries based on your job's latency tolerance. For critical jobs, you might retry 5 times with a base delay of 30 seconds. For batch jobs, you might retry 3 times with a base delay of 5 minutes.

package io.thecodeforge.scheduler;

import java.time.Instant;
import java.util.Random;
import redis.clients.jedis.Jedis;

public class RetryScheduler {
    private static final int BASE_DELAY_SECONDS = 30;
    private static final int MAX_RETRIES = 5;
    private static final String RETRY_COUNT_KEY = "retries:";
    private static final String DLQ_KEY = "dlq:jobs";

    public void handleFailure(String jobId, Jedis jedis) {
        int retryCount = getRetryCount(jobId, jedis);
        if (retryCount >= MAX_RETRIES) {
            jedis.lpush(DLQ_KEY, jobId);
            System.out.println("Job [" + jobId + "] moved to DLQ after " + retryCount + " retries.");
            return;
        }
        long delay = (long) (BASE_DELAY_SECONDS * Math.pow(2, retryCount) + new Random().nextInt(1000));
        long newScheduledTime = Instant.now().getEpochSecond() + delay;
        jedis.zadd("delay_queue", newScheduledTime, jobId);
        jedis.incr(RETRY_COUNT_KEY + jobId);
        System.out.println("Job [" + jobId + "] rescheduled in " + delay + "s (attempt " + (retryCount+1) + ")");
    }

    private int getRetryCount(String jobId, Jedis jedis) {
        String val = jedis.get(RETRY_COUNT_KEY + jobId);
        return val == null ? 0 : Integer.parseInt(val);
    }
}

Monitoring and Observability for Job Scheduler

A job scheduler is a black box until a job doesn't run. Instrument every component: (1) Queue depth and lag: how many jobs are waiting and how long they've been delayed beyond their scheduled time. (2) Worker pool utilization: percentage of workers busy, queue backlog. (3) Job completion rate: processed jobs per second, success/failure ratio. (4) Retry count distribution: number of jobs on first attempt vs multiple retries. (5) Scheduler health: check that the scheduler process is alive and polling. Use Prometheus metrics and Grafana dashboards. Also export job-level logs with a structured format (job ID, execution time, result) to a central log aggregation system.

Define SLOs: e.g., 99% of jobs start within 5 seconds of their scheduled time. Alert on any deviation. Use distributed tracing to correlate scheduling events with worker execution.

package io.thecodeforge.scheduler;

import io.prometheus.client.Counter;
import io.prometheus.client.Gauge;
import io.prometheus.client.Histogram;
import io.prometheus.client.exporter.HTTPServer;

public class MetricsExporter {
    static final Gauge queueDepth = Gauge.build()
        .name("scheduler_queue_depth")
        .help("Number of jobs in delay queue.")
        .register();

    static final Counter jobsProcessed = Counter.build()
        .name("jobs_processed_total")
        .help("Total jobs processed.")
        .labelNames("status")
        .register();

    static final Histogram jobLatency = Histogram.build()
        .name("job_execution_latency_seconds")
        .help("Time from scheduled time to job start.")
        .buckets(0.1, 0.5, 1.0, 2.0, 5.0, 10.0)
        .register();

    public static void main(String[] args) throws Exception {
        HTTPServer server = new HTTPServer(1234);
        // In a real scheduler, call update methods periodically
        queueDepth.set(245);
        jobsProcessed.labels("success").inc(1423);
        jobsProcessed.labels("failure").inc(12);
        System.out.println("Metrics server running on port 1234");
    }
}

Frequently Asked Questions