Design a Job Scheduler

Problem Statement

Design a distributed job scheduler that supports one-time and recurring (cron-like) jobs. The system must guarantee at-least-once execution, handle worker failures with retry and exponential backoff, support job priorities, and scale to millions of scheduled jobs with thousands of concurrent workers.

Requirements

Functional

• Schedule one-time jobs at a specific timestamp or recurring jobs via cron expressions
• Execute jobs on a pool of distributed workers with priority-based ordering
• Retry failed jobs with configurable exponential backoff (max 3 retries by default)
• Provide job status tracking: PENDING, RUNNING, SUCCEEDED, FAILED, RETRYING

Non-Functional

• Accuracy: Jobs fire within 1 second of their scheduled time
• Reliability: At-least-once execution guarantee -- no silently dropped jobs
• Scale: 10M scheduled jobs, 100K jobs/minute execution throughput
• Fault tolerance: System continues operating if individual workers or scheduler nodes fail

Core Architecture

1. Scheduler Service -- Polls the job store every second for jobs whose next_fire_time <= now. Uses a distributed lock (Redis SETNX) to ensure only one scheduler instance picks each job. Enqueues the job into the appropriate priority queue. For recurring jobs, computes the next fire time from the cron expression and updates the store.

2. Priority Queue (Kafka/Redis) -- Three priority tiers: HIGH, MEDIUM, LOW. Workers consume from HIGH first. Kafka topics partitioned by priority level, or Redis sorted sets with priority as score. Provides backpressure: if workers are overwhelmed, jobs wait in the queue rather than being dropped.

3. Worker Pool -- Stateless workers that pull jobs from the queue, execute them (HTTP callback, gRPC call, or shell command), and report results back. Each worker sends heartbeats every 10 seconds. If a worker dies mid-execution, the job is re-enqueued after a visibility timeout (like SQS).

4. Dead Letter Queue and Alerting -- Jobs that exhaust all retries are moved to a DLQ for manual inspection. Alerting fires for: DLQ size exceeding threshold, job execution latency spikes, and worker pool capacity dropping below minimum.

Database Choice

PostgreSQL for the job definitions table: job_id, cron_expression, next_fire_time (indexed), payload, status, retry_count, priority. The next_fire_time index enables efficient polling (WHERE next_fire_time <= NOW() AND status = 'PENDING'). Redis for distributed locks (scheduler leader election), rate limiting, and as an optional fast queue for high-priority jobs. Kafka for the durable job queue when at-least-once delivery is critical.

Key API Endpoints

POST /api/v1/jobs
  -> Body: \{ name: "daily-report", cron: "0 9 * * *", callback_url: "https://...", priority: "HIGH", payload: \{\} \}
  -> Returns: \{ job_id: "J-100", next_fire_time: "2024-01-01T09:00:00Z" \}

GET /api/v1/jobs/\{job_id\}/status
  -> Returns: \{ job_id: "J-100", status: "SUCCEEDED", last_run: "...", next_run: "...", retry_count: 0 \}

DELETE /api/v1/jobs/\{job_id\}
  -> Returns: \{ deleted: true \}

Scaling Insight

The critical scaling challenge is the thundering herd problem at minute boundaries -- many cron jobs fire at :00 seconds (e.g., "every hour" jobs). Solution: add a configurable jitter (0-30 seconds) to each job's fire time. This spreads 100K jobs scheduled for 9:00:00 across the window 9:00:00-9:00:30, reducing peak queue pressure by 30x and preventing worker pool saturation.

Key Tradeoffs

Practical Implementation for .NET Developers

In a .NET application, you would typically implement this pattern using the following approach:

ASP.NET Core setup: Create a service class that encapsulates the logic, register it with dependency injection, and inject it into your controllers or minimal API endpoints. The built-in DI container handles lifecycle management.

Entity Framework Core: For database interactions, EF Core provides the ORM layer. Use migrations for schema management and raw SQL for performance-critical queries. Consider Dapper for read-heavy paths where EF Core's overhead matters.

Azure integration: If deploying to Azure, leverage managed services — Azure Cache for Redis, Azure SQL, Azure Service Bus, Azure Cosmos DB. These eliminate operational overhead and provide built-in monitoring through Application Insights.

Testing: Use xUnit with Testcontainers for integration tests that spin up real databases in Docker. Mock external dependencies with NSubstitute. The WebApplicationFactory class lets you test your entire HTTP pipeline in-process.

Monitoring: Add Application Insights telemetry to track request latency, dependency calls, and custom metrics. Use structured logging with Serilog to make production debugging possible:

Log.Information("Processing order {OrderId} for {CustomerId}", orderId, customerId);

This gives you searchable, structured logs in Azure Monitor or Seq.

System-Specific Clarifying Questions

Before designing Job Scheduler, ask questions specific to THIS system:

1. Who are the primary users? Understanding the user base shapes every technical decision — consumer apps have different requirements than enterprise B2B systems.
2. What is the read-to-write ratio? This determines whether you optimize for fast reads (caching, denormalization) or fast writes (write-ahead logs, async processing).
3. What is the geographic distribution? Users in one country vs. global users fundamentally changes your data replication and CDN strategy.
4. What is the acceptable latency? Some features need sub-100ms responses, others can tolerate seconds. This determines your caching and architecture strategy.
5. What is the consistency requirement? Some data (payments, inventory) needs strong consistency. Other data (social feeds, recommendations) can be eventually consistent.

Architecture Deep Dive

The architecture for Job Scheduler should be designed around the specific access patterns of the system. Do not apply generic templates — every system has unique hotspots, bottlenecks, and scaling challenges.

Write Path: How does data enter the system? Is it bursty (event-driven, flash sales) or steady (sensor data, logs)? Bursty writes need queuing and backpressure. Steady writes can go directly to the database.

Read Path: How is data consumed? Is it fan-out (one write, many reads like social feeds) or point lookups (one read for specific data like user profiles)? Fan-out reads benefit from pre-computation and caching. Point lookups benefit from efficient indexing.

Hot Spots: Where are the bottlenecks? For Job Scheduler, identify the component that will fail first under load and design mitigation strategies: caching, sharding, rate limiting, or async processing.

Sources

• Design a Job Scheduler -- Reference
• Source: System-Design-Overview