A background queue can look healthy while work quietly piles up. This guide shows how to monitor queue depth, oldest-job age, and worker throughput with a web cron job, then turn those measurements into alerts that help operators act before customers notice.
The approach works for email queues, image processing, exports, webhook delivery, billing tasks, and other asynchronous jobs. It complements scheduled API health checks by measuring whether useful work is actually moving, not merely whether an endpoint returns HTTP 200.
Why queue backlog monitoring needs its own check
A queue is designed to absorb short bursts, so a non-zero depth is not automatically a failure. The real warning signs are sustained growth, old items that never complete, falling throughput, or workers that have stopped reporting. A good monitor therefore evaluates several signals together instead of alerting on one arbitrary number.
- Queue depth: how many jobs are ready or waiting.
- Oldest-job age: how long the longest-waiting item has been delayed.
- Throughput: how many jobs completed during a recent interval.
- Failure rate: how many attempts ended in retry or permanent failure.
- Worker heartbeat: when each worker last confirmed that it was alive.
Choose thresholds from service expectations
Start with the delay users can tolerate. If password-reset email should arrive within two minutes, alerting only when 10,000 messages accumulate is too late. An oldest-job threshold of 60 to 90 seconds is more meaningful. For a nightly analytics export, a larger queue may be normal, while a job older than the reporting deadline is not.
Use a warning threshold for early investigation and a critical threshold for paging. Add a recovery threshold below the warning level so the alert does not flap when values hover at the boundary. When traffic varies by hour, compare depth with recent throughput or use separate peak and off-peak thresholds.
Expose a small, safe monitoring endpoint
Create a read-only endpoint such as /ops/queue-health. It should query summary counters, never return job payloads, customer data, credentials, or raw exception traces. A compact response can contain the queue name, waiting count, oldest age in seconds, recent completion rate, failed count, worker count, and an overall state.
{
"status": "warning",
"queue": "email",
"waiting": 184,
"oldest_age_seconds": 78,
"completed_last_5m": 420,
"failed_last_5m": 3,
"active_workers": 4
}Protect the endpoint with a scoped token or network restriction and use HTTPS. Apply the same principles described in securing web cron endpoints: least privilege, constant-time secret comparison, rotation, and no secret in logs.
Schedule the web cron check
Run the check frequently enough to detect a breach before the user-facing deadline. A one-minute interval is suitable for time-sensitive queues; five minutes may be enough for batch workloads. Keep the endpoint fast and place an upper bound on its database queries so monitoring cannot become a new source of load.
Configure the web cron request to expect a success code only when the monitor completed its evaluation. A healthy queue can return a small JSON body with status: healthy. If the queue is degraded, return a controlled non-2xx status or let the endpoint notify the incident channel through your existing alert integration. Do not make the monitor process queued work itself.
Detect trends, not only snapshots
A depth of 500 may be harmless if workers are draining 1,000 jobs per minute. The same depth is serious when throughput is zero. Store a short history of measurements and calculate whether the queue is growing across consecutive checks. Alert when backlog age and growth agree, or when throughput drops below a minimum for several runs.
Require two or three consecutive warning samples for noisy workloads, but page immediately when no workers are alive or a hard deadline is crossed. This small persistence window filters harmless bursts without hiding a real outage.
Prevent duplicate monitors and alert storms
Only one monitor instance should evaluate and notify for a queue at a time. Use a short distributed lock or an atomic database lease, release it in a finally path, and set an expiry so a crashed check cannot block future runs. The pattern is similar to preventing overlapping cron jobs.
Give each incident a stable fingerprint such as queue name plus failure class. Send one opening alert, periodic reminders at a controlled cadence, and one recovery notice. Include the current values, thresholds, first-seen time, dashboard link, and runbook link.
Build an actionable recovery runbook
- Confirm whether producers are creating an unusual burst or workers have slowed.
- Check worker heartbeats, deployment changes, database latency, and downstream APIs.
- Pause only the failing producer when bad jobs are multiplying.
- Scale workers gradually while watching CPU, memory, database connections, and retry traffic.
- Move poison jobs to a dead-letter queue rather than retrying forever.
- Verify that oldest-job age and queue depth decline for several checks before closing the incident.
A monitor should diagnose, not perform destructive recovery automatically. Purging a queue, replaying payments, or increasing concurrency can cause data loss or duplicate side effects and should remain an explicit operational decision.
Test the monitor before relying on it
In staging, stop one worker and confirm that backlog age rises, the warning opens after the intended persistence window, and the message contains usable evidence. Restart the worker and verify that the recovery notice fires only after the recovery threshold is met. Also test an empty queue, a temporary traffic spike, a database timeout, malformed metrics, and an expired authentication token.
Track the monitor itself with request duration and last-success time. The techniques in testing web cron jobs before production help ensure the check fails clearly rather than reporting a false healthy state.
Implementation checklist
- Measure depth, oldest age, throughput, failures, and worker heartbeat.
- Set warning, critical, and recovery thresholds from real service expectations.
- Expose only aggregated, non-sensitive metrics through HTTPS.
- Run a lightweight check at a cadence that fits the queue deadline.
- Use consecutive samples, a lock, and alert deduplication.
- Attach a safe, reviewed runbook and test both failure and recovery.
With these pieces in place, a web cron job becomes an effective independent observer for asynchronous systems. It catches the quiet failure mode that ordinary uptime checks miss: the application is online, but the work is no longer moving.