Broken links quietly damage user trust, waste crawl budget, and can turn a useful website journey into a dead end. A scheduled web cron job can check important pages automatically, detect failures early, and give your team a repeatable repair queue without requiring someone to run a manual audit every morning.
This guide explains how to design a practical broken-link monitoring workflow: what to crawl, how often to run it, which HTTP responses matter, how to avoid false alarms, and how to turn scan results into useful alerts.
Why broken links deserve scheduled monitoring
Links change for ordinary reasons. Products are removed, documentation is reorganized, campaign pages expire, domains migrate, and editors mistype URLs. A one-time audit only describes one moment. Scheduled checks reveal new failures soon after they appear and make link quality an operational process rather than an occasional cleanup project.
The goal is not to crawl every URL every minute. A good monitor focuses on business-critical paths, respects server capacity, and records enough context to help someone fix the problem quickly.
Choose the right pages to scan
Start with a small, high-value seed list: your homepage, navigation hubs, pricing or product pages, support pages, and the most visited articles. Add XML sitemap URLs when the scanner can enforce a safe page limit. Exclude login flows, cart steps, faceted search combinations, calendar archives, and URLs containing one-time tokens.
- Tier 1: revenue, signup, download, and account-help pages checked daily.
- Tier 2: evergreen content and category pages checked two or three times per week.
- Tier 3: large archives checked weekly in rotating batches.
For very large inventories, use the same batching principles described in our guide to processing large data sets with web cron jobs. Store a cursor so each run continues from the previous batch instead of restarting at page one.
Define what counts as a broken link
A link checker should evaluate more than a simple connection failure. Record the source page, destination URL, link text, HTTP method, final URL, response code, response time, and number of redirects.
- 404 and 410: usually genuine broken destinations.
- 500–599: server failures that may be temporary and should be retried.
- 401 and 403: not always broken; the destination may intentionally require authentication or block automated clients.
- 301 and 308: valid permanent redirects, but internal links should eventually point to the final address.
- 302 and 307: review repeated temporary redirects, especially when they persist for weeks.
- 200 with an error page: a soft 404 that needs content-based detection.
Build a safe checker endpoint
Your scheduled endpoint should accept a limited workload, load the next batch of source pages, extract only allowed HTTP and HTTPS links, and request destinations with strict timeouts. Use a descriptive user agent, follow a modest redirect limit, and prevent requests to private IP ranges so submitted links cannot turn the checker into an internal-network probe.
GET /maintenance/check-links.php?batch=20
Authorization: Bearer Protect the endpoint using the controls in our web cron endpoint security guide. Keep credentials out of public URLs when possible, validate every request, and return a concise machine-readable result.
Prevent overlapping scans
A slow site or an unusually large batch can outlive the schedule interval. Use a database advisory lock or file lock so a second run exits cleanly while the first is active. The implementation pattern in preventing overlapping cron jobs applies directly to link scans.
Make each batch idempotent. If a retry processes the same page twice, it should update the existing observation rather than create duplicate incidents.
Use retries without hiding real failures
Retry network timeouts and 5xx responses after a short delay, but do not repeatedly hammer a failing destination. A useful policy is one immediate retry followed by confirmation in the next scheduled run. Open an incident only when the same destination fails twice or when a critical internal link returns a definitive 404 or 410.
For detailed timing and backoff patterns, see cron job retry strategies. Separate transient transport problems from persistent content problems in reports.
Design alerts people can act on
A notification should say what failed and where it was found. Include the source page, destination, response code, first-seen time, latest check, and suggested action. Deduplicate alerts by source and destination, and send a recovery notice when the link works again.
Send urgent alerts for broken checkout, signup, download, and primary-navigation links. Put low-priority archive issues into a daily digest. This keeps the channel useful and prevents alert fatigue.
Create a repair workflow
Classify each confirmed issue into one of four actions: update the destination, remove the link, restore the missing page, or add a relevant permanent redirect. Fix internal links at the source even when a redirect exists; this reduces redirect chains and preserves a cleaner information architecture.
Track ownership by section or content type. A report that routes documentation failures to the documentation team and product-page failures to merchandising gets resolved faster than a single unowned spreadsheet.
Schedule the job at a sensible frequency
Daily checks are appropriate for most high-value websites. Hourly scans are usually unnecessary unless content changes constantly or links control time-sensitive transactions. Weekly scans can cover long-tail archives. Use common cron expression examples to choose a schedule, then test the endpoint manually before enabling automation.
Measure the health of your link graph
Useful metrics include pages scanned, unique destinations checked, confirmed broken links, recovered links, redirect chains, median response time, and mean time to repair. Trend these numbers instead of judging a single run. A falling backlog and shorter repair time show that the workflow is improving.
Final checklist
- Start with critical seed pages and capped batches.
- Distinguish permanent failures, transient errors, redirects, and soft 404s.
- Use strict timeouts, safe URL validation, authentication, and overlap protection.
- Retry selectively and confirm failures before opening noisy incidents.
- Send deduplicated alerts with source-page context and ownership.
- Repair the source link, not only the destination redirect.
Once the checker endpoint is ready, schedule it with FreeCronJob and review the first few runs closely. A small, reliable daily scan is more valuable than an aggressive crawler that creates noise or puts unnecessary load on your site.