A website search index becomes stale whenever products, articles, documentation, or support records change faster than the index is rebuilt. Visitors then see missing pages, outdated titles, or results that point to deleted content. A scheduled web cron job can keep search data current without relying on a person to run an indexing command.
This guide explains how to automate search index updates safely, choose between incremental and full rebuilds, prevent overlapping runs, and publish a new index without interrupting search.
Define what the search index contains
List every searchable content type and its source of truth. Record the stable document ID, title, body, canonical URL, language, category, visibility, update timestamp, and any ranking fields. Exclude drafts, private records, expired products, and pages blocked from public search.
Keep extraction rules in one versioned service instead of duplicating them in the scheduler. The cron job should start a controlled indexing workflow; it should not contain business logic in a long query string.
Choose incremental updates or a full rebuild
An incremental job reads records changed since a trusted checkpoint and applies inserts, updates, and deletes. It is fast and economical for large sites, but it needs reliable change timestamps or an event log. A full rebuild creates the entire index from current data and is easier to reason about, but it may require more time and memory.
Many sites use both: small incremental runs throughout the day and a full reconciliation during a quiet period. Large rebuilds should follow the bounded approach in our guide to processing data in batches with a web cron job.
Create a narrow indexing endpoint
Expose an internal HTTPS endpoint that accepts a mode such as incremental or reconcile, a bounded batch size, and an idempotency key. Authenticate it with a scoped secret header or signed request. Follow the protections in securing web cron endpoints, including rate limits, audit logs, and secrets that never appear in public URLs.
Return quickly if the endpoint only queues work. Use HTTP 202 with a job identifier for asynchronous processing, or a clear 2xx response for a small synchronous batch. Return an honest error status when work cannot start.
Use a durable checkpoint
Store the last successfully indexed change position in the database, not in browser memory or a temporary file. The checkpoint may be a monotonic event ID, a database sequence, or a timestamp paired with a stable tie-breaker ID.
Advance it only after the destination confirms the batch. If a job fails halfway through, restart from the previous checkpoint and rely on idempotent upserts. This avoids silently skipping records.
Process deletes explicitly
Removing a page from the database does not automatically remove it from a separate search engine. Keep tombstone events or a deletion log long enough for the indexer to consume them. For a periodic full rebuild, compare authoritative IDs with indexed IDs before switching the new index live.
Test renamed slugs and canonical redirects as well as hard deletes. A search result should never send a visitor to a permanent 404 when the content moved.
Prevent overlapping indexing jobs
Two rebuilds running together can compete for database connections, overwrite checkpoints, or switch aliases in the wrong order. Acquire a distributed lock with an expiry before starting. If another run owns the lock, exit cleanly and report that the job was skipped.
Use the practical patterns in preventing overlapping cron jobs. Choose a lock lifetime longer than a normal run, refresh it only while the worker is healthy, and record the lock owner for incident review.
Build new indexes away from live traffic
For a full rebuild, write to a versioned index such as site-search-2026-08-08-01. Validate document counts, sample queries, mappings, languages, and required fields before exposing it. Then atomically move a stable alias from the old index to the new one.
Keep the previous version for a short rollback window. Delete old versions with a separate retention policy only after the new index proves healthy.
Set a schedule based on freshness needs
A news site may need five-minute incremental updates, while a documentation site may be fine with hourly changes and a nightly reconciliation. Estimate how long a run takes and leave enough space for normal variance. Do not schedule a full rebuild more frequently than it can complete.
When business hours or local publishing windows matter, apply the guidance in scheduling cron jobs across time zones so daylight-saving changes do not create gaps or duplicate runs.
Retry small units, not entire rebuilds
Retry transient network errors with bounded exponential backoff. Preserve the checkpoint so a failed batch can run again without duplicating documents. Do not restart a multi-hour rebuild from zero because one request timed out near the end.
Our retry strategy guide covers backoff, timeouts, and idempotency. Pause and alert after repeated mapping errors, authentication failures, or malformed source data.
Monitor freshness and search quality
Track the age of the newest successful checkpoint, documents read, documents indexed, deletes applied, batch duration, failures, lock conflicts, and alias switches. Add synthetic queries for a known recent item and a known removed item.
Use the wider practices in cron job monitoring to define alerts. A green HTTP response is not enough if the index is still hours behind the source.
Test the workflow before production
- Run against a copy of production-shaped data.
- Verify incremental inserts, edits, deletes, and slug changes.
- Force a retry and confirm that documents are not duplicated.
- Start two runs and confirm that the lock blocks overlap.
- Validate a versioned rebuild before switching its alias.
- Rollback to the previous index and check that search remains available.
The checklist in testing a web cron job before production is useful for endpoint authentication, timeout behavior, logs, and failure alerts.
Practical takeaway
A scheduled search index update is reliable when source changes are explicit, batches are idempotent, checkpoints are durable, rebuilds are isolated, and freshness is measured from the user's perspective. Start with the slowest schedule that meets the content requirement, then tune it from real indexing data.