Pagination bugs can silently drop or repeat records even when every API request returns HTTP 200. A small scheduled monitor can follow an entire result set, validate page boundaries, and alert you before incomplete data reaches reports, search indexes, billing jobs, or customer-facing features.
Why pagination needs its own monitor
API uptime checks prove that an endpoint responds, but they do not prove that a paginated collection is complete. Offset changes, unstable sorting, expired cursors, concurrent writes, and incorrect next-page links can create gaps or duplicates. These failures are easy to miss because each individual response still looks healthy.
A pagination monitor tests the behavior of the collection as a sequence. It should confirm that every page can be reached, that record identities remain unique, and that the traversal ends exactly once. Pair this with your existing scheduled API health checks so transport health and data integrity are covered separately.
Define the invariants before writing the check
Start with rules that should remain true for one complete crawl:
- Every returned record has a stable unique ID.
- No ID appears on more than one page.
- The sort key moves in the expected direction.
- A cursor or next-page URL never repeats.
- The reported total, when available, matches the number of unique records collected.
- The final page has no further cursor and the crawl stays below a safe maximum page count.
For offset pagination, also record the requested offset and actual item count. For cursor pagination, store a hash of each cursor rather than logging the raw value. This keeps diagnostic logs useful without exposing sensitive request data.
Build a bounded pagination probe
Create a small endpoint or command that requests the first page, follows the API-provided continuation value, and keeps a set of IDs already seen. Stop immediately when an ID, cursor, or page signature repeats. Add strict limits for total pages, execution time, and response size so a broken continuation link cannot create an endless job.
$seenIds = [];
$seenCursors = [];
$cursor = null;
for ($page = 1; $page <= 100; $page++) {
$result = fetchPage($cursor);
foreach ($result['items'] as $item) {
if (isset($seenIds[$item['id']])) {
throw new RuntimeException('Duplicate record detected');
}
$seenIds[$item['id']] = true;
}
$cursor = $result['next_cursor'] ?? null;
if ($cursor === null) {
break;
}
$cursorHash = hash('sha256', $cursor);
if (isset($seenCursors[$cursorHash])) {
throw new RuntimeException('Cursor loop detected');
}
$seenCursors[$cursorHash] = true;
}Return a short machine-readable summary such as unique record count, page count, elapsed time, and a pass or fail status. Avoid returning the full dataset. Your web cron service only needs a clear success signal and enough context to route an alert.
Choose a stable test window
Live collections change during a crawl. Reduce false alarms by requesting an immutable snapshot when the API supports it, filtering to records created before a fixed cutoff, or sorting by a stable compound key such as creation time plus ID. If the API exposes a snapshot token, reuse it for every page in the run.
For high-volume collections, monitor a representative bounded window more frequently and run a full traversal less often. Coordinate the schedule with API rate-limit monitoring so the integrity check does not exhaust the same quota used by production traffic.
Schedule the monitor safely
Run the check often enough to detect regressions before downstream jobs consume the collection. A 15-minute interval may suit a busy operational API, while a nightly crawl may be enough for a catalog. Set the cron request timeout slightly above the normal traversal duration, but below the point where runs could overlap.
Use a lock or idempotency guard described in the guide to preventing overlapping cron jobs. Retry transient network or 429 responses with bounded backoff, following the principles in cron retry strategies. Do not retry a deterministic duplicate or cursor-loop failure; it needs investigation rather than more traffic.
Alert on actionable evidence
An alert should include the endpoint name, failed invariant, page number, safe record identifiers, unique count, and comparison with the previous successful run. Track gradual changes too. A sudden drop in collected records may indicate a missing page even when no direct duplicate appears.
When an alert fires, preserve the cutoff or snapshot token, reproduce the traversal outside the production consumer, and compare boundary records between the last good page and the first bad page. This makes it much faster to distinguish an API regression from normal concurrent writes.
Pagination monitoring checklist
- Use stable ordering and a fixed cutoff or snapshot.
- Track unique IDs and continuation values.
- Cap pages, time, and response size.
- Separate transient transport failures from integrity failures.
- Return a compact health summary.
- Schedule with locks, timeouts, and quota awareness.
- Keep a history of counts for trend detection.
A web cron job turns pagination integrity into a repeatable production check. That closes an important gap between “the API is online” and “the API returned every record exactly once.”