Product feeds connect an online catalog to shopping engines, marketplaces, comparison sites, affiliates, and internal search tools. When a feed is stale, customers may see the wrong price, unavailable products, or links that no longer work. A scheduled web cron job can generate and publish fresh feed files without a manual export.
This guide explains how to build a reliable feed pipeline with incremental extraction, validation, atomic publishing, monitoring, and channel-specific rules.
Start with a canonical product model
Define one trusted product record before writing channel templates. Common fields include a stable ID, title, description, canonical URL, image URL, price, currency, availability, brand, category, identifiers, condition, language, shipping data, and last update time.
Keep source data separate from channel output. Each marketplace may rename fields or impose limits, but those transformations should not change the canonical catalog.
Decide between full and incremental exports
A full export reads every eligible product and produces a complete file. It is simple to audit but can be expensive for large catalogs. An incremental export processes products changed since a durable checkpoint and is faster, but it needs reliable update events and explicit deletion handling.
Many stores use frequent incremental feeds plus a daily full reconciliation. Large catalogs should follow the bounded techniques in processing large data sets in batches so memory and database load remain predictable.
Create a narrow export endpoint
Expose an authenticated HTTPS endpoint that accepts a known feed name, export mode, batch limit, and idempotency key. The endpoint should queue the export or process one bounded unit, not accept arbitrary SQL, templates, or destination paths.
Apply the controls from securing web cron endpoints: scoped credentials, rate limits, audit logs, HTTPS, and secrets that never appear in public URLs.
Filter products before serialization
Exclude drafts, deleted items, private products, unsupported destinations, missing landing pages, and records that violate a channel's required fields. Decide how to represent backorders, preorder dates, variant groups, sale prices, and tax-inclusive pricing.
Keep exclusion reasons in structured logs. A feed with fewer products is not automatically broken, but an unexplained drop should trigger review.
Validate every required field
Check IDs for uniqueness, URLs for valid HTTPS syntax, prices for currency consistency, stock states for allowed values, and image URLs for a supported format. Normalize whitespace and encoding without rewriting meaningful product text.
Validate the final XML, CSV, JSON, or TSV against the channel specification. Treat malformed output as a failed run and keep the last valid feed live.
Generate files in batches
Stream rows to a temporary file instead of building the full feed in memory. Use a stable ordering and store the last processed product ID or event position. Each batch should be safe to repeat after a timeout.
If transformation requires remote images or taxonomy lookups, cache bounded results and enforce short timeouts. Do not let one slow dependency block the entire catalog indefinitely.
Prevent overlapping exports
Two jobs writing the same feed can corrupt a file or publish an older snapshot after a newer one. Acquire a lock per feed and channel, then skip or queue duplicate starts. Record the owner, start time, and expiry.
The patterns in preventing overlapping cron jobs explain safe lock expiry and recovery when a worker stops unexpectedly.
Publish with an atomic file swap
Write to a versioned temporary name, validate it, calculate its size and checksum, then atomically move it to the public feed path. Readers should see either the previous complete feed or the new complete feed, never a partially written file.
Keep one or two previous versions for a short rollback window. Apply retention separately so cleanup cannot delete the active file.
Design channel-specific templates
A marketplace feed, an affiliate feed, and an internal search feed often need different fields and refresh rates. Version each template and test it with fixtures. Share canonical transformations, but keep channel rules explicit rather than filling unsupported fields with guesses.
When feeds trigger downstream webhooks or imports, use the scheduling guidance in HTTP requests and webhooks with web cron jobs to separate generation from delivery.
Retry only safe operations
Retry transient database or upload failures with bounded exponential backoff. Reuse the same export ID and checkpoint so a retry cannot publish duplicate or out-of-order versions. Do not retry validation failures until the source or template changes.
See cron retry strategies for practical timeout, backoff, and idempotency patterns.
Monitor feed freshness and quality
Track the last successful publish time, duration, product count, excluded count by reason, file size, validation errors, channel delivery result, and lock conflicts. Alert when freshness exceeds the business limit or counts change sharply.
Use the broader practices in cron job monitoring. A 200 response from the export endpoint is useful, but the real success signal is a valid, current feed at the expected destination.
Test a production-shaped catalog
- Include variants, sale prices, missing images, deleted products, and multiple currencies.
- Force a mid-batch failure and confirm that retry is safe.
- Start two exports and confirm that locking prevents overlap.
- Publish a malformed test file and verify that the live feed remains unchanged.
- Check rollback to the previous version.
- Compare exported counts with authoritative catalog queries.
Apply the checklist in testing web cron jobs before production to authentication, timeouts, logs, and failure notifications.
Practical takeaway
An automated product feed is dependable when it starts from a canonical catalog, validates before publishing, writes atomically, and measures freshness at the destination. Schedule the slowest cadence that meets channel requirements, then tune it from real export duration and update volume.