Large CSV, XML, or API imports rarely belong in one HTTP request. A single run can exceed timeouts, lock tables, exhaust memory, and leave half-written data when it fails. A web cron job works best as a recurring coordinator that processes a bounded batch, saves progress, and safely continues on the next run.
Separate upload, validation, and import
Accept the source file or remote feed first, store it in a protected staging area, and create an import record with a unique ID. Validate the format, required columns, encoding, size, and schema version before any production rows change.
Keep credentials and source URLs out of the cron query string. Apply the authentication patterns in securing web cron endpoints to the worker trigger.
Choose a bounded batch size
Limit each run by rows, bytes, or elapsed time. The correct size depends on validation cost, database latency, and the scheduler timeout. Start conservatively, measure, and adjust without changing the logical result.
Commit after each safe batch instead of holding one transaction for the entire file. Short transactions reduce lock duration and make retries cheaper.
Persist a durable checkpoint
Store the source version, next offset or cursor, processed count, accepted count, rejected count, timestamps, and status in the database. Update the checkpoint in the same transaction as the imported rows whenever possible.
Do not rely only on a file pointer held in memory. A worker restart must be able to reconstruct exactly where processing should resume.
Make every row idempotent
Give each source record a stable external key and enforce the desired uniqueness in the database. Decide whether an existing record should be skipped, updated, versioned, or rejected. A retry must produce the same final state rather than duplicate data.
The techniques in cron retry strategies help separate transient failures from permanent validation errors.
Validate before writing
Normalize dates, numbers, identifiers, and text encoding explicitly. Reject impossible values with a structured reason. Avoid silently converting malformed data into plausible but incorrect records.
For cross-row relationships, stage normalized data first, then apply it in a controlled phase. This makes missing references and duplicates easier to report.
Prevent overlapping workers
Acquire an atomic lock per import ID before reading the next batch. A global lock may unnecessarily block unrelated imports, while no lock can let two workers claim the same checkpoint.
Use an expiration and safe recovery policy for abandoned locks. See preventing overlapping cron jobs for practical patterns.
Handle errors by category
Row-level validation failures should usually be recorded and skipped while the batch continues. Infrastructure failures—database outages, storage errors, or rate limits—should stop the batch without advancing the checkpoint past unfinished work.
Use exponential backoff for transient failures and a maximum attempt count. Send permanently failed records to a review queue with enough context to correct and replay them.
Report progress honestly
Expose states such as queued, validating, running, paused, completed, completed with errors, and failed. Progress should use processed source records, not only successful writes, so operators understand rejected rows.
Log batch duration, throughput, memory use, database time, checkpoint movement, and error counts. The monitoring signals in cron job monitoring help detect stalled or slowing imports.
Protect production performance
Run imports during appropriate windows, cap concurrency, use indexed lookup keys, and avoid expensive per-row queries. Preload reference maps or use bulk upserts where the database supports them.
Pause automatically when database latency or replication lag exceeds a safe threshold. An import is not successful if it makes the application unusable.
Test restart scenarios
Terminate a worker midway through a batch, retry the same checkpoint, change batch sizes, inject malformed rows, and simulate storage or database outages. Verify that counts remain correct and no record is duplicated.
Use the web cron testing checklist before enabling a live schedule.
Implementation checklist
- Create a durable import record and protected staging source.
- Validate format and schema before production writes.
- Process bounded batches under a per-import lock.
- Commit data and checkpoints consistently.
- Use stable keys and idempotent upserts.
- Separate row errors from infrastructure failures.
- Monitor throughput, stalls, and final reconciliation.
Batch imports turn an unpredictable long-running job into a sequence of small, observable steps. Durable checkpoints, idempotent writes, bounded transactions, and clear error handling make the workflow safe to retry and easy to operate.