Scheduled SFTP transfers are common in reporting, billing, inventory, and partner integrations, but a reliable workflow needs more than a timed upload. It must select the right files, avoid duplicates, verify integrity, protect credentials, and leave an audit trail. A web cron job can trigger that workflow independently of the application server while a background worker performs the actual transfer.
Define the transfer contract
Document the source directory, destination path, filename pattern, expected frequency, time zone, maximum size, encryption requirements, retention, and who owns failures. Agree whether the receiver expects one file, a batch, or an atomic ready signal.
Keep the cron endpoint lightweight
The scheduled URL should authenticate the request, generate a run ID, enqueue eligible files, and return quickly. Network transfer belongs in a worker so the HTTP request does not time out. Protect the trigger using the guidance for secure web cron endpoints.
Prefer key-based authentication
Use a dedicated SFTP account with access only to required directories. Store the private key in a secrets manager, restrict its filesystem permissions, rotate it, and pin the server host key. Never place credentials or private keys in the cron URL, logs, filenames, or alerts.
Verify the host identity
Host-key checking prevents a connection from silently reaching an impostor. Provision the expected fingerprint through a trusted channel and treat unexpected changes as incidents. Planned server migrations should include an approved fingerprint update window.
Select only complete files
Producers should write to a temporary name and rename atomically when complete. The transfer worker should ignore partial extensions, very recent files, and files whose size is still changing. A manifest can define the exact batch when multiple files belong together.
Use deterministic file names
Include a business date, source identifier, and stable sequence when needed. Avoid spaces and locale-dependent timestamps. File names should help operators understand a batch without exposing customer names or other sensitive data.
Make every transfer idempotent
Create an idempotency key from the destination, logical file ID, and content checksum. Record it before upload and update the status transactionally. A repeated cron trigger should resume or confirm the same operation, not create another remote copy.
Prevent overlapping batches
Use a lock per partner or destination when file order matters. Expire abandoned locks carefully and record their owner. The patterns from preventing overlapping cron jobs apply directly to transfer queues.
Upload atomically
Send the file to a temporary remote name, verify it, then rename it to the final name. A receiver that watches the directory will never see an incomplete final file. Confirm that the server supports atomic rename within the same filesystem.
Verify integrity after upload
Compare file size and a cryptographic checksum when the protocol or partner workflow supports it. Otherwise download a small verification sample or use a signed manifest. Do not mark a transfer complete merely because the connection closed without an error.
Handle acknowledgements
Some partners place a receipt or result file in a return directory. Track upload completion and business acknowledgement as separate states. Set a deadline for the acknowledgement and alert when it is late even if the upload succeeded.
Retry only transient failures
Connection resets, short timeouts, and temporary server limits may be retried with exponential backoff and jitter. Authentication failure, host-key mismatch, or permission denial needs intervention. Use the limits in cron retry strategies.
Control concurrency and bandwidth
Limit simultaneous transfers per destination and globally. Large files can consume network capacity needed by the application. Use chunked or resumable transfers when supported, but verify the final artifact before publication.
Protect sensitive files
SFTP encrypts transport, not necessarily storage. Encrypt high-risk payloads before transfer, restrict local staging access, and delete temporary files on schedule. Log identifiers and checksums rather than confidential contents.
Build actionable alerts
Include partner, logical file ID, stage, attempt, run ID, error category, and next action. Never attach sensitive payloads or credentials. Group repeated failures for the same destination into one incident.
Monitor the complete pipeline
Track queue age, transfer duration, bytes sent, success rate, acknowledgement latency, and time since the last expected file. A successful scheduler does not prove that the partner received usable data. Apply cron monitoring practices to every stage.
Test failure and recovery
Test wrong fingerprints, expired keys, permission errors, dropped connections, partial uploads, duplicate triggers, checksum mismatch, missing acknowledgements, and worker restarts. The checklist for testing web cron jobs helps validate scheduler behavior too.
Launch with a practical checklist
- Document the file contract and ownership.
- Use least-privilege keys and host-key pinning.
- Select complete files and upload atomically.
- Deduplicate with stable identifiers and checksums.
- Retry transient faults and alert permanent ones.
- Track acknowledgements, retention, and audit history.
A dependable SFTP workflow treats scheduling, transfer, and business acceptance as separate states. With a small authenticated web cron trigger and a carefully designed worker, routine file exchange can remain secure, repeatable, and easy to diagnose.