A nightly SFTP fetch imports half a file, repeats a file or stops after the supplier's server changes

Why a scheduled job imports half a file

A scheduled fetch looks in a directory and takes what is there. A supplier that uploads a large file over several minutes makes the file visible long before it is finished. If your job runs in that window it downloads and imports a truncated file, and a shrunken catalogue can look like a catalogue that has lost products. Nothing in SFTP says a file is finished; the supplier has to signal it.

There are four signals, in order of reliability. The supplier uploads under a temporary name and renames the file when it is complete, and the fetch only picks up final names; sftp has a rename command, so this works on a standard server, though you should ask the supplier to confirm how their own server behaves. The supplier writes a small marker or manifest file after the main file, ideally with its size and checksum. Your job waits until the size has stopped changing for an agreed time, which is weaker because an upload can pause. Or your job compares the size and checksum with the supplier's stated figures.

• Agree the signal with the supplier in writing and put it beside the layout description.
• Do not use resume to patch a partial file unless the partial copy is known to match the source; the OpenSSH manual warns that otherwise the result is likely to be corrupt.

Importing the same file twice, or yesterday's file again

A job with no memory fetches the newest file each run, so a supplier who misses a day causes yesterday's file to be imported twice. The cure is a small record of what has been imported: file name, size and a checksum, written only after a successful import. A file whose name and checksum are already recorded is skipped and logged as already imported, and a run that finds no new file by the agreed time raises an alert instead of finishing quietly. The older guides on freshness order and on full versus partial snapshots cover what to do with the content itself; this guide only decides whether a file should reach the importer at all.

A changed host key is a stop sign

SFTP runs over SSH, and the first time you connect your machine records the server's host key. If the server presents a different key later, the OpenSSH manual says the strict settings refuse the connection, and only turning checking off lets a changed-key host connect. The refusal has innocent causes, such as the supplier rebuilding or moving its server, and one serious one, someone impersonating the server. A script that is set to accept any key to make the error go away has given up the protection the check exists to provide.

The safe response is that a person confirms the new key's fingerprint with the supplier through a separate channel, then updates the recorded key. In a script, BatchMode turns off prompts, so the job fails instead of waiting; the failure must raise an alert that someone sees.

• An authentication failure is a different thing: it means the login was refused, not that the server's identity changed.
• Set connect and keep-alive timeouts so an unresponsive server cannot hold a run open for hours; ConnectTimeout and ServerAliveInterval exist for that, and the keep-alive default is off.

Other diagnoses to rule out

Before changing the job, rule out causes that look the same. The supplier may have moved the files to a new path or changed permissions. An IP allow-list on the supplier's side may no longer include your server after you moved it. The run time may be before the supplier publishes, which the guide on time zones and clock changes covers. The credentials may have been rotated. Each has a different owner, and none is fixed by a change to the completeness rule.

A safe first investigation

Read the last three runs in the job's log: file name, size and time of each import. Look for two runs with the same file, one much smaller than the others, or a long gap. Then, with your own credentials and not ours, list the supplier's directory by hand on a day the problem occurs and compare timestamps and sizes. Do not send credentials, host names you consider private or real files. A description of how the supplier names and uploads files, and what went wrong on which day, is enough to start.

How the paid job is accepted

The job sftp-scheduled-supplier-fetch-complete-files starts from £445, quoted after we know the completeness signal the supplier gives, the number of file patterns and how failures should be reported. It is tested against a synthetic SFTP server with invented files, never your supplier's real server or credentials. Acceptance needs five results: a file still being written is not fetched and the finished file is imported once; a second run with the same file imports nothing and logs it; a second run started while the first is still fetching imports nothing twice; a changed host key on the synthetic server stops the job with the agreed alert; and no new file by the agreed time raises the agreed alert. Applying the change with your own credentials stays with your maintainer. Prices are untested proposals, and payment follows the agreed checks and your sign-off. Nothing is booked or charged by an enquiry.