Automatic File Import

An import routine watches a source for files and brings them in without anybody doing it by hand.

Where to find it

Architect Panel → Data:

• File Stores — the stores themselves, and View File Store on a row
• Large Uploads — in-progress and stalled upload sessions
• File Import Routines — watched sources and their mappings

Architect Panel → Activity:

• Error Log — where a failed import is recorded

What a routine defines

• The source and its type — where files arrive.
• Check frequency — how often it is swept.
• Limit per check — how many files are taken in one pass.
• Processed action — what happens to a file once handled.
• Error action — what happens to one that fails.
• Logging level.

Each routine also names where the data goes: the database, the table, and the mapping.

The limit per check is a safety valve

Without it, a source that suddenly receives ten thousand files is processed in a single pass, and everything else waits. A limit turns a flood into a queue that drains.

Set it to something the system handles comfortably rather than to the largest batch you can imagine.

Move processed files

The processed action is what stops a file being imported twice. Moving it to a done folder, renaming it, or deleting it — any of these, but not "leave it where it is and hope".

A routine that re-reads the same file every sweep is the commonest way a duplicate import happens.

Failures need somewhere to go

The error action decides where a file lands when it cannot be processed. A quarantine folder is much better than deletion, because the file is usually the evidence of what went wrong.

And somebody has to look at that folder. A quarantine nobody checks is a queue of lost data.

Frequency against latency

Sweep as often as the business actually needs. Every minute for orders arriving from a trading partner; hourly for a nightly export. Frequent sweeps of an empty source are cheap but not free.

Logging level is a real choice

Verbose logging is invaluable while setting a routine up and expensive once it is running thousands of files. Turn it up to diagnose, down to operate.

Watch the volume

A routine that normally imports twenty files and today imported two thousand is telling you something — a re-send from a supplier, a stuck process at their end, or a loop. It is worth noticing before the data is in.

Test with a bad file

Not just a good one. A truncated file, one with the wrong columns, one that is empty. Confirm each ends up in quarantine rather than partly imported.

Worked example

An organisation sweeps a supplier drop folder every ten minutes, taking fifty files per pass, moving processed files to a dated archive and failures to a quarantine folder reviewed each morning. A malformed file test confirmed nothing was partly imported.

Recommendations

• Always set a limit per check.
• Move processed files so they cannot be re-read.
• Quarantine failures and have somebody read the folder.
• Test with malformed files, not only good ones.