Loading

Introduction to Data Extraction

Data extraction is the platform's ingestion pipeline. It takes a structured file or payload arriving from outside, pulls named values out of it, transforms them, and writes them into a datastore.

Where to find it

Architect Panel → Data:

  • Data Extraction — the extraction definitions, their field mappings and transformations
  • Datastores — the destination the extracted data is written to

What it is for

Anywhere data arrives repeatedly in a machine-readable form and somebody would otherwise rekey it: a supplier's nightly stock file, a partner's CSV of referrals, an API posting orders, a monitoring system e-mailing an XML report.

What it is not

It does not read scanned documents. There is no optical character recognition here, no reading of invoices or identity documents from images, and no confidence scoring of recognised text.

This matters because the two are easy to confuse. Extraction expects data that is already structured — the value is in a known position or under a known path. If your source is a photograph of a delivery note, extraction is not the tool.

Three ways data arrives

  • API — a caller posts the payload to the platform.
  • E-mail — a message arrives at a monitored address, and its attachment is processed.
  • File upload — somebody uploads the file, or a routine places it.

The same extraction definition serves all three. The input method is recorded on each run, so you can tell how a particular batch arrived.

How a run works

  1. Data arrives by one of the three methods.
  2. The extraction definition identifies the format and reads it.
  3. Each mapped field is located by its path or column and pulled out.
  4. Transformations run against the extracted values.
  5. The result is written to the destination datastore.
  6. The run is logged, and an optional callback runs afterwards.

Everything is logged

Each run writes a log entry recording the input method and what happened. This is the first place to look when a nightly feed produces nothing — it will usually tell you the file arrived and a path did not match, which is a different problem from the file never arriving.

The destination can be elsewhere

A definition names its destination explicitly, including the host and database. Extraction is therefore usable for loading data into somewhere other than the local datastore, though the common case is much simpler than that.

Worked example

A distributor receives a supplier's stock file as CSV each night by e-mail. One extraction definition maps four columns — supplier part code, description, free quantity and price — into a staging datastore, converting the price from pence to pounds on the way. Nobody opens the e-mail; the log confirms 2,400 rows processed at 03:12.

Recommendations

  • Confirm your source is structured before choosing extraction.
  • Start with one file and a handful of fields, then widen.
  • Check the log after the first live run, and after any change at the sender's end.
  • Load into a staging datastore first where the data is not fully trusted.