Setting Up an Extraction Rule

An extraction definition is the mapping between what arrives and where it goes. Most of the work is in the field mapping.

Where to find it

Architect Panel → Data:

Data Extraction — definitions, field mappings and transformations
Datastores — the destination datastore and its field names

What a definition holds

Name — how you will recognise it later.
Destination — the host, database and datastore to write into.
Field mappings — one row per field, pairing a source path with a destination field.
Transformations — ordered actions applied to extracted values.
Callback — an optional function run after a successful extraction.

Paths depend on the format

For JSON and XML, a path walks the structure — order/customer/reference descends through the nesting. For CSV, spreadsheets and delimited files, the source is a column.

Get the path from a real file, not from the sender's documentation. Documentation describes the intended shape; the file tells you the actual one, and the two differ more often than anybody expects.

Map the minimum first

Start with the fields you cannot do without, prove the pipeline end to end, then add the rest. A definition with forty mappings that fails somewhere is much harder to diagnose than one with four that works.

Transformations

Transformations are ordered actions applied to a value after it is extracted — trimming, reformatting, converting units, substituting codes. They run in sequence, so order matters: trimming after a length check is not the same as trimming before it.

Keep transformation to shape rather than meaning. Reformatting a date is a transformation; deciding which of two suppliers a code belongs to is business logic, and it belongs in the destination datastore's own rules where it is visible.

A missing path is an error, not a blank

When a mapped path does not exist in the incoming data, the run reports it rather than quietly writing nothing. That is the behaviour you want: a silently empty field is how a feed runs for a month before anybody notices half the data is missing.

It also means a sender changing their format breaks the run visibly, which is the earliest you could possibly find out.

Testing

Get a genuine file from the sender, not a hand-made sample.
Run it and read the log.
Check the destination records field by field for the first file.
Run a second, different file — ideally one with a missing optional value.
Only then connect the live feed.

Ask for a file with edge cases

Empty optional fields, a record with unusual characters, the longest value they will ever send. Senders will provide one if asked, and it is far better than discovering the limits in production.

Worked example

A referral partner posts JSON to the API. The definition maps referral/id, referral/person/name, referral/person/dob and referral/urgency into a referrals datastore, with a transformation normalising the date format and another mapping the partner's urgency words onto the internal scale. Three test payloads are run, including one with no urgency supplied, before the partner is given the live endpoint.

Recommendations

Build paths from a real file.
Map a few fields, prove the pipeline, then widen.
Keep transformations to formatting, not decisions.
Test with an edge-case file before going live.