External Data Sources

Connecting to a Data Source

External data sources let the platform read — and where you allow it, write — data that lives somewhere else, without copying it in by hand.

Where to find it

Architect Panel → Integration & Connections:

• Data Sources — the console — connect, test, map objects
• Data Source Sync History — what each sync run did
• Data Source Writeback Queue — changes queued to go back out

Architect Panel → Automation:

• Tasks — External Data Source Sync and Writeback, both every 5 minutes

The nine drivers

• PostgreSQL, MySQL / MariaDB, Microsoft SQL Server / Azure SQL — direct SQL connections.
• MongoDB — via a connection URI.
• Snowflake — with key-pair authentication.
• Salesforce — by connected app or username-password flow.
• GraphQL endpoint and REST API — for services rather than databases.
• Google Sheets — via a service account.

Despite the section being called external databases, more than half of these are not databases. A REST API or a spreadsheet can be mapped onto a datastore exactly as a SQL table can.

Each driver declares its own fields

Rather than one generic connection string, every driver publishes the fields it actually needs — host, port, database and SSL mode for PostgreSQL; account, warehouse, role and a private key for Snowflake; a spreadsheet ID and a service account key for Sheets.

This is why the form changes when you change the driver, and why the prompts match the vocabulary of the system you are connecting to rather than a lowest common denominator.

Credentials are held encrypted

Every credential field — passwords, tokens, client secrets, private keys, connection URIs — is stored encrypted rather than in plain text.

Use a purpose-made account

Create an account on the remote system for this connection alone, with the narrowest rights that do the job: read-only unless you are writing back, and scoped to the specific tables or objects you need.

Reusing a person's credentials means the integration breaks when they leave, and it makes the remote system's audit trail attribute the platform's activity to them.

TLS

Each driver exposes its own transport security setting, and there is a certificate verification option on the connection. Turn verification on. Encryption without verification protects against passive interception but not against connecting to the wrong server, which is the harder problem.

Limits

Set the query timeout and the maximum live rows deliberately. They exist so a slow or enormous remote query degrades one screen rather than the platform, and the defaults are a starting point rather than an answer.

Test before mapping

The connection records its last test, the resulting status and any error. Get a clean test before you map a single object — every subsequent problem is easier to diagnose when the connection itself is known good.

Worked example

A finance team connects a read-only PostgreSQL account to their reporting warehouse, with SSL mode set to verify-full and a 15-second timeout. The test succeeds, the DBA confirms the account can see only two schemas, and object mapping starts from there.

Recommendations

• One purpose-made remote account per connection, read-only by default.
• Enable certificate verification.
• Set timeouts and row limits rather than accepting defaults.
• Get a clean test first, always.

Where the Data Source Lives

Where the remote system runs decides how it should be used. The same connection that is comfortable for live reads in one deployment is unusable in another.

Where to find it

Architect Panel → Integration & Connections:

• Data Sources — the connection, its timeout and row limit
• Data Source Sync History — run durations, which reveal latency problems

Same region

The remote system runs in the same cloud region as the platform. Round trips are a few milliseconds, there are no egress charges between them, and network rules are simple.

This is the only pattern that comfortably supports live reads on an interactive screen.

Another region

The connection works, but every query pays the round trip — typically tens to a couple of hundred milliseconds. That is tolerable for a scheduled sync and painful for a screen that issues several lookups while somebody waits.

Where the source is in another region, prefer syncing on a schedule and reading the local copy.

On-premises

Common for established line-of-business systems. The platform reaches the customer's network over a site-to-site VPN.

Plan for the link being down sometimes — a VPN that drops for ten minutes overnight is unremarkable, and a design that assumes constant availability will produce alerts nobody can act on. Scheduled syncing tolerates this naturally; live reads do not.

Software as a service

Salesforce, Google Sheets, and services behind REST or GraphQL. These are reached over the internet, and their own rate limits and concurrency rules apply.

Respect those limits when setting sync intervals. A five-minute sync against an API with a modest daily quota will exhaust it, and the failure arrives as a throttling error hours after the change that caused it.

Choosing live or synced

The decision follows from the above more than from preference:

• Live — same region, low latency, data that must be current to the second.
• Synced — anything remote, anything intermittent, anything rate-limited, and anything you want to report on.

When in doubt, sync. A local copy is faster, survives the remote system being unavailable, and can be indexed and joined; the cost is that it is as current as the last run.

Network security

• Allowlist the platform's egress addresses at the remote firewall rather than opening the service broadly.
• Require TLS, with certificate verification, unless the traffic is inside a controlled private network.
• Give the connection its own credentials and rotate them on a schedule you actually keep.
• Separate read and write accounts where writeback is enabled.

Watch the sync durations

Sync history records how long each run took. A run time that is creeping upwards is the earliest warning of a source outgrowing the arrangement, and it is much easier to act on before the run stops fitting in its interval.

Worked example

A customer's on-premises SQL Server is reached over an IPsec VPN, with a read-only account opened by their DBA. Because the link occasionally drops overnight, the objects are configured to sync hourly rather than read live, and the sync history is checked weekly — where a run time rising from 40 to 90 seconds over two months prompted an index on the remote table.

Recommendations

• Live reads only for same-region sources.
• Sync anything remote, intermittent or rate-limited.
• Allowlist egress addresses rather than opening the service.
• Review sync durations as an early warning.

Mapping Objects onto Datastores

A connection gives access. Mapping is what turns a remote table, Salesforce object, sheet or endpoint into something the platform can use like any other datastore.

Where to find it

Architect Panel → Integration & Connections:

• Data Sources — the console — map objects and their fields
• Data Source Sync History — each run, its row count and any error
• Data Source Writeback Queue — outbound changes awaiting delivery

What a mapped object holds

• Remote object and its namespace — the table, object or path.
• Local datastore — where it appears.
• Mode — read live, or sync into a local copy.
• Primary key — how a remote row is identified.
• Cursor field and type — what makes a sync incremental.
• Sync interval.
• Write enabled and delete mode.
• Field mappings — remote name and type to local name and type.

Include only the fields you need

Each field is included or excluded individually. Excluding what you will not use makes syncs faster, the local datastore comprehensible, and — where the remote system holds personal data you have no need for — reduces what you are responsible for holding.

That last point is worth taking seriously. Syncing a customer table in full because it was easier than choosing columns means you now hold, and must protect and dispose of, data you never needed.

The primary key must be right

It is how the platform decides whether an incoming row is new or an update to one it already has. Get it wrong and a sync either duplicates everything on each run or overwrites unrelated records.

Use the remote system's own stable identifier. A composite of "name plus postcode" will look correct until somebody corrects a spelling.

Incremental syncs need a cursor

A cursor field — typically a last-modified timestamp or an ascending ID — lets each run collect only what changed since the last one. The last position reached is remembered between runs.

Without a cursor, every run is a full reload. That is fine for a few hundred rows and unworkable for a few million, and it is the difference between a sync that finishes in seconds and one that does not finish inside its interval.

Check the remote field is genuinely updated on every change. A last-modified column that some updates bypass produces a sync that silently misses records — the hardest failure here to detect, because everything appears to work.

Writeback

Write is off unless you enable it, and there is a separate delete mode. Changes queue and are delivered by the writeback task.

Enable it only where the platform genuinely owns the data, and think hard about deletes. A delete propagated to a system of record is the least recoverable thing in this feature, and "we can restore it" is a conversation with somebody else's DBA.

Set the interval to the need

Match it to how quickly the data must be current, not to how quickly it could run. A five-minute sync of a table that changes twice a day is load with no benefit — and against a rate-limited API it is a quota problem waiting to happen.

Watch the sync history

Each run records its start, end, status, row count and any error. A run count that suddenly drops to zero usually means the cursor has advanced past a gap or a remote permission changed — worth a look before somebody reports missing data.

Worked example

A Salesforce account object is mapped to a local datastore with eleven of its forty fields, keyed on the Salesforce ID, cursored on last-modified date, syncing every 30 minutes and read-only. Reporting joins it to local case data without touching Salesforce. When the sales team later needs a status written back, writeback is enabled for that one field with deletes disabled.

Recommendations

• Map only the fields you need.
• Use the remote system's own identifier as the key.
• Always configure a cursor for anything beyond a few hundred rows.
• Leave writeback and deletes off until there is a specific reason.