Bringing Documents In

Adopting an Attachment

Everything else in this section applies to documents. A file uploaded through an ordinary file field on a record is not one — and on an existing installation there are thousands of them.

Where to find it

Architect Panel → Datastores, edit the datastore, and tick Attachments Become Documents. The work is done by the Adopt Uploads As Documents task, under Automation & Tasks.

What adopting gains you

Version history, restore, classification, an access log, content search, library columns, a review cycle, and disposal with the record. Nothing is copied and nothing moves: the upload field goes on working exactly as before, and the document is an additional way to reach the same file.

Why it is a task and not automatic

A file is uploaded against a field before the record it belongs to exists, which is why the association can only be worked out afterwards by reading the record. A sweep also catches files that arrived by any route — the record screen, the API, the mobile app — and is safe to run again.

The first run is a backfill

Which is why the task ships disabled and in preview mode. Ticking the box on a datastore with eleven years of attachments means adopting eleven years of attachments, and that is not something to arrange by accident.

Read the preview, for one specific reason

The preview counts what it would adopt and then samples storage to see how many of those files can actually be reached.

That matters more than it sounds. On any installation whose database has been copied from elsewhere, file rows routinely point at objects that are not in this installation’s storage. Adopting them fills the extraction queue with “the file could not be reached” and looks exactly like a broken system. Where everything sampled is missing, the preview says so in capitals.

Running it again creates nothing

A file that is already a document is recognised and skipped. The task runs hourly once armed, which is how a file uploaded this morning becomes searchable this afternoon.

Adopted files are queued, not read

Text extraction happens on its own schedule, because opening every file is a network fetch per document on S3 and does not belong inside a sweep holding a thousand of them.

It is per datastore, images included

A datastore of signature images or avatars should simply not turn it on. Indexing images needs OCR; adopting them without it gains version history and an access log and no searchability.

Worked example

A team ticks the box on their case datastore and runs the task in preview. It reports 4,180 attachments, of which 25 sampled files were all reachable. They enable it out of hours; the first run adopts the lot, the index catches up overnight, and content search covers eleven years of correspondence the next morning.

Recommendations

• Preview before arming, always, and read the storage line.
• Start with one datastore that matters, not all of them.
• Do not tick it on image-only datastores.
• Expect the first run to be long and the rest to be nothing.

The File Store Bridge

A file store file can become a document without moving. No bytes are copied and the store goes on working exactly as it did.

Where to find it

The File store tab on the Document Library, which appears only where you hold Read on at least one store. Pick a store, walk the folders, adopt a file.

The version history was always there

The reason this is worth doing. A file store writes a new version row on every upload and then serves exactly one of them, forever — so on an estate of any age there are years of superseded versions sitting in storage that no screen could open.

Adopting a file records each version it has, and the document’s history is how you open them. A tick box records the earlier versions as well as the current one.

What else it gains

Restore, classification, an access log, a review cycle, an owner, expiry, acknowledgements, and content search. What it keeps: its folders, its own permissions, and its own lifecycle.

Adopting does not widen access

The file store’s own Read permission is checked on every download, before the document’s. Somebody who can read the document but holds nothing on the store is still refused the file — two gates, both of which must pass.

That is why the picker only offers stores you can read: offering one you cannot would produce a document nobody could open.

It usually narrows access instead

Adopted without a record link — which is the normal case, since file store files are not attachments on a case — the document is readable by the person who adopted it and by grant, and by nobody else. Link it to a record to widen it.

Two things it refuses

• A share link. A link is entitlement with no login, and the store’s permission check needs security groups a token holder does not have.
• Destroying the file under retention. Disposal removes the document and leaves the file where it is — see Documents and Disposal.

Files already adopted are marked

With their document number, rather than hidden. Somebody looking for a file they adopted last week needs to see that it is there, not to find it missing.

One way only

The file store is not told that a file is now a document, so its own browser shows no sign of it. The relationship is deliberately one-directional.

Worked example

A policy library has lived in a file store for six years, with the current version of each policy served and four or five superseded ones kept. Twenty policies are adopted with their histories; the store is untouched, and for the first time somebody can open the 2021 version of the safeguarding policy to answer a complaint about 2021.

Recommendations

• Record the history when the old versions matter, and leave it off when they do not.
• Link adopted documents to a record if a team rather than a person needs them.
• Do not expect the store to change. It will not.

Reading Scans with OCR

A scan is a picture of text. Without OCR it is stored, served and permanently unfindable — and a scanned PDF is the worst case, because it parses perfectly and yields nothing.

Where to find it

OCR is configured in the installation’s configuration file rather than in Site Settings, beside the storage settings. It ships switched off. The document search status report names the provider and says whether it reads files here or elsewhere.

Three providers, and one is not like the others

• Tesseract — a program on your own server. Nothing leaves.
• AWS Textract — the file leaves this system.
• The AI gateway — a vision model, under its own provider role. The file leaves this system.

A classified document is not posted to a third party

The rule worth knowing before you choose. A classification that forbids export means “may be looked at, must not leave” — and sending the content to somebody else to be read is the most complete way of leaving there is.

So an off-site provider is refused those documents, and they are recorded as a failure naming the classification rather than skipped quietly. A scan that is unsearchable because of a rule is a decision somebody may want to revisit, and they can only revisit it if they can see it.

The same document goes through a local provider unimpeded. The rule is about where the file goes, not about OCR.

An unrecognised provider counts as off-site

A typo in one configuration line must not become the way classified scans reach a third party.

Scanned PDFs are retried

A PDF goes to the PDF reader first, which is right for every other PDF. Where that comes back with no words at all, and the provider can be handed a PDF, it is retried through OCR.

Not every provider can: Textract reads a PDF itself, a vision model is sent an image and cannot be handed one, and Tesseract needs the pages rendered first — which needs a second program installed.

It asks for a transcription and nothing else

A model told to “describe this document” writes prose about it, which then goes into the search index and is searched as though it were the document’s own words. Somebody searching for a phrase would find a paraphrase of it, or not find the phrase that is plainly on the page.

Empty is a failure, not a success

Consistent with the rest of indexing. Storing an empty extraction as done leaves a document that is permanently unfindable and looks indexed, which is worse than a visible failure.

Worked example

An organisation with twenty years of scanned correspondence installs Tesseract on its own server rather than using a hosted service, because a third of the archive is classified. The whole archive becomes searchable, nothing leaves the building, and the few documents that fail do so with a reason attached.

Recommendations

• Choose local if anything you hold forbids export. It is the only provider that can read everything.
• Install the PDF rendering program as well, or scanned PDFs stay unreadable.
• Check the character counts after a bulk run, exactly as for any other extraction.
• Read the status report to confirm which provider is live and whether it is off-site.