Document Indexing

Indexing Documents

Uploaded documents can have their text extracted and stored, so the contents are searchable rather than only the filename.

Where to find it

Architect Panel → Data:

• Documents — the documents themselves

Architect Panel → Configuration:

• Site Settings — the File Uploads and Audio/Video Conversion groups

What is recorded

For each document and version: the extracted text, how many characters and pages it produced, which extractor was used, the state of the extraction, any error, and when it happened.

So an extraction is a tracked operation rather than a side effect, and you can tell the difference between a document with no text and one that failed.

A missing extractor refuses rather than pretends

The important behaviour. Text extraction depends on libraries, and where the one needed for a format is not installed the platform says so instead of extracting nothing and reporting success.

That distinction is the whole point: indexing an empty string and calling it done leaves a document that is permanently unfindable and looks indexed.

Character and page counts are the health check

A hundred-page document that extracted forty characters did not extract properly. The counts tell you that immediately, and nothing else does — the document appears indexed and simply never matches a search.

Indexing is a search feature, not a storage one

The document is stored either way. Indexing decides whether somebody can find it by its contents, which for a document library is most of its value.

It has a cost

Extracted text is stored alongside the document, so a large library roughly doubles in one sense. For most organisations that is worth it; for a very large archive of files nobody searches, it may not be.

Extracted text is the document’s content

Which means it carries the same sensitivity. A contract’s extracted text is the contract, and permissions on searching should reflect permissions on reading.

A search that returns snippets is a search that discloses content.

Re-index after a version

Extraction is per version, so a replaced document needs extracting again. A search matching the previous version’s text is a subtle and confusing failure.

Check a sample

Upload one of each format you expect, search for a phrase from the middle of each, and confirm it is found. That is the only proof indexing works for your actual documents.

Worked example

An organisation indexes its contract library and checks character counts after bulk uploads. A batch showing near-zero counts turned out to be scanned images rather than text documents, which needed a different extractor — visible immediately rather than discovered by a failed search months later.

Recommendations

• Check character and page counts after bulk uploads.
• Treat extracted text as the content for permissions.
• Re-index new versions.
• Test a search per format you accept.

What Can Be Indexed

Six extractors between them cover the document formats the platform can read.

Where to find it

Architect Panel → Data:

• Documents — the library being indexed

The extractors

• PDF.
• Word and OpenDocument — .docx, .doc, .odt, .rtf.
• Spreadsheets — .xlsx, .xls, .ods, .csv.
• Presentations — .pptx, .ppt, .odp.
• Plain text, HTML and e-mail — .txt, .html, .htm, .eml, .md, .json, .xml.
• Scanned images, through optical character recognition.

Availability is checked, not assumed

Each extractor depends on a library, and the platform checks whether it is present. Where one is missing, extraction for those formats is refused with a stated reason rather than silently producing nothing.

Only plain text is always available, because it needs no library.

Scanned documents are a separate problem

A scanned page is an image. Ordinary extraction reads nothing from it, so it needs optical character recognition, which needs a configured provider.

If your library is largely scans, that is the capability that matters, and it is worth confirming before assuming documents are searchable.

A PDF is not necessarily text

The common misunderstanding. A PDF produced from a word processor contains text; one produced from a scanner contains an image of text. They look identical to a person and completely different to an extractor.

The character count is how you tell them apart.

Recognition is not perfect

Optical recognition produces approximately the right text. Good enough to find a document; not good enough to rely on for a figure or a reference number without checking the original.

Check what your library actually contains

Before deciding anything. A sample of fifty documents, their formats and their character counts, tells you whether your archive is searchable and what would make it so.

Ask for text where you can

A supplier sending a text-based PDF instead of a scan is a change worth requesting once, and it removes the recognition problem entirely for everything that follows.

Say what is searchable

Users assume everything is. Somebody who cannot find a document concludes the search is broken rather than that the document is an image, and they stop trusting the search.

Worked example

A sample of a document library found a third were scanned PDFs with near-zero character counts. Recognition was configured for those, and the supplier producing most of them was asked to send text-based files instead — which removed the problem for everything received afterwards.

Recommendations

• Sample your library before assuming it is searchable.
• Character count distinguishes a text PDF from a scan.
• Ask suppliers for text rather than scans.
• Tell users what is and is not searchable.