Document indexing extracts the text from uploaded files (PDF, Word, Excel, slides, plain text) and stores it in the search index. After indexing, full-text search finds the document by its contents rather than just by filename.
How It Works
- File uploaded and stored.
- An index task is queued.
- Worker picks the file, runs the appropriate extractor:
- PDF: pdftotext (Poppler) or pdfminer.
- Word: docx / mammoth.
- Excel: openpyxl / xlsx2csv.
- PowerPoint: python-pptx.
- Plain text / HTML: read directly with UTF-8 detection.
- Scanned image / image PDF: Tesseract OCR (when enabled).
- Extracted text fed to the search engine (typically Elasticsearch or MeiliSearch).
- Document record updated with extraction status, length and a short preview.
OCR for Scanned Documents
Scanned PDFs and images contain no machine-readable text. With OCR enabled, the platform runs Tesseract over the rendered pages to recover text. This is slow (1-3 sec per page) so OCR is opt-in per upload field or per file type.
Worked Examples
- Internal contracts library: Every PDF indexed; lawyers search by contract content (“force majeure”) rather than filename.
- Public knowledge base: Index articles and embedded attachments so site search returns the right doc.
- Compliance archives: Scanned policy docs OCR'd nightly; auditors search across decades of paperwork.
- Resume database: Recruiters search CVs by keyword (“Python”, “team lead”).
Note: Don't index sensitive documents that shouldn't appear in general search results. Use the per-document “index = no” flag, or store sensitive material in a designated private path that the indexer skips.
