Corpus design patterns

The engine matches against exactly what you index — which makes corpus design the main architectural decision of a self-hosted deployment. These are the patterns that work, built from the primitives in Bring your own corpus.

One instance, one corpus (the default)

Everything in one Postgres database; every check matches against all of it. Right for a single team with one body of reference material — a publication's archive, a company's content library. Zero extra moving parts.

Instance per boundary

When check results must never mix across a boundary — separate classes, separate clients, separate journals — run one engine instance (or at least one database) per boundary. The engine is deliberately single-tenant (NOPLAG_TENANT_ID exists only as a hook for layering your own tenancy), and a database per boundary is simpler and safer than building multi-tenancy above it:

• one compose stack per client with its own DATABASE_URL, or
• one Postgres server with a database per class, engine containers pointed at each.

Cost scales with boundaries, but so does isolation — a subpoena, deletion request, or bug in one corpus can't touch another.

The submitted-work pattern

For teaching and editorial workflows, the corpus is often prior submissions: index every submitted document, and each new submission is checked against all earlier ones. Two ways to build it:

• API-first — POST /v1/corpus/documents after each accepted submission. Byte-identical re-uploads return the existing document, so retries are safe.
• Folder-first — drop submissions into a directory tree and re-run scripts/ingest_folder.py on a schedule; re-runs are idempotent.

Check-then-index ordering matters: index a document after checking it, or it will match itself at 100%.

Reference-material pattern

The inverse: a fixed corpus of source material (textbooks, style guides, published articles, your own back catalog), checked against by many documents that are never themselves indexed. Bulk-load once with the folder ingester or your own pipeline via ingest_platform_document() / fingerprint_document(), then only re-ingest when the reference set changes.

Keeping a corpus clean

• Removal — DELETE /v1/corpus/documents/{id} for API-uploaded documents; SQL for bulk loads (deleting a documents row cascades to its chunks). Old reports keep their scores but lose the deleted source's rows.
• After every bulk change, run ANALYZE, and rebuild the document-frequency table once you're at the scale where it matters — see Operating at scale.
• Don't index junk. Boilerplate-heavy documents (cover sheets, forms) produce noisy matches everywhere. Curate what goes in; the engine is honest about what it finds, including the noise you fed it.