Bring your own corpus

The engine matches against what you index. A fresh install ships with a small Wikipedia sample corpus for the demo; real deployments replace it with their own documents.

Three ways to feed it

# a folder of .txt / .md / .pdf / .docx
DATABASE_URL=... python scripts/ingest_folder.py /path/to/documents
 
# one file over HTTP
curl -F "file=@paper.pdf" http://localhost:8000/v1/corpus/documents

For a custom pipeline, call ingest_platform_document / fingerprint_document from noplag_engine.workflows.corpus.

Corpus scopes

Two corpus scopes exist in the schema:

• rows with tenant_id set — uploaded through the API, manageable via GET/DELETE /v1/corpus/documents;
• platform rows with tenant_id NULL — bulk loads: the sample corpus, folder ingests.

Checks match against both.

Storage sizing

Chunk text is stored twice on purpose: chunks.fingerprints (narrow, GIN-indexed, what retrieval scans) and chunks_text (the text the aligner and report previews read). Expect roughly 1.5–2× the plain-text size in Postgres after indexing.

Removing documents

DELETE /v1/corpus/documents/{id} (API-uploaded docs), or plain SQL for bulk loads — deleting a documents row cascades to its chunks. Old check reports that referenced a deleted source lose those source rows (cascade), keeping their overall scores.

After every bulk load

Run ANALYZE (python -m noplag_engine.maintenance.analyze) and — once you're in the tens of millions of chunks — rebuild the document-frequency table. See Operating at scale.