Benchmarks

The engine is evaluated with the standard plagdet metric on PAN-PC-11; the harness lives in the repo's eval/ directory.

Reference: Potthast, M., Stein, B., Barrón-Cedeño, A., & Rosso, P. (2010). An Evaluation Framework for Plagiarism Detection. COLING 2010, 997–1005.

Read the numbers with the scope in mind

• On verbatim and lightly-obfuscated material (the harness's synthetic PAN-style set, and the verbatim portion of PAN-PC-11), plagdet lands around 0.8+ with precision ≥ 0.95.
• On the full PAN-PC-11, which is dominated by heavy paraphrase, summarization, and cross-language obfuscation, an L0/L1 engine scores near zero by design — fingerprints only match text that is actually reused near-verbatim. That gap is exactly what the roadmap's paraphrase and semantic layers exist to close.

Do not read any single plagdet figure as general-purpose detection accuracy, and when reporting numbers from this harness, always state which partition and obfuscation mix they were computed over.

Reproducing the numbers

Fetch the corpus (see eval/datasets/pan_pc_11/README.md in the repo — the corpus is several GB and not committed), point DATABASE_URL at a scratch Postgres with the engine migrations applied, then from the repo root:

python eval/run_eval.py --part test
python eval/run_eval.py --part test --output metrics.md

eval/build_pan_subset.py can carve a smaller stratified partition (symlinks, no copy) for faster parameter sweeps:

python eval/build_pan_subset.py --part subset --verbatim 150 --obfuscated 150 --negative 100
python eval/run_eval.py --part subset

Without the corpus or a reachable Postgres, run_eval.py prints a [NO CORPUS] placeholder and exits 0.