Transparent evaluation
A defensible benchmark measures the complete review path—not only whether a model found a name in extracted text.
Use a versioned matter containing PDFs, scans, images, email families, office documents, spreadsheets, CSV data, duplicates and inaccessible exceptions. Synthetic data avoids exposing client information and permits repeatable expected outcomes.
Record target text, page or cell location, category, rationale, document-family context, expected production decision and whether human review is mandatory. Hash the manifest and source files.
Record application revision, prompts, model versions, OCR processor versions, configuration, regional endpoints and timestamps. Results without version evidence cannot be reproduced or compared fairly.
A benchmark run should fail closed when the configured model, OCR service or output schema cannot provide a valid answer. Deterministic guessing must not be counted as successful AI review. Partial runs should identify completed documents, failed documents, retry state and every production blocker.
Report raw counts alongside percentages. A high aggregate accuracy can conceal a complete failure on spreadsheets, scans, faces, signatures, privilege or document families.
Publish the corpus version, manifest hash, test date, environment, limitations and full metric definitions. Separate observed test results from product claims and never use real customer content without explicit authority.
Use the legal redaction QA checklist to validate individual productions.