Test whether a whole document is ready for work

Overview

A document extractor can score well while leaving your team unable to use its output. The gap appears when several individually plausible fields must be correct together. Before choosing a confidence threshold, run a small experiment that measures the complete business record, not just isolated fields.

The following experiment is hypothetical. Its numbers illustrate a decision method; they are neither customer results nor a product benchmark.

Set the acceptance rule before opening the results

Imagine a distributor preparing supplier invoices for an accounts-payable reviewer. A record is ready only when supplier identity, invoice number, currency, total and purchase-order reference agree with the document and the relevant master records. Passing this test does not authorize payment. Existing approval controls remain in place.

Select 100 invoices without reusing training examples. Have a knowledgeable reviewer label the required fields, then resolve ambiguous labels with a second person. Include ordinary documents and difficult cases, but preserve their categories so an intentionally difficult sample is not mistaken for the normal production mix.

Original experiment table: a hypothetical test composition, not a recommended universal sample size.

Freeze the extraction version, normalization rules and test documents. Decide what constitutes a match: for example, harmless whitespace may be ignored, but a currency difference cannot. Keep source text alongside normalized values so reviewers can reconstruct disputed decisions.

Read two scores side by side

Google’s Document AI documentation describes precision, recall and F1 against annotated documents, including individual-label metrics. It also notes that labels without suitable annotations or extraction support can be absent from aggregate metrics. A strong aggregate therefore needs a coverage check. Google Cloud, verified October 29, 2026.

Our additional business measure is whole-record readiness: all required checks pass, with no unresolved ambiguity. Do not estimate it by multiplying field scores; errors can cluster on the same damaged document.

Original results table: 20 field errors are distributed across 16 documents. Field correctness here is a custom count, not Google’s product metric.

The difference matters operationally. Sixteen records still need attention even though only twenty fields are wrong. Track missing outputs and processing failures in the document denominator; excluding them would make the workflow look easier than it is.

Compare routing rules without touching live payments

Use a separate development sample to select a proposed rule, then evaluate that frozen rule on the held-out set. In this hypothetical result, the rule sends 70 records forward and holds 30. Independent review discovers two incorrect records among those sent forward and fourteen among those held.

Original routing matrix: 2 of 70 forwarded records contain errors; holding 30 records catches 14 of the 16 incorrect records.

That is a business conversation, not a pass certificate. Are the two escaped errors harmless descriptions or wrong currencies? Can staff review thirty records within the promised turnaround? Would mandatory supplier matching catch an error that confidence alone missed?

Keep the raw counts visible. One hundred documents cannot establish safety for rare, expensive failures. Before expanding, test additional suppliers and languages, inspect high-impact error types separately, and repeat the evaluation after material model or layout changes. The useful deliverable is a defensible routing decision with known limits, not a single impressive percentage.