Document AI development for real business files

Representation

Preserve the structure needed to explain every value

The canonical representation determines what the product can know and review. For a contract, heading hierarchy and defined terms may connect a clause to its scope. For an invoice, row and column coordinates connect quantity, unit price, tax and total. For a workbook, the formula and sheet context may matter more than displayed text. Store these relationships directly instead of hoping a later model can reconstruct them from flattened text.

Provenance should survive every transformation. A normalized date still points to the original characters and page region. A calculated value identifies its source fields and rule. A summary links to the passages behind each material statement. Keep the original file, processing version and intermediate representation under an appropriate retention policy. This lets a reviewer verify, an operator diagnose and an evaluation reproduce.

Evaluation

Measure the hierarchy of failure, not one accuracy score

Start evaluation before model selection. Split cases by document class, source, quality, language, layout and business difficulty. Test file acceptance, document classification, page completeness, text and layout recovery, field extraction, table relationships, cross-document matching, validation and end-to-end outcome. A field cannot be credited if it came from the wrong document or wrong entity.

Weight or report errors by business consequence without hiding rare critical cases in an average. Exact matching works for identifiers and permitted values; normalized comparison helps dates and amounts; relationship tests verify table position; expert rubrics handle clauses and findings. Maintain negative cases where the correct result is missing, conflicting or unsupported. Protect part of the set from everyday tuning and version expected answers with reviewer rationale.

• Report quality separately for every high-volume and high-risk document class.
• Test missing pages, duplicates, rotations, poor scans and mixed languages.
• Score source-location correctness as well as extracted value.
• Track critical errors independently from aggregate accuracy.
• Re-run downstream validation, not only extraction, after changes.

Review

Review should reduce uncertainty, not become manual re-entry

A reviewer needs the proposed value, original page region, surrounding structure, rule failures and the reason the item was routed. Jump directly to the evidence and keep zoom, page navigation and table context available. Group fields that must remain consistent. An amount, currency and period should be reviewed together rather than as three unrelated queue items.

Capture more than the corrected value. Record whether the cause was image quality, classification, layout, ambiguous source, model extraction, validation rule or changed business definition. Preserve both raw and accepted states. This makes product improvements targeted and prevents the same reviewer effort from recurring. Corrections enter evaluation or training data only with permissions, provenance and deliberate approval.

• Route by materiality and uncertainty rather than reviewing every field.
• Let users reject a document class, page set or relationship.
• Show alternatives without encouraging blind selection of the top score.
• Capture reason codes useful for engineering and operations.
• Keep reviewer identity and final approval in the case record.

Production

Extraction creates value only when the downstream result is correct

Define what the next system accepts and what authority the document result carries. A draft record may require review, while a validated low-risk class may proceed automatically. Apply schema, type, range, reference and duplicate checks before creating an external effect. Use idempotency keys and retain the destination identifier so retries do not create a second invoice, supplier or case.

Reconcile after integration. The accepted value in the review system, the payload sent and the record created should agree. If the output is a Word, Excel or PDF document, validate its required structure as carefully as extracted data. Monitor source changes because a new template, scanner or language mix can alter performance without any code release. Keep a safe manual path for unsupported or time-critical cases.

• Separate extraction confidence from permission to create a record.
• Validate business invariants before any downstream write.
• Make external effects idempotent and auditable.
• Reconcile accepted source values with created destination values.
• Treat output-document correctness as its own quality gate.