Vision Models for Invoices and Forms: Where Accuracy Breaks

A distributor in Indore buys from forty-odd suppliers, and their invoices reach the accounts team in very different shapes. The large FMCG (fast-moving consumer goods) companies email clean PDFs, regional suppliers send scanned printouts, and transporters send a WhatsApp photo taken in the loading bay with the invoice held flat on a carton. All of it has to end up in Tally as invoice number, date, line items, GST and total, and today someone keys it in by hand. A vision model can read most of these documents, but its accuracy depends on how each one was produced, and the places where it breaks can be named in advance.

What a document model does

A vision model, in this context, is a model that takes an image of a page and returns the information on it. For invoices and forms it has two jobs. The first is reading: turning pixels into characters, the step traditionally called OCR (optical character recognition). The second is understanding layout: working out that a particular number is the invoice total and not the subtotal, because of where it sits and what label is next to it. Some products do these as two separate stages, and newer models do both at once, but both jobs still have to happen. Document reading is one of the computer vision uses on our AI solutions page.

A few terms from Indian accounting will come up, so here they are defined once.

• GST is the goods and services tax. An invoice usually splits it into CGST and SGST (the central and state shares) or shows IGST for interstate sales.
• A GSTIN is the GST identification number printed on every registered supplier's invoice.
• An HSN code is the product classification code used for GST, printed per line item.
• A PO (purchase order) is the order your company raised before the supplier invoiced you.
• The vendor master is the list of approved suppliers held in your accounting system, with their GSTINs and bank details.

Four kinds of document that look alike

To the accounts team every one of these is "an invoice". To the model they are four different jobs, and the difference is in how the document was produced.

Document	What the model must do	Where it usually struggles
Digital PDF with a text layer	Assign existing text to fields	Unusual layouts, totals on page two
Scanned printout	Recognise characters from pixels first	Skew, speckle, faint print
Phone photo	Correct perspective, then read	Shadows, blur, a thumb in the corner
Print with handwriting or stamps	Separate overlapping marks	Pen corrections, a "PAID" stamp over the total

Sort incoming documents into these four groups before you measure anything, and keep the accuracy figures per group. A single average hides the fact that the model may be fine on PDFs and poor on photos. For a distributor like the one above, most of the volume could well be photos, so the average would describe the documents that matter least.

Where the reading goes wrong

Every supplier has its own template. The invoice number sits top right on one and under the logo on another, labelled "Inv. No", "Invoice #" or "Bill No" depending on who printed it. HSN codes and the CGST and SGST split land in different columns. Tables wrap long descriptions over two lines, and a two-page invoice puts the total on the page the model did not receive.

Pretrained document models cope with much of this, which is why a pilot on a handful of suppliers tends to look good. Failures cluster by layout, though, so keep an error count per supplier. A supplier whose invoices fail repeatedly needs either a few labelled examples for fine-tuning or a fixed template rule written just for them.

Handwritten digits are the common failure in amounts and dates: a 1 read as a 7, a 0 as a 6, a crossed 7 as a 4. Stamps and signatures overlap the fields they land on, and the model may read the stamp's text as the field's value. Faxed and low-resolution scans lose decimal points and thin characters, which is how 1,250.00 becomes 125000.

Some of this can be fixed before the model runs. Straighten and crop the page, check the resolution, and reject pages below a minimum, so that the person who took the photo retakes it on the spot rather than three days later. The rest is handled after the model, with checks.

How one invoice moves into Tally

Here is how it works, one document at a time.

1. The document arrives by email, scanner or WhatsApp.
2. Page checks look at resolution, skew and page count, and ask for a retake if needed.
3. The model extracts the fields, each with a confidence score.
4. Each field is checked against data finance already holds.
5. The invoice is either posted to Tally or sent to a person, with the failed fields marked.

1. 01Document arrivesPDF, scan or photo
2. 02Page checksresolution, skew, pages
3. 03Model extractsfields and scores
4. 04Record checksvendors, PO, sums, dates
5. 05Post or reviewTally or a person

Checks against existing records decide the route, not the model's score alone.

Step four is the one that makes the system trustworthy. The model's output is a claim, and finance already holds the data that can test most of those claims. The supplier name and GSTIN go against the vendor master. The PO number goes against open orders, and the invoice amount against what was ordered. Line items should add up to the subtotal, and subtotal plus GST should equal the total. The invoice date should fall in a plausible window, with the due date after it. The invoice number should not already be posted for that supplier, which also catches the duplicate a transporter sends twice because nobody replied to the first message.

The important part is the arithmetic. A single misread digit almost always breaks a sum, so the sum checks catch errors that the model's confidence score does not. The same pattern of checking extracted values against known rules applies to contracts, as described in extracting parties, dates and amounts from contracts.

Deciding which invoices a person sees

The score a model attaches to a field is a hint, not a calibrated probability, meaning a score of 0.9 does not reliably mean the field is right nine times in ten. So the route should combine that score with the checks, rather than rely on one threshold for every supplier.

Post without review

• Known supplier and layout
• Every check passed
• No handwriting or stamp on a field
• Amount under finance's limit

Send to a person

• Any check failed
• New supplier or layout
• Handwriting or stamp near a field
• Score below the field's threshold

The review screen is part of the system. It should show the document with the questioned field highlighted and the extracted value beside it, so the reviewer confirms or corrects one field instead of re-reading the page. If it shows only a form, review takes as long as typing the invoice in, and the accounts team will reasonably go back to typing.

Store every correction with the document it came from. Those corrections are labelled examples, and they are exactly what you fine-tune on when one supplier keeps failing. If your inputs also include emailed orders and statements, structured extraction from emails and PDFs explains how to validate the output against a schema.

Where invoice projects slip

The most common slip I have come across in write-ups is measuring on clean PDFs during the pilot and going live on phone photos. Close behind it is trusting the model's confidence with no check against the ledger. Review screens that show values without showing where on the page they came from are another. So are corrections that go straight into the ERP (enterprise resource planning system) without being attached to the document, which means the model never improves.

The last one is treating every field as equal. The total, the supplier and the invoice number must be right. A line description can be approximate, and reviewer time spent on it is time taken from the fields that matter.

Before go-live

1. Collect fifty real documents from each of the four groups, with the correct values recorded by hand.
2. Report accuracy per field and per document group, not as one number.
3. Write down the checks each field passes through, and which failure sends it to a person.
4. Build a review queue with the field highlighted and corrections stored against the document.
5. Keep a per-supplier error count that someone reads every month.

Start with step one, and expect it to take longer than planned, because the photos are often scattered across field staff's personal WhatsApp chats. Until you have them, any accuracy figure you have seen describes someone else's invoices.