36 held-out real invoices: what we measured, and what we didn't

In the last post I said I wouldn’t publish an accuracy number without the method, the sample size and the uncertainty behind it. Here are the first two. They’re narrow on purpose.

Those two sentences are the whole claim. They cover one kind of invoice from one year, and they aren’t an accuracy figure for invoices in general. The rest of this post explains where they come from and what they leave out.

What we measured

The invoices are US television stations’ bills for political advertising from 2020, published in the FCC’s public inspection files and catalogued by project-deepform (opens in a new tab). They’re real and public, and many are hard: scans, dense airtime tables, invoices that run across several pages.

The held-out invoices were never used to tune extraction. We ran them three times through the production extraction code of 12 October 2026. Our benchmark runner called that code directly rather than going through the public API.

How we measured

The labels, the answers each result is scored against, were made in steps:

• Two AI labelers worked independently from the PDF pages only, without seeing each other’s work or Temlavo’s output.
• An adjudicator settled every disagreement against the PDF.
• I checked a sample of documents by hand.
• For supplier names, written text wins over logo lettering: a logo counts only when no name is written on the invoice.

A field counts as correct when it matches the label. Ignoring case, spacing and punctuation doesn’t change the result, so the misses aren’t just formatting.

The 95% interval comes from resampling whole invoices, not single fields, because fields on the same invoice aren’t independent. The three runs reused the same OCR text, so their spread shows how much the extraction itself varies.

What the numbers don’t show

• One domain: 2020 US TV-station political-advertising invoices. Modern commercial layouts aren’t in it.
• A small held-out set, which is why the interval is wide.
• Labels that are AI-produced and spot-checked, not certified by people.
• No OCR variation, because the runs shared OCR text.
• The held-out invoices’ combined scores were seen in two test runs earlier that day, though never used to change anything. Publishing them now means they aren’t held out anymore, so the next round needs new documents.

What still went wrong

OCR sometimes drops a totals block at the bottom of a page. On a couple of invoices, extraction named the advertiser instead of the agency being billed. Sometimes a $0.00 tax comes back where no tax is printed. Supplier names needed the most care in labeling, because stations print call letters, logos and legal names in different places.

The checks catch some of these and not others. When a totals block goes missing, the total is empty, and an invoice without a total goes to needs_review. A wrong supplier name, or a $0.00 tax that agrees with the other amounts, passes every check. That’s the blind spot from the last post: checks compare the extracted values with each other, and they don’t read the invoice again. See Checks.

What’s next

A broader benchmark: more kinds of invoices, new held-out documents, and the method and outputs published so anyone can rerun it. Until then, these are the only accuracy numbers I will publish.

Credits

The documents come from FCC public inspection files, catalogued by project-deepform (opens in a new tab). The labels are ours. Neither the FCC nor project-deepform endorses this work.

Try one of your own invoices in the playground without an account, or start with the docs.