Engineering / Document AI
The right number can still be the wrong answer
What building Nila taught me about document boundaries, structured extraction, and verification that gives false confidence.
An invoice extractor can return valid JSON, copy numbers that really occur on the page, and produce totals that add up—and still put a tax amount into the taxable-value field.
That was the difficult part of building Nila. Reading characters was only one problem. I also had to decide which document a page belonged to, what each number meant, and what the system could honestly claim to have checked.
I built this as part of a solo accounting-software venture in 2026. The product was tested against real business documents, but it did not reach customer adoption. This article separates the implemented pipeline from experiments I rejected.
Product overview · Extraction code
First, find the documents
A PDF is a container, not necessarily an invoice. It can hold several invoices, continuation pages, quotations, proformas, and unrelated material. Sending each page independently to an extractor loses that structure.
Nila separates discovery from invoice reading. Discovery uses small page images to identify document boundaries. The extraction stage reads the grouped invoice using native content where suitable, or larger page images.
The discovery schema separates two questions:
- Page role: does this page start a document, continue one, or contain irrelevant or unreadable material?
- Document type: is that document a tax invoice, credit note, debit note, proforma, quotation, or something else?
A continuation page inherits the preceding document’s type. That prevents the second page of an excluded document from becoming a new invoice simply because its heading is missing.
The grouping rules also cannot rely on the seller alone: a packet can contain many consecutive invoices from the same seller. Document numbers, buyer identity, page numbering, and closing totals provide additional context.
Discovery schema and grouping rules
Input preparation can damage the facts
“Use the PDF’s text” sounds safer than asking a vision model to read a scan. But the conversion itself can change what the extractor sees.
The extraction notes record a September test where enabling a learned layout stage detached decimal points: 4810.50 became 481050, with the decimal appearing elsewhere. A second conversion route also split a GSTIN across columns. Plain MuPDF preserved those values on the same invoice.
That result supported a narrow decision: leave that layout stage disabled in this route and retain the failing invoice as a regression case. It did not prove that layout models are generally worse.
The implemented sales route uses guarded native tables for eligible digital pages and images for scans or failed conversions. If a scanned continuation needs images, the whole invoice takes the image route. This keeps one document’s interpretation together.
Recorded failure and current routing
A schema is part of the extraction algorithm
A schema tells the model which distinctions it must make. A field named tax leaves too much unspecified: line tax or invoice tax? Combined GST or one component? Printed or calculated?
Nila’s contract explicitly represents printed tax components, their labels, rates, amounts, and scope. Shared Elixir code then interprets those facts for the accounting workflow.
For example, a printed combined GST amount does not mean the document separately printed CGST and SGST. Dividing a total into two equal components may be an accounting calculation, but it must not masquerade as source evidence.
The line-item schema uses positional arrays to avoid repeating every field name for every row. One module owns the order and decoding, keeping the compact provider format out of downstream business code. It makes responses smaller. It doesn’t make the model read anything more correctly.
Strict JSON Schema validation answers whether the response fits the contract. It cannot establish that a number belongs to the field where the model placed it.
Schema and decoding · Printed tax interpretation
What a faithful reading looks like
This bill prints a GST total of ₹22,860.00, and Nila captures ₹22,860.00. The two close-ups below are taken from the same review screen.
Printed on the bill
Extracted by Nila
That equality is one check. It does not prove that the model assigned every line or tax component correctly. For example, a schema that turns one combined tax total into both CGST and SGST would double the tax while still returning valid JSON. I tested that failure directly.
A clearer schema that performed worse
I tested a format that separated a tax breakdown from its combined total, and a variant that also replaced positional items with named fields. Five reviewed bills, five repeats, and three arms produced 75 requests, with one timeout.
| Experimental format | Faithful printed summaries |
|---|---|
| Earlier tax list | 18 / 25 |
| Separate breakdown and total | 10 / 25 |
| Same separation, named item fields | 7 / 24 |
A faithful summary had to preserve the figures actually printed in the invoice summary, including their absence. Plausible arithmetic alone did not count. The new formats sometimes turned one combined tax amount into two component amounts and doubled the tax.
I rejected these candidates. The result applies to five familiar bills, not general extraction accuracy. The comparison arm was another experiment, not production; schema and related instructions changed together. All arms used the same inputs, provider settings, and equivalent examples. Regrading retained responses reproduced these counts.
Source presence is a weaker check than it looks
Suppose an extracted amount occurs somewhere on the invoice. That establishes presence. It does not establish whether the amount is a unit price, taxable value, tax component, discount, or grand total.
Repeated numbers make this especially dangerous. A verifier can accept a wrong field by finding a legitimate occurrence in a different part of the document. Arithmetic checks help with contradictions, but a model can preserve an invoice total while assigning its components incorrectly.
I investigated a more elaborate verifier that indexed OCR values and tried to bind claims to labels, rows, columns, and source locations. It also assigned invoice-wide readiness verdicts.
The design began rebuilding a layout-specific extractor around the model: guessed table structure, broad labels such as “Amount,” and distance-based matching. It rejected correct values while still accepting some values in the wrong roles.
Test the verifier against fixed mistakes
To compare verifier revisions, the experiment held the model candidates and OCR response fixed. That isolated verification behaviour from model variability.
The retrospective records a labelled sample of 15 invoices and 200 claims: 170 scan-correct candidates, 17 semantic extraction errors despite correct OCR, and 13 OCR-origin errors.
The baseline, a proposed safety change, and the full shadow verifier each caught only 1 of the 17 semantic errors. All that extra machinery bought almost nothing.
There was another failure in the baseline: it could remove a value after a source check failed, then verify the smaller artifact again. Across the recorded artifacts, the flagged count fell from 22 before deletion to 13 afterward. The result appeared safer because evidence of the failure had disappeared.
This is why “how many results passed?” is a poor metric on its own. A useful evaluation also asks which correct claims were lost and which incorrect claims survived.
The experimental verifier was removed. It was not a deployed evidence-grounding system. The retrospective also does not endorse the baseline as complete or error-free.
Full comparison, limitations, and removal decision
Completeness needs its own test
An earlier text-based experiment on a 57-page packet produced 53 invoice artifacts without provider failures. The human grouping contained 54 invoice segments. OCR had omitted an entire item and tax table on one page.
No transport error occurred. High OCR confidence did not reveal the missing table. The pipeline’s successful execution therefore did not prove that it had captured the packet completely.
That historical result belongs to the earlier OCR/text route, not a benchmark of the later native-and-vision sales pipeline. Its useful lesson is that provider success, readable text, correct fields, and complete document coverage need separate evaluation.
What I kept in the product
The working system has narrower, explicit responsibilities. Extraction produces structured artifacts and source evidence. Deterministic code checks arithmetic and identifiers. The sales domain owns corrections, tax projection, period rules, and output readiness. The review interface gives the accountant the source bill and the draft together.
Those boundaries make failures easier to investigate. They do not justify a claim that every ambiguous field is detected, or that every accepted invoice is correct.
My main lesson was to evaluate the exact claim a safeguard makes. If a check says a value is present, call it a presence check. If an experiment claims semantic verification, test wrong-role values that are genuinely present on the page. And if a more elaborate system fails that test, remove it.