Preventing Exception Propagation Into ERP Systems

How document parsing actually fails in production

Document exceptions don't announce themselves. A malformed field, a misread vendor ID, a check that fails to match an account: none of these trip an alarm on their way into an ERP. They just become the ERP's data, indistinguishable from everything entered correctly, until someone downstream discovers a payment run is wrong or a purchase order references a vendor that doesn't exist. The only reliable place to stop that is upstream of the integration boundary, at the extraction layer itself, before a single field crosses into the system of record. A smarter ERP validation rule catching this after the fact is solving the problem one step too late, and no amount of downstream cleverness fixes a problem that started at ingestion.

Traditional OCR reads a document the way a scanner reads a barcode: character by character, with no real sense that a document is a structured object rather than a strip of text. It has no concept of a relationship between a table cell and the header sitting above it. Template-based OCR tries to patch this by requiring a predefined layout for every document type it expects to see, which works fine until a vendor changes their invoice format by two pixels and the whole thing breaks. That's the architecture working exactly as designed, on documents it was never designed for. It's the architecture working exactly as designed, on documents it was never designed for.

Consider what happens when a vendor invoice shifts its column positions slightly outside the bounding box the parser expects. The field doesn't appear. No error, no flag, no log entry, no warning to the approval engine waiting on that number. Absence is a lot harder to catch than a wrong value, and most pipelines aren't built to notice something that was never there in the first place.

Pull three documents at random from a real client folder and the pattern appears fast, not eventually. A scanned purchase order merges field labels into the values themselves, so "Total: $4,200" becomes an unreadable blob instead of two separate fields. An insurance claim form loses two entire sections because the parser can't handle the embedded fonts. A financial statement comes out as clean-looking text, except the line breaks land mid-sentence, which quietly breaks every regex pattern built to parse it downstream. None of this is an edge case. It's what the first three documents anyone checks tend to look like.

Open-source parsing pipelines handle clean, text-layer PDFs with predictable, single-column layouts reasonably well. Accuracy drops fast on scanned pages, multi-column formats, and tables that span a page break, because the underlying extraction model assumes the page holds still and behaves. A technical guide from unsiloed.ai puts the production track record bluntly: fewer than 10% of in-house parsing pipelines ever make it to production, because edge cases pile up faster than engineering teams can patch around them.

The failure doesn't stay contained, either. If a parser drops a table, scrambles the reading order, or flattens a nested section into flat text, every system that consumes that output inherits the mistake with no attribution attached. A retrieval pipeline built on top of it serves bad context. An agent acting on it acts on nonsense. An ERP integration writes the garbage straight into the ledger. Teams sink real effort into picking the right model and tuning prompts, while the pipeline feeding those prompts gets a fraction of the attention. That imbalance puts a ceiling on results no amount of prompt engineering can lift.

The failures that matter most to an ERP, hidden by aggregate accuracy metrics

An overall accuracy number is close to useless for catching the failure that actually costs money. Averaging across document types hides exactly the problem the average is built to smooth over: a parser scoring 99% on invoices from one supplier and 75% on invoices from another produces a blended figure that reads fine on a dashboard, while one entire vendor relationship quietly generates bad data on every invoice it sends.

Measuring accuracy field by field is the only fix that addresses the problem. A 2025 peer-reviewed study (arXiv:2505.01555) did exactly that: reviewers manually examined a random sample of 200 records, at a margin of error of plus or minus four percent at 90% confidence, comparing each extracted field against a verified ground truth. The per-field breakdown told a different story than an overall number would have. Year came in at 100%, Purpose at 97.49%, Apparatus at 92.96%, Agent at 89.95%, Season at 87.94%, with an overall average of 94.72%. That average alone would never have flagged Season as the field needing intervention. Only the field-level breakdown does that.

Confidence scores carry a bigger trap, and it's the more dangerous one. A well-calibrated system reporting 0.95 confidence should be right about 95% of the time at that score. Plenty of systems aren't well-calibrated: benchmark data published by airparser.com shows systems reporting 0.95 while actually landing closer to 80% accuracy. Low-confidence output tends to route to human review anyway, so it corrects itself. High-confidence output that's actually wrong is the case that does real damage, because nothing downstream catches it before it reaches the ERP.

Calibration quality isn't uniform across field types, either, and overconfidence at the high end of the probability range is a known risk the EXTRACTCONF confidence engine research (arXiv:2606.24420) addresses. Pull 100 extractions a system marked above 0.90 confidence and check them by hand. If more than 10% turn out wrong, the confidence scores aren't trustworthy, per the same airparser.com benchmarking.

A deeper measurement problem causes all of this, as ConfBench (arXiv:2608.01792) shows by pointing out that most document benchmarks get built on clean, high-quality samples that produce mostly high-accuracy predictions, leaving the low- a... A broader measurement problem underlies all of this: most document benchmarks get built on clean, high-quality samples that produce mostly high-accuracy predictions, leaving the low- and mid-accuracy range too thin to study calibration properly. Real production documents look nothing like that. Scanning artifacts, physical wear, and faded ink drag down both accuracy and the reliability of the confidence score attached to it, and clean benchmark data doesn't capture that variation.

Building the extraction gate: confidence thresholds, routing logic, and coverage trade-offs

The design principle that produces a working gate is simple to state: route by confidence, don't pass everything through uniformly. Set a threshold, say 0.85, and anything scoring below it goes to a manual review queue instead of straight to the ERP. Raise the threshold and the automated stream gets more accurate, but more documents go to the human queue. Lower it and automation goes up while the accuracy of what's flowing through automatically goes down. No better model makes that trade-off disappear. It just moves the line.

The 2026 paper on the EXTRACTCONF confidence engine (arXiv:2606.24420) puts a number on the payoff. At 80% coverage, the system hits 99.1% accuracy on the automated stream, a 25.8 percentage point jump over the 73.3% baseline rate. The goal is maximizing how much volume can run automatically at a given quality bar. It's maximizing how much volume can run automatically at a given quality bar, and treating the documents that fall short as exactly where the cost of human review earns its keep.

In practice, a well-calibrated parser that routes high-confidence documents through automatically and sends the remainder to a human beats a system that pushes all of its output through automatically at a lower average accuracy. Fewer documents processed automatically, but a much cleaner set of records landing where it counts. That trade is worth making almost every time.

There's a blind spot even a good threshold can't close, and it's the one most teams don't budget for. Uncertainty signals share a fundamental weakness: they can't catch an error the model is internally consistent about. Catching that requires an external verification signal, something like grounding support that checks the extracted value against an outside reference rather than just checking whether the model agrees with itself.

The gate also has to operate at the level of individual fields. A document can clear its overall confidence bar while one field inside it sits below threshold, and since ERP records get populated field by field, the gate needs to match that granularity. Extraction errors trace back to document quality more often than to the model itself. Feed a frontier model an unreadable scan and it will still produce a high-confidence token about garbled text, because from its vantage point the noise looks like signal. The fix lives upstream, in document preprocessing, not in a better model downstream.

Agentic validation as the layer between extraction and ERP ingestion

Confidence thresholds catch low-confidence mistakes. They don't catch the ones that look fine. A wrong vendor ID that happens to match the expected format exactly, or a transposed date that still passes format validation cleanly: both clear a confidence threshold without issue, because nothing about their structure looks wrong. They're self-consistent errors, and field-level scoring isn't built to see them.

That's the gap agentic validation fills, as a distinct second layer sitting between extraction and the ERP write. It runs automated checks that ask questions confidence scoring never asks. Does the sum of the individual line amounts actually match the line-item total? Does the vendor listed on the invoice exist in the ERP's vendor master file? Does the PO number resolve to a purchase order that's actually still open? Does the invoice date fall inside the payment terms window the vendor is supposed to be operating under? Does the currency code match what's on file for that vendor? These are cross-references, not confidence estimates, and metadata pulled from the document itself becomes the search key for running them.

LLM-based extraction introduces a risk that traditional OCR mostly avoids: hallucination. A traditional OCR engine that misreads a character was misreading something that actually existed on the page. A language model that gets a value wrong may have generated something that was never there at all. For financial documents and legal agreements, that distinction matters enormously, and it's exactly the risk agentic validation exists to catch before anything crosses into the ERP.

The human review queue needs to be treated as a real, designed part of the system, with clear ownership and a set schedule for checking it. It needs a defined service-level agreement: fields flagged for review get resolved within a set window, with someone accountable for that window. Skip that discipline and the queue grows quietly until the gate stops functioning as a gate and starts functioning as a bottleneck. Corrections made in that queue should also feed back into the extraction model, so accuracy improves specifically on the document patterns that used to fall below threshold. Over time, the burden on the queue should shrink as the model gets better at the actual documents it's seeing in production.

The end state: the ERP integration endpoint only ever receives validated, structured JSON, where every field has either cleared the confidence threshold and passed the agentic checks, or been resolved by a human with a documented decision behind it. The ERP is never where an exception gets discovered for the first time.

The build-vs-buy calculus for a production-grade validation gate

Building this gate is a bigger undertaking than building a parser, and treating it as a smaller add-on is exactly where most in-house efforts underestimate the work. It requires confidence scoring, calibration monitoring to catch drift, threshold-based routing logic, the agentic validation layer described above, a human review queue with real SLA management, a continuous learning loop, and ongoing monitoring for accuracy regression. A parser alone covers none of that, and most teams that set out to "build a parser" end up half-building this gate anyway, badly, under deadline pressure.

Self-hosting at modest scale carries a cost floor most teams underestimate going in. Compute, storage, and egress typically run somewhere between $15,000 and $40,000 a year, and Extend.ai finds that maintaining parser dependencies and writing custom logic for edge cases eats up 20 to 30 percent of an engineer's working hours every quarter, a recurring drain most teams fail to budget for. That's not a one-time integration cost. It recurs, quarter after quarter, for as long as the pipeline stays in-house.

Only a narrow set of cases justifies building anyway: document processing being the actual product a company sells, or a single document type at volume low enough that the economics never really matter. Everyone else is better off buying, and the model landscape makes that easier to admit than it used to be. The open-weight options got meaningfully cheaper by 2026: PaddleOCR-VL-1.6 (Apache 2.0, a fraction of a billion parameters, support for over 100 languages, around 45 pages a minute on one accelerator card), DeepSeek-OCR (a few billion parameters, MIT license), GOT-OCR 2.0 (strong on equations, with a relatively low VRAM footprint), and Granite-Docling (a couple hundred million parameters, Apache 2.0, notably strong on financial tables), according to self-hosting analysis from Spheron. None of them ship with production monitoring, confidence scoring, or regression alerting built in. Whichever open-weight model produces the outputs, the validation gate still has to be built on top of it separately, and that's the part vendors are actually selling, not the OCR itself.

High-volume structured workloads are also a poor match for LLM-based extraction generally. A pipeline running 100,000 invoices a day from a fixed roster of enterprise vendors is better served by structured parsing paired with a validation layer than by routing everything through a language model at that scale. Routing everything through an LLM there is spending compute to solve a problem structured parsing already solved decades ago.

For teams buying rather than building, the evaluation criteria decide which vendor can actually deliver, and they should be weighed ahead of the marketing copy. Look for per-field confidence scores, thresholds that are actually calibrated rather than asserted, field-level SLA guarantees, zero data retention for sensitive financial documents, and a continuous learning mechanism that improves from corrections over time. A headline accuracy number on a vendor's demo set tells little on its own. The real test is asking for per-field accuracy against ground truth on the specific document types in question. A vendor that can't produce that number is asking the customer to absorb accuracy risk without knowing what the risk actually is.

The integration boundary when the gate is working

The integration boundary should function like a contract with firm terms. Only validated, structured JSON crosses it, and every field in that JSON has either cleared a confidence threshold, passed an agentic cross-check, or been resolved through human review with the decision documented. Nothing gets a free pass because it looked plausible.

When that contract holds, the ERP goes back to doing what it's supposed to do: serve as a clean system of record rather than the place where exceptions become visible after the damage is done. Records arrive already resolved. Nobody has to dig through a ledger entry six weeks later to figure out why a vendor ID doesn't match anything real.

The handoff format matters too. Field names, types, and values should map directly onto the ERP's own data model. Designing the extraction schema against the ERP's input schema, rather than against the document's visual structure, removes a whole category of transformation ambiguity before it has a chance to cause trouble.

What gets monitored matters as much as what gets built. Tracking the ratio of automated records to reviewed records over time gives an early signal: a rising review rate on a particular document class means an extraction regression is developing, and it becomes visible in that ratio well before it becomes visible as a data quality complaint from finance. More useful still is the percentage of fields clearing 0.95-plus confidence, paired with the actual accuracy rate within that cleared set. Those two numbers describe whether the gate is doing its job. Overall parser accuracy averaged across everything doesn't, and a team still watching that single number is watching the wrong dashboard.

The loop that keeps all of this sustainable is the same one described earlier: corrections from human review feed back into the model, accuracy improves on the patterns that used to fail, and the automation rate climbs while the review burden shrinks. Not because the documents get easier, but because the gate gets better at the documents it's actually seeing.