Confidence Threshold Calibration for Document Extraction Queues

Confidence threshold calibration for document extraction queues is an ongoing task that demands continuous attention. It's an ongoing engineering discipline, and it needs per-field thresholds built from ground-truth testing, not a single global score or a vendor's marketing benchmark, if it's going to tell you reliably which extractions are safe to automate and which need a human to look at them.

Document processing pipelines route every extracted field somewhere. High confidence, it goes straight through. Low confidence, it lands in a review queue. That routing decision is a production requirement, not a research question, because the cost of getting it wrong shows up on someone's balance sheet or in a compliance audit. Research presented at the IJCAI-ECAI 2026 RobustifAI Workshop, from a team at Perfios, found that the standard ways of generating a confidence score (token-level log-probabilities, verbalized self-assessed confidence, multi-sample self-consistency checks) all tend to collapse toward marking everything as correct once you push them toward the thresholds teams actually need. On DocILE, a benchmark of 55-field invoices with a 26% failure rate built into the document set, log-probability-mean scoring hit only 0.705 ROC AUC and turned into an all-positive classifier at any threshold worth using. Verbalized confidence, where the model just states a number, did about the same at 0.692 AUC.

That gap matters because a wrong extraction nobody flags is worse than one that gets kicked to a human. Financial reconciliation, compliance checks, procurement automation: these run on the extracted value, not the raw document, and a bad value that sails through with a high confidence score poisons everything downstream. Confidence scores aren't the problem here. Treating one global number as a routing oracle, instead of building the real signal field by field, remains the mistake most teams still make.

What a confidence score actually measures, and what it doesn't

Two approaches compete for the job. One has the model report a number about how sure it feels (verbalized confidence). The other reads the log-probabilities the model assigned to the tokens it generated. Both deserve scrutiny before teams trust them by default, for reasons that are mechanical.

Log-probability scoring has a specific failure built in: a capable model can transcribe garbled OCR output with total fluency and still assign that wrong answer a high log-probability, because the score reflects how confident the model is in the token sequence it produced, not whether that sequence matches what the document actually says. Verbalized confidence has its own version of the same problem. Research out of ApplyBoard, presented at IEEE ICPRS 2025, found that a model asked to grade its own extraction leans hard toward saying it did fine, with specificity down at 14.4% for the naive self-critique setup, meaning the model almost never flags its own mistakes.

Input format changes the picture too. A model fed both OCR text and the source image produces more accurate confidence estimates than one fed the image alone, and the gap widens for smaller models with less capability to begin with. That's a lever teams can pull directly: modality choice is a calibration decision, not just a cost decision. Aggregation method matters as well. Taking the log-probability of just the first generated token consistently beats averaging across all tokens or using margin-based methods, and that's a plain implementation choice with a measurable payoff.

Then there's the document itself. A 0.90 confidence score on a clean, well-scanned PDF doesn't mean the same thing as a 0.90 on a blurry fax or a handwritten form, because the score isn't invariant to how messy the source document is. Vendor claims of "95% to 99% accuracy" need to be read with real skepticism, since accuracy at the character level, the field level, and the whole-document level are three different measurements, and the one that actually governs whether automation works is field accuracy. None of this means throwing out confidence scores. It means recognizing where the real unit of calibration has to live: the individual field, not the document as a whole.

Why per-field thresholds are the correct unit of calibration

Field-level confidence doesn't behave uniformly, not even close. One audit of a production extraction pipeline found minimum payment amount coming in at 0.89 confidence, statement balance at 0.779, and payment due date down at 0.675, against an overall model average of 0.781. That's not noise. Amount fields tend to run higher because they follow consistent formatting: digits, decimal points, currency symbols in predictable places. Date fields run lower because dates show up in different formats, different spots on the page, and often depend on surrounding context to interpret correctly.

Apply one global threshold across a field set like that and you get two opposite mistakes at once: date fields get shoved into automation when they shouldn't be, and amount fields get held back from automation when they're actually reliable enough to trust. Parser accuracy swings more than 55 percentage points depending on the document type, which makes a single fixed threshold close to useless across a mixed queue of invoices, statements, and contracts.

Cost matters just as much as accuracy here, and this is the part most teams get backwards: they treat the threshold as a tuning knob for accuracy alone, when it's really a tuning knob for consequence. A wrong number in a field that drives a financial decision costs a business something entirely different than a wrong value in a field nobody downstream reads closely. So the threshold isn't a technical setting, it's a business decision wearing an engineering parameter's clothes. Setting the bar at 0.85 means accepting roughly a 15% error rate in exchange for the other 85% moving through untouched, and that trade-off has to get made separately for every field, not once for the whole document.

Comparison logic needs to match the field type too. One published methodology accepted a numeric value as correct if it landed within 0.5% of the ground-truth figure. ConfBench's own comparison library, called Stickler, handles string fields with character-level similarity, numeric fields with a proximity tolerance, and dates with a dedicated date comparator. Thresholds that respect these differences start with testing against ground truth that looks like what actually shows up in production, not a vendor's demo set.

Building a ground-truth evaluation set that reflects production reality

Clean test documents will lie to you. Existing document benchmarks skew heavily toward high-quality samples, leaving the low- and mid-accuracy range too thin to draw real conclusions from. Calibrate against clean data and the thresholds fall apart the moment production documents show up degraded, which they always do.

Sampling has to reflect the actual mess: pull from the real production corpus, and make sure it includes low-resolution scans, multi-column layouts, tables that break across a page, files with embedded fonts a parser might choke on. Test only against tidy, well-formatted documents and the surprise arrives right after launch, not before.

ConfBench's own method is worth using as a model for scale. Researchers ran 20 controlled degradation pipelines against a verified set of documents, producing 1,346 document variants and more than 70,000 entity-level evaluations spanning the full range from clean to badly degraded. The degradation tiers were deliberately uneven: the high-impact tier covers combinations of severe degradations, while lighter pipelines cover the everyday degradations that appear constantly in real production volume.

Annotation quality deserves its own scrutiny, separate from everything else. Calibration testing turns out to be far more sensitive to labeling mistakes than plain accuracy testing is, because one mislabeled entity doesn't just cause one error, it corrupts every confidence bucket that entity falls into and skews every calibration metric built on top of it. Ground-truth labeling can't get handed off to whoever's cheapest. It needs people who understand the field types, plus a dedicated review pass that checks for consistency across labelers. Heading structure, reading order, table integrity, page references, and schema fidelity all need separate attention when building the evaluation set, since accuracy alone isn't the whole picture. Once that ground-truth set actually reflects production, the real work starts: finding out exactly where confidence scores stop tracking correctness.

The failure modes that make calibration harder than it looks

Thresholds drift. A parser trained against one mix of documents slowly runs into inputs that look nothing like what it trained on, and a threshold that worked fine in development quietly stops matching reality as the production document mix shifts underneath it.

The dev-to-production gap shows up constantly in practice: set the threshold at 0.90 in development, watch it perform well, then ship it and discover that a meaningful share of those high-confidence extractions in production are wrong anyway, because production data simply looks different from the dev set. Thresholds need validation against data that resembles production, always, not against whatever sample was convenient to build with.

Some failures are baked into how these models work, and no amount of retuning fixes them. Checkbox and boolean fields make this especially visible. On an ambiguous scan, a model might land close to 50/50 on whether a box is checked, and any confidence score above 0.5 in that situation tells you almost nothing, since the field is a binary flip between two opposite meanings and the model is essentially guessing.

Layout failures are their own category, and arguably the most dangerous, because they're invisible in the score itself. A single header shifted out of place can cause an extraction agent to pull the wrong value entirely while still reporting high confidence: the confidence score reflects the model's certainty about the token it picked, not whether that token got mapped to the right field. Embedded fonts cause a different problem: a parser that can't handle the font encoding might drop entire sections of a document, and a missing field generates no confidence score at all, because absence doesn't produce a signal to distrust.

A benchmark called ParseBench lays out a broader failure taxonomy across five capability areas: tables (merged cells, layered headers, tables spanning page breaks), charts (pulling exact data points), content faithfulness (skipped text, invented text, reading order mistakes), semantic formatting (strikethrough, superscript, bold, hyperlinks that carry meaning), and visual grounding. The research behind ExtractConf states the underlying issue directly: extraction errors are often caused by things the model has no way to observe, unreadable source material, ambiguous page layouts, OCR noise, which means any confidence signal drawn purely from the model's own internals is blind to the most common causes of failure in production. These failure patterns share a shape: confidence is high when it should be low, or missing entirely right when a flag is most needed. Fixing that takes more than nudging a threshold. It takes building the confidence signal from more than one source.

Constructing a reliable routing signal from multiple confidence sources

A trustworthy signal has to draw from several places at once: what the model actually generated, what the document image quality looks like, where on the page the extraction came from, what the OCR layer says independently, and whether separate extraction passes agree with each other.

One concrete design pattern for this is a two-call setup, sometimes called Hunter and Mapper. The Hunter call extracts each field on its own, working off a prompt structured around filling in schema slots, and it tends to invent values when a field is ambiguous or genuinely missing. The Mapper call works the opposite direction: it scans the whole document without anchoring to specific fields and surfaces candidate values grounded in what's actually printed on the page, but it tends to miss fields that aren't visually obvious. Where the two calls disagree turns out to be informative on its own, since they fail in different ways, and that disagreement carries information neither call's internal confidence score captures alone.

The results back this up. On the DocILE invoice set, the dual-call approach reached 0.928 ROC AUC and cut selective prediction risk by 70% compared to plain log-probability-mean scoring. In the highest confidence band, accuracy reached 99.1%, a 25.8-point jump over the baseline rate. It transferred to a different document type too: zero-shot testing on CORD receipts hit 0.858 AUC with no retraining, suggesting the signal generalizes rather than just fitting one dataset. Post-hoc recalibration on top of that cut expected calibration error by 89% and Brier score by 43%, though that's a separate step applied after deployment, not part of setting the threshold itself.

There's another path worth knowing for teams where a dual-call setup adds too much cost. ApplyBoard's research found that classifying embedding vectors to distinguish correct extractions from wrong ones hit an F1-score of 0.98, a strong result from a lighter architecture. Self-consistency, running the same extraction five separate times and checking agreement, sounds appealing but doesn't pay off: it reached only 0.744 AUC, barely ahead of plain log-probability scoring, while costing five times as much in API calls per field. Skip it. Model choice itself affects all of this before any threshold gets set: within a single model family, calibration quality scales predictably with capability, but across different families, parameter count alone tells you almost nothing about how well-calibrated a model's confidence will be. With a reliable, multi-source confidence signal in hand, the actual work of setting thresholds field by field can begin, and that work has to encode what an error actually costs, not just whatever number the model happens to output.

A practical framework for setting per-field thresholds across a document queue

Start by sorting fields by what an error costs, not by what kind of document they appear in. Financial amounts, compliance dates, policy numbers: these need thresholds at 0.90 or higher, because a wrong value here carries real financial or regulatory weight. Archival or descriptive fields can sit lower, since routing them to a human costs more than the occasional mistake would. Either way, the threshold is a business call dressed up as a config value, not a default to accept because a vendor shipped it that way.

Derive the actual numbers from ground-truth testing, not a vendor's marketing sheet. Run the multi-signal confidence engine against a production-representative evaluation set, and for each field, find the confidence value where precision clears the error rate that field's cost tier can tolerate. Use comparators suited to the field type: numeric fields within roughly 1% tolerance, dates checked for calendar equivalence rather than exact string match, string fields judged on character-level similarity, following the same logic ConfBench's Stickler library uses.

From there, build a three-tier routing structure. High confidence goes straight to automation, but only after that accuracy has been checked against ground truth directly, never assumed from a vendor's claim. Mid-confidence extractions go to a human reviewer with the extracted value already filled in, ready for a quick correction rather than a blank field. Low confidence, or a missing extraction altogether, gets kicked back for full manual handling or reprocessing.

Recalibrate per model before any of this goes live. Calibration quality varies a lot across models, and a post-hoc correction step rescales the raw confidence numbers for threshold routing without touching the model's underlying ranking of which extractions are more or less trustworthy. Skipping this step isn't really an option for a model that runs overconfident: a raw 0.85 from a model prone to overconfidence doesn't mean what a 0.85 means from a model that's actually well-calibrated, and treating them the same is exactly the mistake this whole discipline exists to catch.