Feedback Loop Design Between Reviewers and Extraction Models

A feedback loop between reviewers and extraction models is a data pipeline. It is a data pipeline, and its output quality depends entirely on what gets captured when a human fixes a wrong value, how that fix gets labeled, and where it gets sent afterward. Most teams treat human-in-the-loop review as a staffing question: how many people, how fast they click through queues. That framing misses the actual ceiling on extraction accuracy, which is set not by review speed but by whether the correction data carries enough structure to change model behavior.

A reviewer who overwrites a wrong value is producing a signal. Whether that signal ever reaches the model in a usable form is a separate decision, made or unmade by whoever designed the correction schema. Three things routinely go wrong before a correction does any good. It gets stored but never routed anywhere near training. It gets routed but arrives stripped of the context that would explain why the model failed. Or it gets applied, but lumped in with every other correction as if a misread character and a hallucinated field carry equal weight, which dilutes the signal until it's mostly noise. A loop that closes just means corrections land somewhere. A loop that learns means corrections change what the model does next, in a way you can point to and measure.

What a reviewer correction contains, and what gets lost by default

Every correction event has at least four parts to keep: which field was wrong, what the model predicted, what the reviewer typed instead, and how confident the model was at the moment it made the prediction. Most review interfaces capture exactly one of these, the corrected value, because that's the minimum needed to fix the record in front of the reviewer. Everything else gets thrown away the moment the reviewer hits submit.

What disappears by default is precisely what a training pipeline needs. The confidence score at prediction time is gone, so nobody can go back and calibrate thresholds against it later. The document's own characteristics, scan quality, layout variant, which vendor's template it came from, never make it into the record. The error class goes uncaptured too: was this a misread character, two fields swapped, a value the model just skipped, or one it invented? And there's no flag for whether the reviewer hesitated. A correction someone made with confidence is a different kind of evidence than one made reluctantly, and treating them the same throws away information.

Research on prompt-based feedback mechanisms bears this out directly. The MADP paper (arXiv:2605.17159, May 2026) describes a technique called Prompt Fine Tuning with Feedback Inheritance, which folds human corrections back into the model's behavior without retraining any weights. The mechanism is designed to fold typed, contextualized corrections back into prompt construction, making the structure of the correction record central to how feedback gets applied.

The schema for a correction event is, in other words, a design decision with consequences for training, not a footnote to the review interface. It deserves to be treated as part of the model's interface to the world, built before anyone starts sketching what the reviewer's screen looks like.

Confidence scores as routing inputs, not quality badges

Confidence scores usually show up on screen as a kind of reassurance, a number next to a value that tells the reviewer how sure the model claims to be. That's the wrong use for them. Their real job is as a routing predicate: above some threshold, auto-approve; below it, send to a human. The threshold itself, though, is a calibration problem, and treating it as a config default you set once and forget is how systems quietly drift into bad habits.

Calibration quality is not consistent across models, and the gap can be large. The ConfBench benchmark (2026), which ran over 70,000 entity-level evaluations across more than 1,300 document variants, found calibration ranging from near-perfect to badly overconfident. A model reporting 96% confidence is not making the same claim as a different model reporting 96%, even though the number on screen looks identical. Copying a threshold from a vendor's demo, or from whatever the last system used, means inheriting somebody else's error rate without knowing it.

Calibration also varies by field, not just by model, and field-level granularity reveals overconfidence and miscalibration that document-level averages hide. Numeric fields tend to calibrate well. Free-text fields show real overconfidence at the high end, claiming near-certainty on values that turn out wrong more often than the number implies. A financial figure that drives a payment decision deserves a stricter threshold than a field that only feeds an archive. Building a calibration set from actual production documents, bucketing predictions by confidence band, and checking each band against ground truth is the only way to know where a given field actually sits. The threshold, once set, should reflect what a wrong value in that field costs.

The EXTRACTCONF confidence engine (arXiv:2606.24420, 2026) shows what this buys in practice: at 80% coverage, meaning 80% of predictions get auto-approved without human review, it hits 99.1% automated accuracy, against a 73.3% base rate for the underlying extraction. That gap exists because confidence is used to route selectively, not because the underlying model got better on its own. And this routing layer sits directly upstream of the feedback loop: a poorly calibrated router lets the wrong corrections through, or misses the ones that matter, and the training data built on top of it inherits the flaw.

Structuring the correction record so it can train, not just fix

A correction meant to feed training needs more fields than one meant only to patch a record. At minimum: the canonical field identifier, not whatever label the UI happened to display; the predicted value exactly as the model returned it; the corrected value the reviewer entered; the confidence score at inference time, kept as-is rather than recalculated after the fact; an error class drawn from a fixed vocabulary, something like misread, transposition, missing, hallucinated, or layout-caused; a document fingerprint identifying the layout or template; and, where it's available, a flag for reviewer certainty.

Error class is the single field that does the most work. A misread character points at the OCR layer and calls for better text extraction. A layout-caused transposition points at the parser and calls for routing logic fixes. A hallucinated value sitting in a field the model claimed high confidence on points at recalibration. Three correction events that look identical on the surface, an expected value swapped for a different one, can demand three entirely different fixes depending on what class they belong to.

MADP's feedback inheritance approach depends on exactly this kind of typing. Corrections carry context that gets folded into prompt construction directly, without touching the underlying model's weights at all, which means the correction functions as an input to prompt engineering as much as a label for training. Document fingerprint plays a similar structural role: it ties an individual correction to a population of documents that share it. If forty corrections on one vendor's invoice template share the same error class, that's a routing problem specific to that template, not evidence the extraction model needs retraining across the board. Building this schema is work that has to happen before the review UI gets built, not after, since the UI's job is to surface the fields a reviewer needs to fill in, not just present a blank box for the corrected value.

Routing corrections to the right remediation layer

Three layers can absorb a correction, and sending it to the wrong one wastes the signal. The OCR or perception layer handles misread characters, degraded scans, and fonts the text extraction step can't parse cleanly, none of which has anything to do with the semantic model downstream. The parser or routing layer handles layout problems: multi-column documents that get read out of order, table boundaries the system misjudges, documents classified into the wrong processing path before extraction even starts. The extraction or semantic layer covers what's left: transposed fields, values invented out of nothing, fields missing on a layout the system has otherwise handled correctly before.

A layout-caused error sent to model retraining adds noise to that training run without fixing the parser logic that actually caused the mistake, and the reverse holds just as true. Architecture work on document pipelines generally supports keeping these layers separate: treat OCR as its own perception step, separate from the semantic extraction stage downstream. The same modular logic applies to how corrections get routed after the fact, since each layer can be monitored and improved on its own timeline.

In practice this looks like a classifier that reads the error_class and document_fingerprint fields on a correction and dispatches it to the right queue. OCR-class errors go toward swapping out the text extraction engine or improving pre-processing for that document population specifically. Layout-class errors update a parser routing rule tied to a particular template. Semantic-class errors go into a queue for extraction model fine-tuning or prompt updates. The ParseBench evaluation, which ran 14 different parsing methods against roughly 2,000 enterprise pages across five capability dimensions, found no single parser wins across every document type. That result is itself an argument for routing as the real architectural lever, rather than betting on one model to handle everything. A pattern that recurs across a population is signal worth acting on, while a single one-off correction usually isn't, so corrections are worth aggregating by document fingerprint before dispatching them anywhere.

Active learning as a strategy for concentrating reviewer effort where it produces the most signal

Passive review, where every document below a confidence threshold lands in a human queue, produces less and less signal per hour of reviewer time as volume grows. Reviewers end up spending as much time on trivial near-misses as they do on the failures that actually matter. Active learning flips the arrangement: instead of queuing everything below a line, the model flags the specific cases where its own uncertainty runs highest and asks for input only there. That concentrates reviewer attention on the documents where a correction will actually move the needle, and it cuts review volume without cutting signal quality, since reviewers are looking at harder cases rather than a random slice of the backlog.

In active learning configurations, the model proactively flags uncertain cases and requests human input, so reviewer effort only goes where it's needed. Making this work operationally requires the model to emit an uncertainty signal that's distinct from its confidence score. A prediction that's high-confidence and wrong, and a prediction that's low-confidence and right, both deserve review, but for opposite reasons, and a system that only tracks confidence can't tell them apart.

This connects straight back to calibration. Active learning only concentrates effort correctly if the uncertainty signal driving it is itself trustworthy. An overconfident model routes the wrong documents to review and, worse, misses its own systematic failures precisely where the feedback loop needed to catch them. MADP's production deployment, run against 955 real documents through January 2026, reported a 97.0% full-pipeline automation rate, with only 3% of documents needing a non-AI fallback. With full HITL configuration layered on top, document-level accuracy on the evaluation subset reached 98.5%. The gap between those two numbers is the value that well-targeted human review recovers, and it's a modest gap precisely because the review effort behind it was concentrated rather than spread thin.

Measuring whether the feedback loop is working

A feedback loop produces two kinds of evidence, and they're easy to confuse. Short-term, corrections clear the backlog: the queue empties, records get fixed, throughput looks healthy. Long-term, the real test is the model's errors on that same document population declining afterward. A queue can run fast and still feed a loop that's learning nothing.

Recurrence rate, tracked by error class and document fingerprint together, is the metric that tells the two apart. If the same error class keeps appearing in the same document population after corrections have been applied against it, the correction never reached the remediation layer that could have fixed it. Recurrence dropping on a specific template is evidence of targeted improvement.

Canary sets help here: a fixed, held-out group of documents with known ground truth, re-run after every model update, shows whether the update helped the population it targeted without quietly breaking something else. Accuracy needs tracking at the field level, not just the document level, because a document-level number can hide degradation in one specific high-value field. Field-level tracking can surface substantial variation in confidence across fields within the same document that document-level averages would otherwise obscure. Only field-level instrumentation would have surfaced a gap that wide. Calibration itself needs periodic rechecking too: re-run the calibration set on a schedule, and if the error rate inside a given confidence band starts climbing, the threshold that was correct at launch has quietly gone stale. Organizations running HITL workflows have reported quality gains in the range of 45 to 60% on AI quality metrics, but that number only means anything if the metrics were defined at field level before the loop went live. Review queue throughput is an operational number to track. A fast queue feeding a badly structured correction record improves throughput. It does not improve accuracy, and conflating the two is how a system convinces itself it's learning when it's really just processing faster.

How continuous learning differs from periodic retraining for document pipelines

In batch retraining, corrections pile up over weeks, a model update gets scheduled, the new model deploys, and the cycle resets. The lag between an error appearing and a fix landing can run to weeks, and in that window, systematic errors on a newly onboarded vendor template just sit there, uncorrected, until the next scheduled cycle comes around.

Continuous learning closes that gap by ingesting corrections and applying them incrementally, whether through prompt adaptation, as with MADP's feedback inheritance mechanism, fine-tuning against a rolling window of recent data, or updates to parser routing rules. The model adjusts to the document population it's actually seeing right now, rather than the population that existed back when it was last trained. Document pipelines feel this lag acutely, since new vendor templates, revised regulatory forms, and format changes appear on a rolling basis. A model that was accurate at deployment degrades as soon as it meets document types it has never seen, and a static retraining schedule can't keep pace with that.

MADP's feedback inheritance mechanism handles this without retraining the underlying language model at all: corrections get inherited as prompt context, so extraction behavior adapts to new document types as corrections accumulate, and weight updates aren't required for every adjustment. A secondary result stands out too, on the sustainability side: MADP's hybrid approach combining AI with human-in-the-loop review cut CO2 emissions and energy consumption by 69% each compared to a fully manual process. Keeping automation rates high, 97.0% in the production deployment cited earlier, compounds that efficiency rather than forcing a fallback to manual processing during retraining downtime. Build the correction pipeline so it can route to both mechanisms: fast prompt-level adaptation for errors that need a quick fix, and slower weight-level fine-tuning for errors that recur often enough to justify it. Which mechanism a correction goes to should depend on its error class and how often that class recurs.

Four failure patterns where the feedback loop breaks in production

The correction sink: corrections get stored, faithfully, and then nothing reads them. The symptom is a review queue that clears on schedule while accuracy on the same document types stays flat month over month. The cause is usually simple: there's no downstream consumer pulling from the correction store, so the loop never actually closes. The fix is to instrument that store with a real consumer that dispatches to remediation queues, and to track consumption rate as its own metric, separate from how fast records get stored.

Undifferentiated noise: corrections do get consumed, but the pipeline treats a one-character OCR slip and a hallucinated dollar figure in a high-confidence field as equivalent training signal. The fix runs straight back to the error_class field: without it, a training pipeline has no way to weight a systemic layout failure differently from a rare edge case, and the two get blended into a signal that teaches the model very little.

Threshold drift: a confidence threshold gets set at launch based on a calibration set that reflected the document population at the time. Months later, new templates and new vendors have shifted that population, but nobody re-ran calibration, so the threshold that was accurate then is quietly wrong now, routing too much or too little to review without anyone noticing until accuracy numbers start slipping.

Layer misrouting: a correction lands in the wrong remediation queue, most often a layout-caused error sent to model fine-tuning instead of parser logic, or the reverse. The correction gets "handled" in the sense that someone acted on it, but the underlying cause never gets touched, and the same error class keeps recurring on the same document population, which is exactly the pattern a recurrence-rate metric is built to catch.