sw/
← projects
Augmodo·2025–26

Uncertainty-driven QC automation

Test-time augmentation turns every product match into an uncertainty score, so the most confident matches can be auto-approved and human QC goes where the model is unsure. Validated on 20,305 QC-reviewed tasks.

Problem

Every product match the shelf pipeline makes goes to a human for quality control. Most of that review effort is spent confirming what the model already got right or catching what it obviously got wrong, and the QC queue grows with every store. The question was simple: can the model tell us which matches it can be trusted on?

What I did

I used 20,305 tasks that QC had already reviewed as ground truth: 5,766 (28%) confirmed correct and 14,539 (72%) confirmed incorrect. For each detection I ran test-time augmentation, predicting on several perturbed views of the same crop and measuring how much the answers disagree, to get an uncertainty score. Low uncertainty means the model is confident and likely right; high uncertainty means it is unsure and the task should go to a human.

I then analysed the score against QC outcomes with ROC, precision-recall and accuracy-versus-rejection curves across thresholds, to find where auto-approval is safe and what each threshold costs.

Result

Uncertainty separates good matches from bad ones: ROC AUC 0.71. The steep early part of the precision-recall curve is the safe auto-approve zone. The least uncertain 20% of tasks were 60% correct, against 28% across all tasks, more than twice the base rate.

In practice that means auto-approving roughly the 20% least uncertain tasks and routing the rest to QC. At the operating points studied, this would cut QC workload by an estimated 35–40% while still catching about 60–70% of bad matches, at the cost of sending about 30% of good ones to review anyway. All of this came from the existing model, with no retraining: the uncertainty signal was there for free. The obvious next steps are calibrating the score, adding more augmentation views, and training the matcher with uncertainty in mind, each of which should push the separation, and the share of work that can be automated, well beyond this first pass.

system

scroll sideways on small screens

Uncertainty-driven QC flowEach product match is predicted on several augmented views. Disagreement between views gives an uncertainty score. Matches below a threshold are auto-approved; the rest go to human QC. Of all 20,305 reviewed tasks 28% were correct; of the 20% least uncertain, 60% were correct.Product matchdetection + lookupTest-time augmentationsame crop, perturbed viewsUncertainty scoreview disagreementThresholdtunableAuto-approve~20% least uncertainHuman QCmodel unsure, riskylowhighShare correctall 20,305 tasks28%20% least uncertain60%ROC AUC 0.71Estimated at the operating points studied:QC workload −35–40% · ~60–70% of bad matches still caught · ~30% of good ones still sent to review
schematicHow the uncertainty score routes work. Proportions are from 20,305 QC-reviewed tasks.
esc