I stopped quoting overall accuracy after the first week. A model can be 95% right on easy SKUs and still fail the ten items that cause the radios to go off.
The metrics I keep on a small dashboard: top-1, top-3, explicit reject rate, dwell-confirmed rate, median time-to-result, and weekly active operators.
Stratify by pain
I tag SKUs as easy, lookalike, or unlabelled-known-problem. A drop in lookalike top-3 is an incident. A drop in easy top-1 means preprocessing or a lighting change.
I do not average those together. Averages hide the aisle that is on fire.
Watch abandonment
If people open the camera and leave before results, the tool is too slow or too wrong. That curve is more honest than a confusion matrix you computed on last month's stills.
