I keep a holdout set that is not pretty. It is last week's production queries with the eventual truth, including unknowns and rejects.
If a new model wins on the pretty set and loses on last week, it does not ship. That rule has saved me from cleverness.
Include the humans in the metric
A model that is slightly less accurate but much faster can win on the floor. I look at time-to-identity and abandonment alongside hit rate.
A slower, slightly smarter model that people skip is not smarter.
Refresh the holdout
A frozen eval set from launch day becomes a souvenir. I rotate in new weeks so we evaluate against current packaging and current lighting, not the warehouse we imagined.
