A visual embedding does not care that the carton has a tiny SKU in the corner. OCR does not care that two cartons share a font. Together they catch cases either one drops.
I run a cheap OCR pass on the same crop I use for matching. If OCR returns a structured code that exists in the table, it can jump the queue. If OCR is noisy, the visual ranker still has a vote.
Fusion should be boring
I do not train a giant multi-modal model first. I fuse at the candidate list: boost SKUs that appear in both, downrank conflicts, and send conflicts to a human.
Conflicts are interesting. They are often a relabelled box or a vendor using your SKU font for their own code.
Fail independently
If OCR is down, vision should still work. If the matcher is warming up, a typed or OCR code should still search. Coupled failure is how a mixed pipeline becomes more fragile than either tool alone.
