Distilling a Visual Model for Cheaper Phone Inference

Published 2025-08-03 by Faraz Rahimi

A smaller student model that keeps top-3 hit rate is worth more on the floor than a teacher model that wins a notebook bake-off.

Tags: Machine Learning, Mobile, Computer Vision, Latency, unlisted

I distill when cloud cost or latency shows up in the aisle, not because a paper was trending. The teacher can be heavier. The student has to live on the device or on a cheap CPU.

I evaluate on last week's floor photos, not only on the teacher's favorite set. If top-3 drops on lookalikes, I do not ship the student.

Size is a product spec

Megabytes and milliseconds belong in the same sentence as accuracy. Operators feel the milliseconds. Finance feels the megabytes at volume.

Keep the teacher for the hard queue

Uncertain items can still go to a heavier pass when the network is there. Hybrid again. Ideology never put a box on a shelf.

Original post: https://farazrahimi.com/posts/distilling-a-visual-model-for-cheaper-phone-inference