2B-Parameter Models Now Fit on Phones: What On-Device Inference Changes

Through distillation and structured pruning, 2B-parameter models now approach last-generation mid-size cloud models on several benchmarks, and the real change is they fit inside a phone. Latency is the dividing line: cloud round trips cost hundreds of milliseconds, on-device inference tens, which makes or breaks a voice assistant. Product teams get a practical split: devices handle instant, privacy-sensitive responses; complex reasoning stays in the cloud; a routing policy decides who handles what. Device-cloud collaboration is now an engineering requirement.