Hugging Face and Cerebras make Gemma 4's voice conversations flow like talking to a real person
- Today's voice AI isn't dumb, it's slow. There's a multi-second pause after every sentence, and it doesn't feel like talking to a person.
- Hugging Face and Cerebras swapped out the slowest stage for ultra-fast inference hardware — 1,851 tokens per second, 35x a typical GPU.
- But Doubao, GPT-Live, and Gemini Live are now end-to-end, full-duplex models — they can listen and speak at once and pick up on tone. This pipeline is a four-stage relay running half-duplex, so it can't match their conversational rhythm.
- Its value lies elsewhere: every one of the four stages is open source and swappable, the whole thing can run offline on your own machine, and you get access to the text layer in between.
- To close the rhythm gap, it starts guessing your answer before you finish speaking — and throws the whole thing out and starts over if it guessed wrong. By default, the speech recognition only understands 25 European languages; Chinese needs a separate setup.
Voice AI's bottleneck today is speed
Model quality is already good enough — the answers are correct. What actually breaks the experience is latency: you finish a sentence, and it takes a few seconds of silence before anything comes back.
What makes it worse is the inconsistency. Nine times out of ten the response lands just over a second; the tenth time it drags out to three seconds. You think it didn't hear you, say it again, and now you're both talking at once. Engineers measure this with a specific metric: out of a hundred responses, how slow are the slowest few? That's called P95 latency. The average looks fine and tells you nothing — the sluggish feeling comes entirely from those outliers.
It's like a phone call with someone who's always half a beat slow to respond. They're not dumb, and their answers are fine, but those few seconds of silence between sentences make it impossible to have a normal conversation.
They swapped out the slowest stage
First, who is Cerebras — without that context, the "35x" figure doesn't mean much.
It's a chip company that took a different path from Nvidia. Instead of slicing a wafer into hundreds of chips and wiring them back together, it builds the entire wafer as one giant chip. The payoff: data doesn't have to shuttle between separate chips, since the model weights sit directly in on-chip memory, which is far faster to read from. That design isn't especially useful for training models, but it's very good at inference — running an already-trained model to generate output fast. Selling that speed has been Cerebras's core business for years.
Voice AI runs through four stages between hearing you and answering. The middle one — coming up with the answer — is by far the slowest. No matter how fast the other three run, that one drags the whole chain down.
Hugging Face and Cerebras handed that stage to Cerebras's inference hardware, running Google DeepMind's openly released Gemma 4 31B.
Each of the four stages runs on its own thread. Except for the middle stage, which by default goes over the network, the other three run entirely on-device
A 35x speedup means those particularly slow outliers disappear too. That's directly where the P95 improvement comes from: once the answer-generation stage's time gets compressed way down, the whole chain's variance naturally shrinks with it.
They also turned off the model's "thinking," passing --responses_api_reasoning_effort none at connection time. The same model is smarter with thinking on when writing code, but in conversation, thinking on means it can't keep up with speech pace.
Gemma 4 31B has appeared on this site twice before. This time it shows up in a different role: the brain that can be pulled out and swapped in this open pipeline whenever needed.
You can't take those models with you. This one, you can
To see what this pipeline is actually worth, you first need to separate it from Doubao and GPT-Live — they're not competing on the same axis. Those companies compete on who talks more like a human. This one competes on who owns what.