They swapped out the slowest stage

First, who is Cerebras — without that context, the "35x" figure doesn't mean much.

It's a chip company that took a different path from Nvidia. Instead of slicing a wafer into hundreds of chips and wiring them back together, it builds the entire wafer as one giant chip. The payoff: data doesn't have to shuttle between separate chips, since the model weights sit directly in on-chip memory, which is far faster to read from. That design isn't especially useful for training models, but it's very good at inference — running an already-trained model to generate output fast. Selling that speed has been Cerebras's core business for years.

Voice AI runs through four stages between hearing you and answering. The middle one — coming up with the answer — is by far the slowest. No matter how fast the other three run, that one drags the whole chain down.

Hugging Face and Cerebras handed that stage to Cerebras's inference hardware, running Google DeepMind's openly released Gemma 4 31B.

A 35x speedup means those particularly slow outliers disappear too. That's directly where the P95 improvement comes from: once the answer-generation stage's time gets compressed way down, the whole chain's variance naturally shrinks with it.

They also turned off the model's "thinking," passing --responses_api_reasoning_effort none at connection time. The same model is smarter with thinking on when writing code, but in conversation, thinking on means it can't keep up with speech pace.

Gemma 4 31B has appeared on this site twice before. This time it shows up in a different role: the brain that can be pulled out and swapped in this open pipeline whenever needed.

You can't take those models with you. This one, you can

To see what this pipeline is actually worth, you first need to separate it from Doubao and GPT-Live — they're not competing on the same axis. Those companies compete on who talks more like a human. This one competes on who owns what.

Start with the incumbents. Real-time full-duplex voice isn't new. Doubao's voice models have supported real-time duplex conversation since versions 2.0 and 3.0, and this April it rolled out native full-duplex Seeduplex to its full user base. OpenAI shipped GPT-Live in July, listening and responding as you speak. Gemini's Live API is more direct still — audio in, audio out, with no separate recognition, generation, or synthesis stages in between.

This class of model has conversational rhythm built in. It can listen and respond simultaneously, and tell the difference between you pausing to think and you actually being done. Gemini has a setting called Proactive Audio, where the model decides for itself whether to speak up, letting it sit as a silent listener indefinitely. When ByteDance announced Seeduplex, it claimed the false-interruption rate was cut in half and the rate of the model talking over the user dropped 40%.

What Hugging Face built here takes the old route: four stages in a relay, each handling recognition, answer generation, or synthesis on its own. It's half-duplex — it can only listen or only speak at any given moment. Interrupt it and it goes quiet, but it can't murmur an "mm-hm" while you're mid-sentence.

So on "talks like a human," this pipeline loses to Doubao and GPT-Live, and that gap is architectural — it can't be closed.

But there's another side to this ledger. However well those three converse, you're only ever renting: the model lives on their servers, you're calling an API, it's gone the moment you're offline, and if they change pricing or terms or pull a feature, you have no say. This one flips that. The thing is yours.

Where it actually wins

"The thing is yours" comes down to one document: a license. Every stage in this chain, including the brain itself, Gemma 4 31B, is Apache 2.0. That license means you can use it commercially, modify it, and redistribute it, all without asking anyone's permission first.

What that buys you, concretely, comes down to four things.

Same client code — just repoint the endpoint to your own server and the conversation carries on as normal. Source: huggingface/speech-to-speech repo documentation

The next few sections cover how it closes that gap.

It closes the rhythm gap by jumping the gun

Half-duplex's one-question-one-answer rhythm can't naturally match listen-while-speaking. This pipeline's answer is to start early: it begins thinking before you finish talking, betting you won't drastically change course. The next three mechanisms are all one connected system.

It doesn't wait for you to finish before it starts thinking

Once the hardware compressed the answer-generation stage, the biggest remaining chunk of latency moved somewhere unglamorous: confirming you're actually done talking takes time too.

The machine judges that you've finished by counting silence. You pause, it starts counting, and only once the count clears a threshold does it act. During that wait, nothing useful happens — it's purely confirmation. But the threshold can't be too short either, or it'll jump in the moment you pause to catch a breath.

Toggle the two tabs below to see where the time actually goes.

It didn't cut the confirmation time — it hid the working time inside the wait.

This is also why the fast hardware matters so much: the answer has to be fully computed within that one second, or you're back to waiting anyway. Thinking mode alone would blow past that budget, hence turning it off.

Add one more sentence, and it redoes the whole thing

Jumping the gun is only safe if a wrong guess doesn't stick. Its fix: throw the whole segment out and restart.

If you speak up again during that one-second window, that version is scrapped. It doesn't try to tack your next words onto its guess — instead it keeps the earlier audio, stitches it together with what you just added into one complete sentence, and re-runs transcription and generation from scratch. So answer quality never suffers; the model always sees one full sentence.

The cost is a full re-run every time you change your mind. Pause and restart four separate times, and the backend redoes the work four times over. That's exactly why cheap, fast hardware matters so much here.

There's also a detail specifically built to accommodate continuations: normally, starting a brand-new utterance needs 384 milliseconds of detected speech to register, to avoid a cough triggering it — but continuing an existing sentence only needs 192 milliseconds, half the threshold. It would rather treat your pause as still-talking than cut you off.

Once it starts speaking, that turn is locked

Whether you can still change your mind comes down to one action, not a fixed duration: the moment the synthesis module starts producing sound, that turn locks immediately, and anything you say after counts as a new turn. Before that lock, it can keep reopening the window for up to 7 seconds — plenty of room to think, catch a breath, or trail off and add something.

Interrupt it mid-sentence, and it stops instantly

The previous section covered changing your mind before it starts talking. This one covers cutting it off while it's already speaking.

The hard part is that the instant you interrupt, all four threads are mid-flight: the model's still generating text, synthesis is still chunking audio, and the send loop is still streaming it to your ear. Chasing down and killing each one individually is slow and easy to miss.

Interruption can also be turned off. With it off, your speech still gets transcribed, but the system won't stop — it finishes whatever it's currently saying. That's useful for a fixed phone-support announcement or a robot reciting a safety notice.

Once it's set up, here's what it can do

Three commands to get it running

Start the service first, then connect and talk to it from another terminal.

pip install speech-to-speech
export OPENAI_API_KEY=...
speech-to-speech

The OpenAI key is required here because, with no parameters set, the middle stage defaults to OpenAI's gpt-5.4-mini. To swap in this article's subject, just point the model name and endpoint at it instead.

speech-to-speech \
    --mode realtime \
    --stt parakeet-tdt \
    --llm_backend chat-completions \
    --tts qwen3 \
    --model_name "google/gemma-4-31B-it:cerebras" \
    --responses_api_base_url "https://router.huggingface.co/v1" \
    --responses_api_api_key "$HF_TOKEN" \
    --responses_api_reasoning_effort none \
    --responses_api_stream

python scripts/listen_and_play_realtime.py --host 127.0.0.1 --port 8765

If you'd rather not send anything out over the network, keep the middle stage on-device too: in one terminal, host a smaller Gemma 4 with llama.cpp, and in another, point the pipeline at it.

# Terminal 1: llama.cpp hosting Gemma 4
llama-server -hf ggml-org/gemma-4-E4B-it-GGUF -np 2 -c 65536 -fa on --swa-full

# Terminal 2: point the voice pipeline at this local service
speech-to-speech \
    --mode realtime \
    --stt parakeet-tdt \
    --llm_backend responses-api \
    --tts qwen3 \
    --model_name "ggml-org/gemma-4-E4B-it-GGUF" \
    --responses_api_base_url "http://127.0.0.1:8080/v1" \
    --responses_api_api_key "" \
    --responses_api_stream \
    --enable_live_transcription

A few boundaries worth knowing up front

The repo lists its own set of limitations — worth a read before installing.

The most immediate one: the default recognition setup only understands 25 European languages, and Chinese isn't among them — you'll need to swap in Whisper or Paraformer. The command below only runs on Apple Silicon Macs, and it also happens to swap the middle stage for a local, on-device model. On other systems, install speech-to-speech[faster-whisper] or [paraformer] and configure the matching parameters.

speech-to-speech \
    --stt whisper-mlx \
    --stt_model_name large-v3 \
    --language zh \
    --llm_backend mlx-lm \
    --model_name mlx-community/Qwen3-4B-Instruct-2507-bf16

Two more things that'll trip you up during install: the optional audio-enhancement module needs numpy 1.x, while Pocket TTS needs numpy 2.x, and the two can't coexist. On Linux, the default speech-synthesis package expects CUDA 12.8 — mismatch that and it'll fail, so install the matching version first; the repo provides CUDA 13, 12.4, and CPU-only builds.

Want to try it without installing anything first? The online demo works right away, no setup needed. The free quota resets daily at UTC: 5 minutes if you're not logged in, 10 minutes for a logged-in free user, unlimited for PRO subscribers and members of a few designated organizations. Running it locally on your own machine isn't metered at all.

Of the four stages, the slowest is the middle one — generating the answer. No matter how fast the other three run, that one drags everything down. This update hands that stage to Cerebras's inference hardware: instead of slicing a wafer into small chips, that company builds the whole wafer as one giant chip, with model weights sitting directly on-chip — output comes out far faster. At the same time, the model's "thinking" gets turned off: it makes the model smarter for coding, but in conversation it can't keep pace.

Just two parameters changed. Every stage in this chain is swappable the same way: recognition can become Whisper, synthesis can become Kokoro, and the answer-generation stage can become a local llama.cpp instance. But once the hardware sped that stage up, the biggest remaining chunk of latency moved somewhere unglamorous — confirming you've finished talking takes time too.

The machine judges that you're done by counting silence. You pause, it starts counting, and only once the count clears a threshold does it act — nothing useful happens during that wait. Count too short, and it jumps in the moment you pause for breath. This pipeline's fix is to overlap "confirming" with "working": 64 milliseconds of silence is enough for it to assume you're done, recognition and answer-generation start immediately, and the computed answer is held for a full second before it's spoken — your window to change your mind.

"35x faster" doesn't mean much on its own. Converted into something concrete: how long each approach takes to think up a 60-word answer.

Getting it running is one line, pip install speech-to-speech; the web demo works the moment you open it and grant mic access — 5 minutes free per day without logging in, 10 minutes logged in. Two boundaries worth knowing upfront: the default recognition setup only understands 25 European languages, so Chinese needs Whisper or Paraformer instead; and it only serves one person at a time by default, so serving a team means manually raising the pipeline count, with each instance consuming its own share of model memory.