OpenAI's Postmortem: How We Built GPT-Live's Real-Time Voice System in Six Months
Making AI speak without stuttering is about more than a fast model. Every link in the chain, from the moment you press the button to the sound in your ear, has to hold up.
- Older voice AI relied on a small model to guess when you were done talking. Guess too early and it cut you off; guess too late and it felt sluggish. This generation removes it altogether.
- Tough questions get handed off to GPT-5.5 running in the background while the front-end model keeps talking to you. You can hear it in a recording in this piece: for the dozen or so seconds the background task runs, the front-end never stops talking.
- A bonus fix: they reworked WebRTC's connection handshake, collapsing six round trips into a single one. The change is heading to the IETF, and both major implementations already support it.
Why older voice AI always sounded a bit off
OpenAI published a rare engineering retrospective on building GPT-Live's real-time voice system in six months—from deleting the "are you done?" model to rebuilding WebRTC's handshake. They're candid about trade-offs at every step. For product people, this post is worth more than a launch announcement: it turns "fast response" from a model metric into a full chain that runs from button to ear.
Before the architecture, here are three recordings. Listen—the difference is in your ears.
The awkwardness of the first two comes down to two root causes.
Root Cause 1: A Small Model Dedicated to Guessing "Are You Done?"
Older voice AI was turn-based, like a walkie-talkie: you talk, you stop, it answers. In the middle sat a small model called a turn detector, whose only job was to decide if you had finished speaking.
That judgment was a no-win game. Guess too early, you get cut off mid-sentence. Guess too late, it stares blankly after you finish—feeling sluggish. It relied mostly on detecting silence, so a pause to think, a side conversation, or a passing car could all be mistaken for "the person is done."
Worse was the sequencing: the small model had to make its call before the much larger model could even start. The two latencies stacked, rather than running in parallel.
Root Cause 2: Three Models in a Row
One generation back was slower still. The earliest ChatGPT voice strung three models together: speech-to-text, the LLM thinking, then text-to-speech. Each step waited on the last, adding latency at every link. Your tone, pacing, and pauses were discarded at transcription—the downstream model only saw a plain line of text.
GPT-Live does two things: kill the detector, then split talking from thinking
OpenAI addressed those two root causes with two distinct cuts.
Cut 1: A Model That Can Listen and Speak at the Same Time
GPT-Live's voice model is full-duplex—it listens and speaks at the same time, more like a phone call than a walkie-talkie. A walkie-talkie forces you to wait until the other side releases the button; a phone call lets both people talk at once.
Since the model makes hundreds of "speak or listen" decisions per second, the dedicated turn detector has no reason to exist. It was removed entirely from the audio pipeline. Pause to think, and it won't barge in. Try to interject, and it lets you.
Cut 2: Split "Speaking" from "Deep Thinking" into Two Models
But fast models aren't always smart ones. Ask a question that needs a live web search, and a lightning-quick voice model fumbles it.
So they made a second change: GPT-Live-1 handles real-time conversation on the front end; GPT-5.5 handles search, code, and retrieval in the background. When you ask for something that needs digging, the front-end hands the job to the back-end, keeps chatting, and weaves the result back when it arrives. That mechanism is called delegation.
This all sounds abstract, but you can hear it. Below is a real conversation recording from the official release, with three tracks: you speaking, GPT-Live-1 responding, and GPT-5.5 running a search underneath: