GPT-4o handles speech, vision and text in one network

OpenAI released GPT-4o on 13 May 2024, trained end to end across text, vision and audio so that every input and output passes through one network. It responds to speech in as little as 232 milliseconds and 320 on average, against roughly 2.8 seconds for the three separate models it replaced.

Background

Talking to ChatGPT before this meant three models in a row. One transcribed the speech, one answered the text, and one read the answer aloud. Every step added delay, and the round trip took around 2.8 seconds.

The arrangement also threw away everything that was not words. Tone, hesitation, emphasis and background sound were lost at the first step, because the model that did the reasoning only ever saw a transcript.

What happened

OpenAI released GPT-4o on 13 May 2024. The letter stands for omni: one model trained end to end across text, vision and audio, so that every input and output passes through the same network rather than being handed between three.

It responds to speech in as little as 232 milliseconds, and 320 on average. On text in English and on code it matches the model it replaced, does better on other languages, and costs half as much through the interface developers use.

One thing about the release is often reported wrongly. The announcement showed spoken conversation, but the same post says that what was made available that day was text and image input with text output. The voice mode reached users in September, four months later.

What followed

Human conversation turns over in about 200 milliseconds, fairly consistently across languages. Below that, an exchange stops feeling like a query and starts feeling like talking.

The old arrangement could not reach it at any price, because the delay was the sum of three models rather than the speed of one. That is why the change was architectural rather than an optimisation, and why other laboratories rebuilt rather than tuned what they had.

Why it mattered

Human conversation turns over in about 200 milliseconds. Below that threshold an exchange stops feeling like a query and starts feeling like talking, and the pipeline it replaced could not get there at any price because the delay was the sum of three models rather than the speed of one.

Sources

• Hello GPT-4o. openai.com. Official
• GPT-4o System Card. arxiv.org. Primary source

Cite this page

AI Achievements. (2024). GPT-4o handles speech, vision and text in one network. Retrieved 2026-08-29, from https://achievements.ai/milestone/gpt-4o-real-time-spoken-conversation

@misc{achievements_gpt_4o_real_time_spoken_conversation,
  title  = {GPT-4o handles speech, vision and text in one network},
  author = {{AI Achievements}},
  year   = {2024},
  url    = {https://achievements.ai/milestone/gpt-4o-real-time-spoken-conversation}
}