alt Hacker NewsWhat I’m missing from this announcement is the capability to use connectors and tools. I don’t really get it - NONE of the frontier assistants can use tools / connectors while in voice mode - Claude, ChatGPT, Gemini, Grok. It seems so obvious: I want to be able to research stuff, pull up documents, jot down notes and do productive work while I’m talking to it, and not end voice mode whenever I need to connect to an app or service.
It’s weird. The old Claude voice mode WAS able to use tools but when they revamped it, it lost that capability and is now pinned to Haiku :(
So, yay for finally a voice mode that’s powered by a frontier model and hopefully as good as Grok voice, but sad to still not see tool use while in voice mode.
(I haven’t tried it yet, only read the announcement)
Replies
I could see it relating to tools having unpredictable latency but if they already do background hand off to 5.5 then it seems like they could just enable it within that context.
I’ve been using gpt-realtime-1 for my personal assistant that runs my company and plans my day. And it works pretty well, even makes tool calls and all that.
But the multi modal stuff has resulted in a lot of debugging with weird events and message and audio sequences having race conditions, but overall it is pretty awesome.
Looking forward to moving to this model later today and will chime back in with results.
> I don’t really get it - NONE of the frontier assistants can use tools / connectors while in voice mode - Claude, ChatGPT, Gemini, Grok.
Is that true? I have a friend who often brainstorms with Gemini (I think just on an Android phone), and he has it actually do stuff related to the conversation (including adding content to notes).
In any case, you can always vibe code one with pi!
A big part of this announcement does seem to be _delegation_ in the background; they give the example of web search but that could be any tool. I haven't tried it yet either but sounds like they've found a reasonable UX that mixes that sort of high-and-variable latency tool calling (potentially with agentic loops) with a continuously speaking live voice.
Last I tried this is exposed via their sdk and can be built.
This is not true. Realtime 2 can use tools.
Because they take too long to run, and have an unpredictable latency and success rate. Seeing loading spinners and error messages in a visual interface is fine, but it would firmly put a natural language conversation in uncanny valley territory.
Regular chat already supports voice input, so might as well use that.
I've been using tools in the OpenAI SDK with voice for well over a year now.
Just super difficult.
See more here:
https://github.com/sibblegp/ODAI/blob/main/routers/app_voice...
I'm pretty sure that's a major part of this announcement? It delegates to GPT 5.5 which then uses tools.
There's a tiktok of Sam Altman reacting to a viral clip of someone using Voice to time themselves on a mile run (it hilariously failed).
Sam's reaction was "Yea, it doesn't have access to tools like a timer. It's a known issue. Should be coming in about a year"
Edit: here's the clip: https://www.youtube.com/shorts/Py2YgJe8fqQ