alt Hacker NewsConsidering the progress on speech recognition, why do YouTube subtitles have little in common with what's spoken in the video? Shouldn't Google be able to do a good job at auch a task?
Replies
I suspect they're using some new cheap multi-modal model. A couple of times now I've seen the generated subtitles fill muffled speech with text about something unrelated happening on screen.
YouTube subtitles usually are not that bad, but they tend to get wrong on key moments.
For example the sentence may be "They decided to rewrite the application in Rust", and the subtitle be "They decided to rewrite the application in rats". They made a single mistake, and it is on the most important word. I guess it can be explained by the fact that these key words tend to be uncommon while the rest of the sentence just flows naturally, that is, you barely need to listen, you can guess most of it.
It worries me a bit because as models get better, they may come up with more and more consistent, but wrong transcriptions. Like a typical LLM hallucination, so confidently wrong that you don't question it.
Scale is a big part of it.
Every minute, 500 hours of video are uploaded to YouTube.
Even Google can't afford to run the biggest baddest model on every single one of those hours.