alt Hacker NewsI learned speech recognition from the 2nd edition of Jurafsky's book (2008). The field has changed so much it sometimes feels unrecognizable. Instead of hidden markov models, gaussian mixture models, tri-phone state trees, finite state transducers, and so on, nearly the whole stack has been eaten from the inside out by neural networks.
But, there's benefit to the fact that deep learning is now the "lingua franca" across machine learning fields. In 2008, I would have struggled to usefully share ideas with, say, a researcher working on computer vision.
Now neural networks act as a shared language across ML, and ideas can much more easily flow across speech recognition, computer vision, AI in medicine, robotics, and so on. People can flow too, e.g., Dario Amodei got his start working on Baidu's DeepSpeech model and now runs Anthropic.
Makes it a very interesting time to work in applied AI.
Replies
> Gaussian mixture models
In what fields did neural networks replace Gaussian mixtures?
In addition to all this, I also feel we have been getting so much progress so fast down the NN path that we haven't really had time to take a breath and understand what's going on.
When you work closely with transformers for while you do start to see things reminiscent of old school NLP pop up: decoder only LLMs are really just fancy Markov Chains with a very powerful/sophisticated state representation, "Attention" looks a lot like learning kernels for various tweaks on kernel smoothing etc.
Oddly, I almost think another AI winter (or hopefully just an AI cool down) would give researchers and practitioners alike a chance to start exploring these models more closely. I'm a bit surprised how few people really spend their time messing with the internals of these things, and every time they do something interesting seems to come out of it. But currently nobody I know in this space, from researchers to product folks, seems to have time to catch their breath, let along really reflect on the state of the field.