There are already some companies doing specialised inference hardware, Cerebras Systems for example. Such designs are still early days and I wouldn't be surprised to see more innovation there. Though because custom silicon design takes time I expect a multi-year cycle.

For training, not sure. But even if training runs on GPUs, once you have the model the main cost is inference.