The Nintendo DS fit in your pocket but it didn't hearld in mainstream gaming.

Gaming went mainstream because it went mainstream. Technogy improved, it became easy to setup and play (unlike a computer at the time), competition was high between Sony and MS which resulted in quick generational leaps. And finally, all the gamers grew up inviting more gamers to be gamers.

Once it became big money, gaming attracted attention propelling it further.

Like any technology, it's a combination of things that leads to high uptake.

My opinion is that VR will go mainstream when it becomes afford, useful and easy to use. IMO we are years away from good useful VR.