Not ridiculous if you’re flying above the atmosphere. SpaceX has proposed point-to-point rocket-powered hypersonic flights that connect New York to Paris in around 30 minutes.
Obviously the real problem with this idea is environmental: emissions would be substantial and nobody wants an extremely noisy rocket port near their city.
And furthermore you would be able to start only in good weather window for takeoff and landing and Gs on Gemini flights (which were doing the same thing) weren't comfortable either.
Whenever I hear people talk about rocket flights I think of the Stephen King short story "The Jaunt". Humans develop near-instant transportation but you have to be unconscious while travelling. A kid avoids being sedated and is driven insane by whatever interdimensional stuff he sees in transit.
Likewise for every fit 20-something being launched at Mach 5 you'd have 10 octogenarians dying of cardiovascular complications.
Musk has proposed lots of things.
How do you imagine that? First thing coming to mind is the loudness of rocket starts and powered landings. Even for airports that would be too loud. At least with current regulations. So you'd probably waste time getting to some dedicated facility, far out in the midst of nowhere to care about, and getting out of a similar hole on the other side of the trip. And again regulations regarding the closure of airspaces and seas for starts and landings, as it's currently done. Which seems rather incompatible with the current system of commercial flight ops, as it's currently done. Other relevant regulations coming to mind are evacution procedures/general survivability provisions for conventional commercial flights, which are mandatory by law.
However I turn that idea, no matter from which point I'm looking at it, I'm not seeing it going anywhere.