I haven't tried a USB-C to 3.55mm adapter but your experience heartens me.

Headphone jack has been a hard line for me. Having recently moved into the world of wireless charging (I keep a phone 5-7 years and just missed wireless charging being normalized on my last phone purchase back in 2020) I think using the USB port for headphone is finally visible.

I spend a lot of the day with my headphones on and the phone on the wireless charging puck. Not being forced to choose between charging and headphones changes the equation.