That's a common problem. It's electrical noise in your signal. The only way I know how to completely eliminate it is using external DA/AD converters and connecting them to the PC using optical wires. We used MADI cards back in the studio back in the day.
you can do a lot with just good power filtering and maybe a ferrule on the USB cable to cut the high frequency stuff before it even gets to device. I'd imagine powered USB hub might help too.
I wonder if there is a market for motherboard targetting musicians that just have extra power filtering on USB power.
There are also just USB devices that have just plugs + some LC filter that might help, for example https://oshwlab.com/wagiminator/usb-power-filter