Like many in this thread, I have a similar setup. I used to use I separate KVM, which did work but had a lot of cabling.

Eventually, I decided on just a simple USB SPDT switch that I designed [0], a USB hub, and manual input switching on the monitor. Even though it takes two actions instead of one, it's a simpler setup and has been quite reliable for years now.

[0] https://shielddigitaldesign.com/posts/2021/susb/