I believe most objections to GUIs come from people who've never experienced good mouse control. If you're not using a high-end gaming mouse and a high refresh rate monitor, with sensitivity/acceleration curve selected with careful thought, on a mouse pad that suits your settings, you probably have bad mouse control.
Watch some videos of top-ranking aim trainer players so see how high the human skill ceiling is for mouse control, and look at the hardware they're using.
I suspect that is a huge factor, indeed. I often see people using bad mice, with uncomfortable shape, bad buttons, bad sensitivity plus often acceleration enabled. Then on top of that, people are putting mouse in some weird locations, like a narrow slit shelf, or far away from the keyboard (because they have a bad sensitivity configured in part). Or even no mouse and just use trackpad (even the superb one like in Macbooks). Of course GUI would feel clunky that way.