The fundamental difference is that mouse input requires unpredictable motor movements. We can't queue mouse clicks because the user doesn't even know where a button is going to appear. Even if they did, the actual physical mouse movement needed to get there is different every time. There is a necessary mouse -> screen -> eyes -> hand -> mouse feedback loop for every action.

Keyboard input can, when designed well, be fully predictable and require no feedback loop with the computer. The motor movement involves a fixed set of physical buttons, which we can fairly easily train to press very reliably, and that training is then universally applicable across all software.

That is why you can buffer keyboard input but usually not mouse input. With keyboard input, I know what the software is going to do and I know exactly what hand movement will get it to do the next thing, so I can perform that movement even before it's ready. With mouse input, it's not really possible.