> they also contain a whole lot of fixed function hardware that is specialized for real-time computer graphics. This hardware isn't exposed as "fancy instructions", it is set up in other ways, which is achieved through a device driver.

Those hardwired fixed-function parts are getting fewer and less important though (ok, and then raytracing units entered the scene...). Also AFAIK something like a texture sampling operation is indeed a 'fancy instruction'(?), even though the actual texture sampling is done by a 'fixed function' hardware unit.

> This also necessitates a device driver to manage scheduling, synchronization, etc. with the main OS.

I bet this type of GPU/CPU handshaking work could also be baked into the hardware (similar to CPU mirocode) and exposed via a standardized memory mapped IO interface (or whatever the current mechanism is for letting the CPU control the GPU). E.g. a CPU also implements similar protocols to talk to the rest of the computer system (e.g. memory and other CPUs), but this is mainly implemented as "hardware protocols" - drivers "just" offer more flexibility for hardware vendors because they can delegate the standardization into software.

...also a lot of the "CPU / GPU software ecosystem" was probably shaped by CPU and GPU vendors historically being different companies, and once the big consolidation started, the conventions were already set in stone.