DFU - great example. If you have a USB device that has a DFU class that needs a custom driver, can dfu-util and the like hook into these userspace drivers? Or do you also need to maintain the application part?
dfu-util actually also just uses libusb under the hood! Any class or device that doesn't have a driver baked into the OS can be implemented like this. And if you'd need the DFU functionality in a different application, you may be able to just simply link parts of the dfu-util tool into your application
Dfu-util is one of those "user space drivers", so if you have a nonstandard protocol you'd have to add it directly to dfu-util. There's no intermediate API.
It's not easy to set up a fake or "remapped" USB device on most OS as far as I'm aware, if you were trying to write an adapter program that modified USB packets.