Maybe attackers would focus on these unused bits for very niche products, but generally no one would waste their time.

In general, drivers make up the largest attack surface in the kernel and many of them are just along for the ride rather than being actively maintained and reviewed by researchers.