> It shouldn't require casting from and to specific pointer types
You don't need to explicitly cast T* to void* (guaranteed to be safe), you only need to cast when converting out of void*.
The rules are basically the same as casting between pointer-to-derived-class and pointer-to-base-class and they make sense.
They make sense but reduce type safety, because once you add the cast the case might hide some real typing issue. I sympathize with the idea that the down-cast should be explicit though.