The pointer might be something you forced. The compiler needs to do the right thing but if you set the pointer to an unaligned address because you have information on the hardware you can get this undefined situation with nothing the compiler can do about it.
Any reason the hardware pointer can't be accessed via the packed structure?
https://news.ycombinator.com/item?id=48205371