Your mental model of integer vs double performance sounds outdated by decades. I’d suggest reading up on instruction performance on agnerfog, should be eye opening.