There's typically only a performance penalty if the unaligned load spans a cache line on modern hardware.