You can check why it’s re-encoding, there’s some guides out there. In my case it was the subtitles! Turns out Safari can’t handle all types of subtitles. Might be worth looking into it.