Which version of svt-av1 are you using? Neon optimisations were added quite recently in svt-av1, and are still a bit incomplete, so it might make a difference.