I haven't given this more than 5 seconds of thought, but wouldn't it make sense to only enable the timing attack prevention for pseudo-terminal sessions (-t)?