I'm fine if a teacher or organization decides that thinking speed is an important criteria to evaluate, in which case I think the same time limits should apply to everyone.

I'm also fine if a teacher or organization decides they just want to evaluate competency at the underlying material, in which case I think a very generous time limit should be given. Here the time limit is not meant to constrain the test taker, but is just an logistical artifact that eventually teachers and students need to go home. The test should be designed so that any competent taker can complete well in advance of the time limit.

I only object to conditionally caring about the thinking speed of students.