That's true, but astigmatism:
- is clearly defined
- can be measured objectively (with autorefractors, keratometers, corneal topographers)
- can be corrected cheaply ($20 glasses) to eliminate any disadvantage in performance or efficiency
Neurodivergence:
- is not clearly defined
- cannot be measured objectively, and is diagnosed using behavioral observations and cognitive tests
- may rely on 'accommodations' that, in the hands of someone without a diagnosis, would be considered cheating
Imagine I don't have astigmatism. If I were to take your glasses, would they improve my performance in college?
Imagine my legs are fine. If I were to take someone's wheelchair and start using it daily, would that improve my performance in college?
Imagine I am neurotypical. If I were to take 2x the time on a test, would my performance improve?
You misunderstand how neurodivergence is be handled in education. It isn't a single diagnosis, and does not have a single accommodation. We use a catch-all word because it makes it easier to talk about as a collection of issues, but that's not how it's diagnosed or treated.
If you would find wearing noise blocking ear muffs, or sitting on a bouncy chair, or using a typing instrument instead of writing, improves your performance on a test, then yes that should be permitted.
(I do also think it would be a good idea if people had longer for many tests or tests had less on them. That kind of speed is rarely an important part of real world workplaces so those tests are rewarding low-value skills.)