That would be sensible if the traffic signs were in English.
Can you read Chinese? Can you identify what this traffic sign means? https://upload.wikimedia.org/wikipedia/commons/thumb/6/68/CN...
Traffic signs have symbols and shapes. You are allowed to drive in the US with an international drivers license if you don’t speak English. Are they going to arrest someone who doesn’t speak English and got a license in another state?
Only if being illiterate also forbade you from driving, which it does not. You don't need to read the law to follow the law.
Traffic signs are readable by almost anybody regardless of English language skills. A vision test is much more safety-valid than an English language test.