> If someone is born without a sense of smell, they don't have people question if they really can't smell anything or if it's a skill issue.

You're talking about something that affects 1 in 10,000 people and is very easily tested for. Whereas seemingly half of all people claim they have aphantasia, and nobody can agree on the definitions of any of the words we use to evaluate it. The situations are not at all similar. I'd be willing to accept that about 1 in 10,000 people are born without some mental capability that others have that tends to be called "visualizing", but that is well under 1% of the people who say they have aphantasia.