No, it means you have something unique to say.
The bar is there, but it is lower than you expect. If you have a truly unique point of view to express, that brings some value to the table, slots will open up.
And I've spoken at plenty of conferences. :) Not always in the glamour rooms/slots. But... I did have one talk fill a room out the door. That was a talk on a difficult/controversial topic, and by then... I was probably about as expert as they came on the issue.
I didn't start with that though. I just started with a simple point of view talk. And I'd argue the second version of that talk is still one of the best I've given in my life.
That doesn't mean every talk has to be unique and special. An "introduction to XYZ" talk may have a bunch of equally valid speakers, which all naturally provide a slightly different angle and there is a bunch of factors going in the decision about who gets the slot.
Some talks are plain craftswork, not unique experiences and still very worthwhile.