Guests don't care about charisma, they care who your previous guests were. He early on got Elon Musk as a guest (AFAIK by writing a paper that was overly favorable to Tesla) and managed to snowball that into a big podcast.

Also guests agreeing to go on your show means they already want to talk about something, so in a way it's more important to shut up than ask good questions.