Yes he did start with a "small loan of a million dollars"!

Buffett started with 1.2 million dollars (105K in 1956) in investments from his family, i.e. his aunt, sister, and father in law.

It is very nearly impossible to get rich without starting with a huge chunk of money, Buffett is no exception.