I have personally been baffled by some Scottish and Indian speakers of English, particularly when I was younger and less experienced. And Singapore English is said to be particularly hard for someone with no previous experience. And I know of a case in which someone from London sat at a table with some in-laws who were speaking a traditional native dialect of southern England to each other and found they understand almost nothing, though that was a few decades ago and the dialect in question is perhaps only spoken by old people today.

When you say you "never had a serious problem understanding people", do you mean you could understand them when you overheard them speaking to each other? Because that, of course, is the real test of how intelligible their language is to you. They may well speak a bit differently when speaking to an outsider. Also, you may be particularly skillful at understanding spoken English. I feel I have got better at understanding British dialects as I got older and gained experience of them. I was terribly confused by some dialects as a child.

With compulsory education almost everyone today has some knowledge of a standard language besides whatever dialects they have learnt. If you want to find someone who only speaks dialect X of language Y you might have to look in places where Y is neither official nor widely taught, or among very old people who never went to school.