Did I at any point in my previous comment say that using non-standard ports was my only line of defence ?

Its security through obscurity, which puts you out of view of the vast majority of the chaos of the internet. It by no means protects you from all threats.