That's only because your ISP won't have routed that packet to you if someone gave it to _them_. However, if someone was able to get to the ISP-side of the connection that you have with your ISP, and send a packet down the fiber/copper line from the ISP side towards your router, and that packet has a dst of your internal network (192.168.0.1 or whatever), your router will happily route that straight on to whatever internal network you have.

This means that if someone decided to be a bad actor and start tapping fiber lines on the poles in your neighborhood, NAT would do literally nothing to protect you from all the packets they start sending your way.