it's a convoluted path to get there, but you can put in delivery instructions into your profile that get surfaced to AMZN delivery folk. I had a similar issue where the AMZN maps would send drivers to a non-existent intersection.
After putting in step-by-step instructions, it's been much less of an issue.
I did do that. And I think that most of the drivers are not local and the comments just made it more likely they would skip on doing what was going to be clearly a difficult delivery.
However, even after my third revision of the notes – to just remove them entirely – deliveries didn’t pick up again, so I don’t know. It’s opaque to me.