I am not saying your experience must be the same.

According to the three japanese people in my group here and some other feedback from people living there before, same as you I guess, and they speak japanese quite ok, our conclusion is that being one more is not as easy as in other countries.

I say this from the strictest respect to japanese. I like them, I like their culture.

If you live there you must know perfectly that just bc they act politely does not mean they are thinking you do not bother them. A japanese would rarely tell you that. And if someone did, it is likely to do it in an indirect way, as most asians do. Japanese are in the extreme of that polite behavior.