That's probably most of it, but the way Japan typically teaches English is sort of notoriously bad. That probably doesn't help either.
> If people do not have a need to learn another language, it becomes an uphill battle.
You do see some Japanese companies talking up the need for English competency; I suppose if more and more companies there use English competency as one thing they're looking for in job applicants, that might cause a shift elsewhere, as suddenly there's a 'need' (and thus a motivation).
There are Japanese diplomats with nearly flawless English skills though…