> I think people in general felt they were much freer to speak using English rather than Japanese.

That's true to my experience with Japanese ESL students (not in Japan). Some explicitly told me that themselves; many others had that vibe.