I'd say that the opposite is more often the reality, which is why HPC systems tend to have high-bandwidth, low-latency networks.

On the contrary. However, they tend to manually manage memory rather than outsourcing it to a language runtime or a distributed key-value store.