geometric constraints don't impact the memory access latency at all for now

I don't know; every time Intel/AMD increase cache size it also takes more cycles. That sounds like a speed of light limit.