For starters, why encrypt something literally in the same datacenter 6 feet away? Add significant latency and processing overhead.
Because any random machine in the same datacenter and network segment might be compromised and do stuff like running ARP spoofing attacks. Cisco alone has had so many vendor-provided backdoors cropping up that I wouldn't trust anything in a data center with Cisco gear.
To stop or slow down the attacker who is inside your network and trying to move horizontally? Isn’t this the principle of defense in depth?
Because the NSA actively intercepts that traffic. There's a reason why encryption is non optional
Service meshes often encrypt traffic that may be running on the same physical host. Your security policy may simply require this.
Encryption gets you data integrity "for free". If a bit is flipped by faulty hardware, the packet won't decrypt. TCP checksums are not good enough for catching corruption in many cases.