Short version: "It turned out our chosen base address for the FMC (FPGA Mezzanine Card) bus had a default memory type of Normal Cached."

They accidentally put an external non-memory device behind the cache. That's never going to work right, but might work some of the time.