A non-obvious reason that I think the yellow background would've looked especially bad to people at the time, is that most people doing non-gaming on PCs at the time were using MS-DOS programs in text mode, with 4&3-bit color, where it was very unusual for the background color to be bright.

(It was technically possible to get a bright background color on PCs in text mode, but very few programs did that.)