I think the main reason is that when they architected it, RDRAM seemed like the better choice based on price and bandwidth at that time, and they underestimated the performance issues it would cause (RDRAM has amazing bandwidth but atrocious latency).

By the time the N64 launched, SDRAM was better and cheaper, and they considered it was too late to make the switch. Allegedly SGI wanted to make changes but Nintendo refused.

Basically they made the wrong bet and didn't want to change it closer to release.