The RAMBUS speed is the main issue. The RDP can literally be stalled over 70% of the time waiting for memory. It's extremely flawed.

They could have used SDRAM and it would perform so much better, and I believe the cost is around the same.

If you wanted to cut something, cut the antialiasing. While very cool, it is a bit wasted on CRTs. Worst of all, for some reason they have this blur filter which smears the picture horizontally. Luckily it can be deblured by appliying the inverse operation.

The whole thing about the texture cache being the worst design decision in the N64 just gets parroted so much, but nobody can cogently explain which corner should have been cut instead to fit the budget.