The whole thing about the texture cache being the worst design decision in the N64 just gets parroted so much, but nobody can cogently explain which corner should have been cut instead to fit the budget.
You could have saved a lot of money by using CDs instead of cartridges.
If you sell games for roughly the same amount as before (or even a bit cheaper), you have extra surplus you can use to subsidise the cost of the console a bit.
Effectively, you'd be cutting a corner on worse load times, I guess?
Keep in mind that the above ignores questions of piracy. I don't know what the actual impact of a CD based solution would have been, but I can tell for sure that the officials at Nintendo thought it would have made a difference when they made their decision.
The N64's CPU, with pretty much every single game released on the platform, is just sitting there idling along at maybe 30% load tops, and usually less than that. It's a 64 bit CPU, but Nintendo's official SDK doesn't even support doubles or uint64!
Of course, Nintendo clearly cared about the CPU a lot for marketing purposes (it's in the console's name), but from a purely technological perspective, it is wasteful. Most of the actual compute is done on the RSP anyway. So, getting a much smaller CPU would have been a big corner to cut, that could have saved enough resources to increase the texture cache to a useful resolution like 128x128 or so.
It should be noted, though, that the N64 was designed with multitexturing capabilities, which would have helped with the mushy colors had games actually taken advantage of it (but they didn't, which here again, the Nintendo SDK is to blame for).