To my knowledge - and I'm not an expert here - the N64 hardware is pretty unique and doesn't really resemble any of SGI's other chipsets. Not in precise capabilities - the XZ, for instance, didn't even support hardware texture mapping - and not in overall technical design.

It does seem a little bit like an ultra-simplified, integrated version of the RealityEngine [0]. The RealityEngine had "6, 8, or 12 Geometry Engines" split out across three to six boards, each powered by an Intel i860XP, that then passed their work along to Fragment Generators. This roughly corresponds to the RSP, which was just another MIPS core (with vector/matrix extensions), passing its work along to the RDP on the N64. I'm not sure how programmable the RealityEngine's pipeline was compared to the surprisingly flexible RSP.

Remember, the constraints for a graphics workstation are really different than for a game console - especially on the low-end, totally different corners are going to be cut. An Indy still needed to be able to generate a high resolution display and allow modelling complex scenes for film and TV; but while some degree of real-time 3D was important, it was expected that artists could be modelling using wireframe or simplified displays. A game console was displaying low-resolution and relatively low-detail scenes, but they still wanted them to look aesthetically "complete" - shading, textures, fog, lighting, particles - while running at real-time speeds. SGI used their expertise and built something custom-fit for the job at hand, rather than just reusing an existing solution.

[0] https://cseweb.ucsd.edu/~ravir/274/15/papers/p109-akeley.pdf