I often see superbly restored SGI equipment at VCF and also own a few SGI equipment that I hope to get to some point in my life but I have never seen any interesting new software or usage of these machines other than the stock "cool" demo programs(The file manager, the gears demo and others running at the same time). Is there any actual cool homebrew occuring on these platforms?
I'm not aware of any cool homebrew, but there is a certain level of cool being able to compile the code for some N64 games using the original IDO compiler on original hardware. You can even compile one of the many decompiled games like Super Mario 64, Banjo Kazooie and more that all will produce the exact binary shipped on the cartridge byte for byte, all from reverse engineered work to create byte matching equivalent C code.
Yes! I don't use my O2 a lot (I think the PSU is flaky, and I'm not super interested in IRIX), but I'm aware of at least https://forums.sgi.sh/index.php, among other similar sites, full of people porting/developing software for IRIX. It's a pretty active community for a 90s workstation platform, the most active one I'm aware of!
The main watering hole for the hobbyist community around these machines vanished from the internet a while back, taking the forums and info around porting software with it.. I guess some of it is available via wayback etc.
I think the lack of a real usable emulator for SGIs is holding back any kind of homebrew. I say this as one of the developer's that got SGI Indy emulation working in MAME. Yes, it works, but it's too slow and too old to be usable. I spent some time after the MAME effort working on a custom high performance emulator for Crimson/Onyx/Reality Engine, but I've kind of burned out again. Maybe some day if I'm really driven again, and had help. I've done most of the reverse engineering already, it's just a lot of code.
I think that if a high performance, usable emulator for some of the big systems existed I think some of the old software might be rediscovered and show up on the internet.