I actually used similar camera draw distance trick in my game Rogue Stargun.
The real way to optimize this stuff really well is for the artist to spend a lot of time making LODS for the distant objects. For the really distant objects, esp for a platform like n64, you can replace the distant objects with billboard imposters which are basically just flat poster textures that swap perspectives at certain angles.
GTA V does this extremely well with many manually made LODs and its very costly
They have a very complicated and robust pipeline that generates all of those LODs automatically. The artists aren't manually creating them.