While I was always a sourced-base/personalized distribution personality type, this is also a big part of why I moved to Gentoo in early 2004 (for amd64, not Risc-V / other embedded per your example). While Pentium-IV's very deep pipelines and compiler flag sensitivities (and the name itself for the fastest Penguin) drove the for-speed perception of the compile-just-for-my-system style, it really plays well to all customization/configuation hacker mindsets.
That is a fantastic historical parallel. The early amd64 days were arguably Gentoo's killer app moment. While the binary distributions were wrestling with the logistical nightmare of splitting repositories and figuring out the /lib64 vs /lib standard, Gentoo users just changed their CHOST, bootstrapped and were running 64-bit native. You nailed the psychology of it, too. The speed marketing was always a bit of a red herring. The ability to say "I do not want LDAP support in my mail client" and have the package manager actually respect that is cool. It respects the user's intelligence rather than abstracting it away.
Since you've been on the ride since '04, I'm curious to hear your thoughts. How do you feel the maintenance burden compares today versus the GCC 3.x era? With the modern binhost fallback and the improvements in portage, I feel like we now spend less time fighting rebuild loops than back then? But I wonder if long time users feel the same.