I'm surprised to see no mention of Guix. It solves all of these issues, and already has a good story both for cross and native builds on a variety of architectures.

Adding new targets is deceptively easy, just copy an existing template and substitute your values.

https://codeberg.org/guix/guix/src/branch/master/gnu/system/...

https://codeberg.org/guix/guix/src/branch/master/gnu/bootloa...