You're roughly describing Chromatone and their Muto method of notation: https://muto-method.com/en/index.html

Sure, if path dependency was were not a thing, this might make more sense. But it takes an extraordinary amount of time to really get good at music and you don't want to be the only person who speaks a completely different language to the people around you. So it makes sense to stick with what everybody speaks.