More like, only if you're playing Schoenberg it's about modulo 12.
Almost all sheet music that you play uses 5 or 7 unequal subdivisions of the octave, and occasionally the rest. Sure, sometimes you'll have one or two of the rest on a regular basis (blues, harmonic/melodic minor), but still most music won't use all twelve.
Isn't modulo-twelve (if we ignore microtonal) the core underlying structure, and then you pick scale (like 2 2 1 2 2 2 1) and a key as a mapping, and then use degree as an 1-based index into the scale?
I don't really know a thing about music, so I could be wrong here - but that mental model at least made some sense to me when I tried to map all the concepts I've heard about into some sort of a mental system.