While true, it's typically not going to be impactful on system performance.
There's a reason, for example, why the linux distros all target a generic x86 architecture rather than a specific architecture.
Ubuntu recently added a more specific target for AMD64v3:
https://discourse.ubuntu.com/t/introducing-architecture-vari...
Some applications may target a generic x86 architecture without any impact on performance.
However, other applications which must do cryptographic operations, audio/video processing, scientific/technical/engineering computing, etc. may have wildly different performances when compiled for different x86-64 ISA versions, for which dedicated assembly-language functions exist.
Not all. CachyOS has specific builds for v3, v4, and AMD Zen4/5: https://wiki.cachyos.org/features/optimized_repos/