Oklab is a great color space that does what you expect[0] much better than HSL/HSV.

[0] https://bottosson.github.io/posts/oklab/. The better a color space matches human perception, the easier it is to certain processing operations, such as converting to grayscale while preserving the perceived brightness.