Yeah no I totally agree. I feel like I have a strong sense of a person's intelligence and their psychological capacity/abilities. I just passively look for it or analyze it in my interactions with them. But, if I don't myself have a grasp of the subtle abstract layers of complexity "above" a certain level, I can't evaluate another person's strengths in those areas, so I can't sense where they sit compared to others (or myself)!
I also think the more you know about things, the more you can see how well other people have integrated those things into their own psyche and how they employ those things, if that makes sense. Two people might both know a certain physics principle but one may elicit a far deeper and insightful employment of that knowledge than the other, even in casual situations.