I think I get what you're saying. Another way to think of it is to think about what is the domain in which we're striving for fairness.

We could say the rules of basketball are "internally fair" if what we're trying to do is determine which team played better. But if we start using the results of basketball games to, say, determine who qualifies for a mortgage, then maybe we would say that loan qualification system is not fair.

In a similar fashion, we could say that some hypothetical admissions scheme is fair in that it selects the students most likely to have the characteristics the school is looking for. (If it instead selects students most able to game the admissions process itself, then it's not fair in that way, but let's assume for now that it's fair in terms of selecting for on post-admission performance.) But the overall resource allocation in society, which depends in part on education, may be unfair even if the "internal fairness" of the educational merit system is fair.

My position is basically that it doesn't make sense to get too focused on that education-internal fairness specifically. I'm more concerned with the overall fairness of our society. And I think that if we made our society as a whole more fair, that would make education more fair as a byproduct. If, on the other hand, we do not make society more fair, making education more fair on its own is an underwhelming result. The main reason to make education more fair would be if you believe it will have knock-on effects that make society overall more fair. I think some of the people pushing for increased "fairness" in education (e.g., via affirmative action requirements) believe it can have large effects of that type; but I believe that, while it may have some effects, those effects will be relatively small.