alt Hacker News> I still contend strongly that it is wrong to distill a person into a collection of numbers
No one's saying a person is a collection of numbers.
> My point in telling this story is that the raw score in a standarized test can be helpful, but it does not tell a complete story. It is part of a broader assessment
And that's very noble and well-meaning. But it can also be used to discriminate and deny opportunity. Raw scores, on the other hand, are hard data. You either have them or you don't. Do factors outside of one's control go into them? Absolutely. Those factors are all-pervasive and will influence anything you might use to determine merit.
> Kaplan, one of the test prep companies. They had very good data about how much their test prep improved test scores.
Of course they would say that. Even assuming it's true, Kaplan test prep books are only around $50 on Amazon. You could also get them for free from a public or school library (or the high seas, if that's your thing).
As I understand it, Kaplan has apps and dashboards and more material if you're a subscriber. Is there data about the marginal score improvement using that versus just the books?
When you get right down to it, everyone's starting point is different. Reducing the number of variables improves opportunity IMO.
My continued assertion is that "hard data" can be misleading, wrong, and discriminatory. It requires judgement to understand what it tells us and whether there is error in the collection of the data itself.
As far as Kaplan, I don't have metrics to share, but what I saw shocked me at the time. It appeared as though they collected data as to how much their test prep improved scores and it was pretty dramatic. And, they were smart enough to distinguish between their test prep books and their in person tutoring. In person tutoring worked better and by a lot, but it all helped.
As far as tests, I've never contended that they shouldn't be done, but that these scores need to be part of an overall portfolio. I have in fact sat on admittance boards for graduate school and the two dominant factors other reviewers used were GPA and where they went to undergrad. Both of these were hard metrics and full of bias. I'll also contend that they weren't particularly effective in identifying students who successfully completed the program.
Something to consider, I do think test scores can help screen candidates. For example, if one is going to admit into a math program, then their quantitative score should probably be above a certain level. After that, I personally don't think it matters. In that way, one could just set minimum criteria to be accepted, which includes GPA, tests scores, research, jobs, whatever, and then randomly select from that group. That would certainly eliminate a huge amount bias, though not all, but most people hate this approach because it contradicts this belief that we live mostly in a meritocracy. They want rankings and they want to know who number 1 is. My continued argument is that it's impossible to distill a person down to a hard ranking in a reasonable matter, especially with test scores, and it's not necessary for college admission since one can select a group of applicants for acceptance without ordering them.