The Grand Illusion
With President Biden’s nomination for the next Supreme Court justice, standardized tests are once more in the news, along with the fallacies offered on both sides.
What both sides fail to admit, at least publicly, is that standardized tests are a tool, nothing more and nothing less. If the tool is poorly constructed, it won’t work well. Even if it’s decently constructed, if it’s applied poorly, the results may not be accurate.
Often overlooked is the fact that tests such as the ACT and the SAT were initially effectively designed to measure the qualities needed by white, predominantly male, upper middle class students to succeed in college. The tests have proved to be, despite claims to the contrary, moderately effective for determining collegiate success for that socio-economic group and for certain hard-working Asian minority students. They’re less effective for other socio-economic groups, for a number of reasons.
Well-designed standardized tests will measure certain results accurately, no matter what detractors claim. The problem is that the results they measure aren’t precisely what the proponents of such tests claim. Tests such as the ACT, SAT, LSAT, MCAT, or GRE measure not only certain types of knowledge, which is their stated purpose, but they also measure indirectly other abilities.
The tests measure the ability to read and comprehend quickly, to recognize and analyze patterns, and to quickly recall facts and techniques and to apply them to a situation, problem, or text presented in verbal or mathematical form.
That means that someone who takes the test who reads quickly and accurately has a tremendous advantage on timed tests, and that advantage effectively allows the test-taker more time and places more pressure on the test-taker who knows just as much if not more but who cannot read as fast. In addition, the tests often don’t measure depth of knowledge or the ability to solve complex and multi-faceted problems.
Tests given at the primary school level can reflect as much the students’ socio-economic backgrounds as their intelligence, because a student from a well-read and well-educated upper middle class background will often have greater exposure to the terms and structures of testing.
Such tests are biased, no matter what backers of the tests say, against individuals who do not read the test language quickly, against individuals from a differing socio-economic background who don’t know all the indirect cultural referents embodied in the test, and against those who have high intellectual levels but who do not process information quickly.
What that does mean is that the tests are generally more accurate in assessing the abilities of an upper-middle-class male who reads moderately quickly than in assessing actual intellectual abilities of someone who comes from a different background.
Such tests can be a useful indicator, but they shouldn’t be used as the sole indicator. Unfortunately, the problem today is that many of the other indicators used previously have become useless. Grade inflation has gotten to the point where there’s almost no statistical difference between students in many schools, and where class rank is often decided by a single bad mark in a single course in the ninth grade [FYI, this isn’t hyperbole]. Neither are outside activities.
Tests also don’t reflect the character and determination of the test-taker. Every year, my wife the professor sees students with good high school grades, high test scores, and good native ability flunk out because they were unwilling or unable, for other reasons, to do work that should be well within their capabilities.
But right now, standardized test scores, flawed and biased as they are, are the most accurate predictor of performance for their original target population, simply because there aren’t any other reliable measures.
For everyone else, whatever other yardsticks are being used to determine their abilities are in fact somewhere between estimates and guesstimates.
A number of years ago I was in a relationship with a woman who had an 11 year old son. Her son (Jay) had a learning disability that impacted his ability to learn new words. In effect, he could not generalize between words – so that words like cat, hat, and bat were totally unique and nothing he learned about the word cat could also be applied to word hat. This meant his process for learning new words was much longer than for the rest of the class who could generalize words (he had no issue generalizing anything else – just words). Jay was a bright kid, but other than Math, Gym, and Art his grades were terrible. For example, if given a verbal test, he could explain at length and with nuance everything he learned from the teacher in Social Studies at a high level. But give him a written test, he was still working on the first question when time expired. In effect, every test was a test of the skills of reading and writing for him in public school.
Jay was very fortunate that they lived in a very large metropolitan area and his mother had enough resources to put him in a private school that specialized in learning disabilities. When they tested, they accommodated for the written language disability, and used an oral test methodology to determine what he really knew. His reading and writing grades were still terrible, but his other grades became much better as they were testing him on the subject matter, without the overwhelming bias of the poor reading and writing.
Jay’s mother and I split shortly before Jay would have started high school, so I don’t know the answer to what was still a major concern. Grades and tests mean a great deal in school, but they don’t mean much in real life. Jay had made great strides in the private school to improve his reading and writing skills but I don’t think they were anywhere near good. Those skills were probably 4+ years behind his age peers. Jay is about 21 today and out of high school. What happens when someone exits the sheltered school life of accommodations and enters the “real world” where there isn’t such accommodations? It’s possible he went to a college that does accommodate, but what about after? I think (hope) he had acquired enough of the reading and writing skills to be able to cope, and thrive, in a world that absolutely requires those skills.
No amount of accommodations in school could compensate for the fundamental skills gap he has. Remedial work has lessened the gap, but the gap remains, is substantial, and will impact his ability to succeed in life. Success isn’t impossible, but will be that much harder for him.
The problem with using standardized tests to predict collegiate success is that it’s a fuzzy goal. Is it getting a degree? Is it learning a lot and growing as a person? Is it paying tuition for four years and not having the institution have to provide expensive services for that one individual?
My freshman year of college taught me that I’d made a huge mistake when I picked that school. I transferred and completed my degree at another school which was a much better fit for me. So my first year was a disaster, but I learned something very important. For the school, was I a collegiate success?
Kevin’s story highlights this point by showing that the student learned a hell of a lot more because of the accomodations provided by his school. Life after school may not provide those accommodations, but that learning will open up possibilities. Expensive for the institution, but precious to the person. To me that’s success.
A multiple-choice test can always be gamed to some degree, but it is subject to mechanical evaluation, as contrasted with a test requiring written answers. I’d credit at least 5% of my score on multiple choice tests in areas where I’m less than confident in my knowledge, to more or less intuitive (for me) test-taking strategy. I’d agree that there is SOME dependence on a particular context, but in general, if one does know the material, whether a reason-by-analogy question uses rural, suburban or inner city context should hardly matter. And of course there are poor and educationally deprived rural white folks out there too, if perhaps not in great numbers.
So the questions might have some bias to or against particular economic or subcultural groups, but they’re not as subject to bias (and far greater expense) in evaluation of the answers as a written test might be, and they at least separate subject matter from composition skills.
Bottom line, NO test but real life is perfect (being much shorter and simpler, it cannot be a perfect model); but I don’t think it’s test bias shortchanging some students so much as their parents (and the educational institutions, if they’re failing even if not overloaded with the uninterested or disruptive) failing to prepare them for either the tests OR lawfully productive real life. Heck, voluntarily READING or even watching TV shows outside of genres most relatable to one’s life can, if the stories give any consideration to realism, offer some broader context that day-to-day experience. In some circles, the peer pressure that defines academic success as submissiveness is also a serious problem…it makes the immediate problem of not getting beaten up into a major distraction from learning.
IMO, people, unless seriously malnourished or abused, are more or less equal at birth (if with individually varied talents and limitations); but upbringing and culture (especially their own culture) contribute heavily to limitations as well as advantages.
It is of course totally politically incorrect to say that even if people are equal, cultures are NOT equal in outcomes nor should they be expected to be. The ones that are disadvantaged perhaps have advantages in certain very narrow circumstances (territorial issues between gangs, say), that are however NOT and SHOULD NOT BE marketable in a broader society (except perhaps in certain styles of alleged art). Change the behavioral incentives to change the culture.
I am always suprised at how many people in the US have A+ grades. In the Netherlands the scaling is from 1 to 10 and 10 is rare. The principal is that above 8-8.5 is for understanding above what has been tought. Of course there are simple test of words for example where you can get everything right but as soon as you have written test perfection is nigh impossible. To have all the A+ grades you have to test only reguritation instead of level of understanding.