The Grand Illusion
… and the test results are in. Utah high school students just received the results of a new test that measures achievement in language, mathematics, and science… and on average, less than half the students met the standards in any of the test areas. Two things about the new test were especially interesting, first that the standards and what was being measured didn’t change and, second, that half of the new test required answers from the student – no guessing from choices provided. While there were schools whose overall student mastery levels reached or exceeded 90% mastery, there were also schools where the mastery levels averaged in the 10%-20% range.
What to me is so obvious, but what has been overlooked for years by both parents and educational bureaucrats, is that multiple choice tests don’t accurately show student subject mastery – they’re far more likely to reward speed readers with moderate subject mastery and great test-taking ability. And multiple choice testing certainly doesn’t measure the ability to reason out a mathematical concept or to write an accurate and grammatically correct paragraph.Those students who excel at multiple-choice testing are generally those who (1) have learned the material and can regurgitate or recall it quickly; (2) those able to read and process the questions quickly, (3) those with a comprehensive understanding of common and standard language and society, and (4) those with higher levels of self-discipline. Not surprisingly, those abilities tend to be associated with families with higher incomes and with students from more demanding schools [most of which are either schools in high income areas or charter schools with extraordinarily dedicated and highly professional staff, as well as generally better resources. There are exceptions, of course, but exceptions, as the saying goes, often prove the rule.
The other long-standing problem with multiple choice tests is that they provide an unrealistic view of the choices in life outside and after schooling. Sometimes, usually very infrequently, life presents you with clear and multiple choices. Most of the time, your choices either aren’t obvious, tend to be between less obnoxious alternatives, or you don’t have a real choice at all. Even when choices seem obvious, they often aren’t, because the most appealing one in the short run may turn into a long-range disaster.
The “advantages” to multiple-choice tests are that (1) they’re theoretically more objective, since whether one adequately written paragraph is better than another can result in subjective grading; (2) theytake far fewer resources and are far easier to grade; (3) they allow theoretically more objective comparisons of teachers, schools, and school systems [except most really don’t measure how much a given teacher has improved the skills of a particular student or set of students].
The fact that Utah test scores dropped drastically across the board when open-ended questions were added underscores dramatically just how limited multiple choice tests are – as almost any veteran classroom teacher can explain.
But then, since multiple choice tests are graded on the immediate answer, and we have two generations in a row raised on multiple choice tests, is it any wonder that we’ve become a society dominated by instant gratification and superficial knowledge, with a continuing decline in true critical thinking?
Re: life not being multiple choice – I got multiple recommendations a while back for a TED talk extolling the benefits of crowd-sourcing answers to the world’s great problems. The paradigm was supported by the relative success of team working on solving big problems in multiplayer games. The speaker never seemed to hit on the point that it is very different to crowdsource an answer to something with a definite and discoverable solution than to an open question.
Being the sort of person who follows through, I went to the speaker’s world-changing website to see how it was going. Tens of thousands of people had responded to a question about feeding the world. Top solutions so far: 1) eat less; 2) gardens on urban roofs. Thank you crowd: job done.
Teaching critical thinking, reasoning skills, etc., require resources, well-educated teachers, and lower teacher/student ratios. Common Core is designed to encourage higher-level reading skills, reasoning at a higher level, and stronger math skills. It should come as no surprise that as state legislators and parents learn more about Common Core resistance to its adoption/implementation is increasing. Legislators do not want to come up with funding, parents resist anything that they do not understand (such as new techniques to solve math problems),and teachers already feel overloaded.
People get what they pay for.
I actually taught a course in critical thinking and analysis for several years at the high school level. The students who took it loved it as something different which THEY could see would help them in life. However, I faced opposition from both administrators and parents because it had no ‘practical’ application; it wouldn’t help them get a job. Need I say more.
Critical thinking is NECESSARY and SHOULD be REQUIRED for getting a job because it allows for adaptation to changing conditions.
I have co-workers – similar duties, same job title, that cannot be trusted to take on/develop a new task because they cannot or will not think around obstacles. Once all the THINKING work has been completed – they are semi-capable of toiling away like mindless drones.
There are several things that concern me. I am not a fan of multiple choice tests, however, they “don’t accurately show student subject mastery” is true of many testing methodologies and not just limited to multiple choice tests. I have not seen data that indicates that bias that comes from family income
is only present in multiple choice tests.
“problem with multiple choice tests is that they provide an unrealistic view of the choices in life “ true, however, I am unsure what type of test or for that matter philosophy gives you a realistic view.
“The fact that Utah test scores dropped drastically across the board when open-ended questions were added underscores dramatically just how limited multiple choice tests are “ could be because of lack of mastery of the material, lack of experience with open-ended questions, or
something else. Your assumption could be correct, but I would certainly need more data and maybe a few experiments, before I would conclude that it was likely correct.
I would be much more likely to blame TV programs and movies for instant gratification than I would multiple choice tests. Since they depict people learning skills that require years in days, forensic results
in minutes that take weeks, etc.
As for a decline in critical thinking, I do not know how much we ever had so I do not know if it has declined. I do know that in all my pre-college public schooling critical thinking was actively discouraged.
I went to large high school a 1000 people in my graduating class, I took an American history course and had a few disagreements with my teacher, about some answers on a couple of exam questions. Mainly why his answer was right and mine was wrong. His response was it was in the book. Mine was here are two other history books written by as far as I can see authors as qualified as the writer of our history book, who hold my position. I was the only person in the school transfered to a different history class mid-year.
Critical thinking at the University level I believe is a function of subject and faculty. In my misspent youth I acquired a Masters in Physics and one in Mathematics and in both those fields critical thinking is a requirement. However, I certainly saw places where critical thinking was discouraged or unnecessary. Including a couple of courses were knowing the philosophy of the professor was sufficient to Ace the multiple choice exams as you just selected the answer most in sync with the professor’s philosophy.
All tests have biases on one sort or another. That, I would certainly grant. Perhaps I should have made one point clearer. You don’t need as great a degree of subject matter mastery for a multiple choice test as for a test where you have to come up with the answer, and I think that was certainly one factor in the lower Utah test scores.