In tests that offer a choice of answers, you can exploit hidden patterns
to get full marks if you know how, says William Poundstone.
Our fates in school and beyond are decided by pop quizzes, final
exams, drivers’ exams, and professional exams. Many are true-or-false,
or multiple-choice. If you get stumped by a question, it usually feels
like you can do no better than guess. But could there be hidden patterns
in these tests that point to the right answer?
That’s what I
tried to find out. Universities often post online archives of old tests
with answer keys. A staggering variety of other tests are available on
the web, too. I crunched statistics on a sample of 100 tests, 34 from
schools and colleges and 66 from other sources, with a total of 2,456
questions. The tests included school, college, and professional exams;
actual and practice drivers’ theory tests from 10 US states; newspaper
quizzes on current events, sports, and celebrities; a Cosmopolitan quiz
(“50 Guy Phrases”); and safety quizzes for electricity, condoms, and
food poisoning. I looked for strategies that would aid a guesser and
computed how helpful they might be.
Why is it possible? Journal
articles and textbooks instruct educators to vary the location of the
right answer randomly, though rarely does the literature acknowledge
that this is easier said than done. Human beings are poor at making
truly random choices – whether it’s picking lottery numbers or playing
cards.
Every test maker is different. Some worry about randomising
their answers, while others never give it a thought. My aggregate data
suggest, however, that you can bet on some widespread patterns even when
going into a test cold.
True or false?
Let’s
start with true-false exams. A teacher uses true-false tests because
they’re the easiest type of test to make up and the fastest to grade.
The test maker who resorts to a true-false exam is cutting corners, and
from the strategic guesser’s perspective, that’s good.
Two
patterns were evident. One is that “true” answers are more common than
“false” ones. The average split was 56% true and 44% false.
This
isn’t hard to explain. True statements come more easily to mind.
Recalling a fact is quicker than inventing a falsehood. Test makers
follow the path of least resistance and produce tests with an excess of
trues.
The other finding is that, as expected, there is more
true-false-true-false alternation than in a properly random sequence.
For example, here’s the answer key to a 20-item test from a college
textbook (Plummer, McGeary, Carlson’s Physical Geology, ninth edition):
FTTFTFFTTFTTFTTTFTTF.
Here’s the same sequence as a series of black and white squares:
This is not as random as it looks. One way to judge randomness is to
count how many times a correct answer (true or false) is followed by the
same correct answer. This occurs just seven times out of 19 (the 20th
answer has no successor).
To put it another way, the chance that the next answer will be different from the present one is 63%. That’s more than the expected 50% for a random sequence.
You
won’t be guessing on every item, let’s hope. For the most part you will
know the correct answers to the questions before and after the
difficult ones. That permits this true-false test strategy:
– Go through the entire test, marking the answers you know, before attempting to make any guesses.
– Look at the known correct answers of the items before and after the
one(s) that’s left you stumped. When both neighbouring answers are the
same (both false, let’s say), guess the opposite (true).
– Should the before and after answers be different, guess true (because true answers are more likely overall).
There
is a rich folklore on multiple-choice test guessing. I remember being
advised to pick the centre choice. Based on my data, that tactic
wouldn’t do much good. On tests with three choices (call them A, B, and
C), the options were about equally likely to be correct. With four
options, the second answer (B) was slightly favoured, being correct
about 28% of the time. That’s compared to the expected 25% for four
answers.
With five options, the last answer (E) was the most commonly correct one (23%). The middle choice (C) was the least favoured (17%).
Random struggle
It
appears that test makers intuitively get the proportions right for
three choices but have trouble doing so when there are more than three.
This is in line with experimental findings that the quality of
randomising decreases as the number of options increases.
So to recap, pick the second answer (B) on four-choice tests and the fifth answer (E) on five-choice tests.
Another
surprising thing I found was that the answers “none of the above” or
“all of the above” were wildly more likely to be correct. In one college
textbook with four-choice questions, “none”/“all” answers were right
65% of the time! And in my total sample, a none/all answer was correct
52% of the time that there was such an answer. Assuming this is even
close to being representative, it’s astonishing.
Another interesting rule of thumb alleges that the longest multiple-choice
answer is most likely to be correct. On this question from the
Washington state driver’s exam, the longest answer (c) is indeed
correct:
To turn right, you should be in:
a. The left lane.
b. The centre lane.
c. The lane that’s closest to the direction you want to go.
d. Any one of the lanes.
Test
makers have to make sure that right answers are indisputably right.
Often this demands some qualifying language. They may not try so hard
with wrong answers.
Continuity errors
Still
another trick is to check for what Hollywood calls continuity errors.
Here’s an example from the guidelines for Brigham Young University’s
faculty in the US.
A word used to describe a noun is called an:
a. Adjective.
b. Conjunction.
c. Pronoun.
d. Verb.
The hapless professor used the article an, hearing the correct answer, adjective, in
his head. Then he wrote three alternatives, not stopping to think that
they begin with consonants. A student could use that slip to deduce the
answer.
Like true-false tests, multiple-choice tests show too much alternation.
It
wasn’t too uncommon to find short tests in which no correct choice ever
repeated twice in a row. The answer key played a game of hopscotch.
For
example, for the three-choice tests in my sample (A,B,C), the correct
choice repeated its predecessor only 25% of the time (versus the
expected 33% for a random sequence). This means that a test taker could
gain an easy advantage when guessing just by avoiding the previous
question’s answer.
Deja vu
One final rule: Always trust your instinct. It works, even without a system.
When
you’re at a complete loss, ask yourself which of the answers sounds
most familiar. Correct answers are more likely to ring familiar. It may
be that you were exposed to the answer once and have forgotten it. All
that’s left is a faint sense of deja vu. Guess the deja vu answer.
None of these methods can replace preparation, of course. But if your
hard work seems to have been in vain, you could do a lot worse than
relying on your gut instincts and smarter guesses.
-bbc
