Happy April 1st! A group of folks and I decided to do something different this year: instead of publishing fake things, we’re publishing real posts on very different topics than our readers usually expect from our blogs. The tech content will be back soon! Check out the other April Cools posts here.
An extraordinary amount of ink has already been dedicated to describing various test-taking strategies. Many strategies, especially for standardized multiple-choice exams, are well-known: don’t get stuck and spend too much time on one question; guessing is worth it if you can eliminate some of the multiple-choice options, etc. Most such advice is sound and will boost your scores — but none of it will help you absolutely ace an exam.
A gold medal from the 2018 PyeongChang Olympics. Just like common advice on staying in shape like "try to exercise at least twice a week" won't get an athlete to an Olympic gold medal, common test-taking advice is similarly insufficient for getting a perfect score on an exam. The advice is directionally correct but not useful: Olympic athletes definitely exercise at least twice a week, but that's not the key to winning gold.
Source:
Korea.net / Korean Culture and Information Service, CC BY-SA 2.0
This is because common advice describes what to do in common situations. A perfect score is hardly a common situation, so common advice doesn’t tend to help. For example, to get a near-perfect score, you need to know how to solve nearly every question on the exam — and that means there’s not a lot of room for guessing. If forced to guess, you should still apply the usual “good guessing” advice, but good guessing alone won’t get you a perfect score.
If your goal is to get a near-perfect score but you haven’t gotten there yet, odds are you need more practice on the questions you find easiest in the exam. This sounds impossible, unrealistic, and unwise, but it’s true. To explain why that’s true in a way you’ll believe, I need to tell you a bit more about myself first.
Math Kangaroo
Growing up, I took part in competitions in many subjects, but math and programming were my favorites. Between 3rd grade and senior year of high school, I literally competed in every math competition my parents and I managed to find.
One of those math competitions was Math Kangaroo, an intense multiple-choice exam where students have 75min to solve 30 problems The exam for grades 1 through 4 actually only has 24 problems, with 8 problems in each difficulty tier instead of 10 per tier. The allotted time is the same, so there’s still a significant amount of time pressure, if slightly less than at higher levels. in three difficulty tiers of 10 problems each: easy (worth 3 points each), medium (4 points), and hard (5 points). Contestants may choose between 5 possible answers for each question, and incorrect answers deduct a quarter of the point value of the problem — so a correct answer in the hard tier adds 5 points to your score, but an incorrect answer subtracts 1.25 points instead.
Every year I was eligible to compete in Math Kangaroo — a 10 year stretch At the time, Math Kangaroo was only available to students between 3rd grade and senior year of high school. Today, there’s also a Math Kangaroo exam for 1st and 2nd graders, but that was first introduced several years after I had finished 2nd grade. — I won the competition by posting the national top score for my grade level. Including the “base” 30 points everyone starts with (to avoid negative scores due to too many incorrect answers) a perfect score at Math Kangaroo is 150 points. For the 24-problem Math Kangaroo exams for grades 1 through 4, the perfect score is 120 instead, but I only did two years of that (3rd and 4th grade).
My usual score was around 140, meaning that I had correctly solved all but one or two problems.
People used to accuse me of cheating. They said scores that high were impossible. They said someone must have given me the answer key. They said the proctors must have given me extra time. They said I must have had help solving the problems: from proctors, from my math teachers, from other students, from the government(?!), from the competition organizers and sponsors.
I never cheated. I solved the same problems in the same 75min as everyone else, and yet I solved them differently than everyone else.
Everyone else spent by far the most time practicing on the hardest problems. I drilled the hardest on the easiest problems.
Everyone else tried to get faster at solving the hardest problems, and found that really hard. I found it hard too!
I couldn’t really get significantly faster at solving the hardest problems. But I could maximize the time I had available to spend on those problems, by making sure I could solve everything else as fast as possible with no mistakes.
My objective was “20 problems in 15 minutes or less” — solve all the easy (3-point) and medium (4-point) problems completely correctly in less than 15 minutes, leaving a full hour for the ten hard (5-point) problems. I drilled this repeatedly until I could solve all ten easy problems correctly in 3–4 minutes (~20s per problem), and all ten medium problems in 10 minutes (~1min per problem), without even a single mistake. That would leave me an average of 6 minutes per hard problem, and in practice I’d usually be able to dedicate about 10 minutes apiece to the one or two problems I found the most difficult.
I have no doubt that most if not all contestants with above-average scores could solve just about any question on the exam if given 10 minutes to spend on only that problem. But based on my conversations with them, it sounded like they almost never had the luxury of that much leftover time. Instead, most were just crossing over into the medium (4-point) problem section when I was already working on the 5-pointers.
To make matters worse, the data also showed that most of the above-average performers still made many mistakes in the easy and medium problem sections, which cost them too many precious points. This is critical: you can’t just solve the easier problems quickly, you also have to do it correctly. That’s why you need to practice so hard on the easy problems — you are working your correctness percentage up to 100% while driving your time taken as close to zero as you can. Not one or the other, but both at once!
Test of speed or test of knowledge?
Almost everyone treated Math Kangaroo as a test of knowledge, when in fact beyond a certain level it became mostly a test of speed. The “master the easy problems” strategy is perfect for tests of speed, and completely useless for tests of knowledge. Before you get a perfect score on an exam, you must learn to tell the difference and adjust your approach accordingly.
How can you tell the difference? First, practice for the exam until you can consistently post above-average scores — this is mostly to ensure you have a sound understanding of the subject matter, which is initially the most effective way to boost your scores. Then, ask yourself if doubling the allotted time for the exam would change how much of it you could solve. If the answer is “yes” then you have a test of speed on your hands.
Based on that, the SAT and other similar exams are all tests of speed when aiming for above-average scores. Why are so many standardized tests designed to test for speed rather than knowledge on the high end of the score distribution? I find this an interesting question, and while I have some guesses, I am definitely not a professional test-writer. If you know the answer, please tweet it to me. Back when I took the SAT, it was still scored out of 2400 points and consisted of three sections: math, writing, and critical reading. I speak English as a second language, so I knew that the critical reading section would be my weak spot. I worked hard to improve my critical reading! But I also knew that math was my strongest side, so I also worked hard to make sure I could flawlessly finish every math section with 10–15min to spare — time I could use to rest and catch my breath during the grueling 4-hour SAT exam. Critical reading sections are a lot easier after an extra-long break!
Conversely, this strategy fails completely if time isn’t the limiting factor on your test performance. The best knowledge-based exams are specifically designed like that: MIT’s “open-book, open-note, open-laptop” take-home exams are notorious for this. Nothing quite convinces you that you don’t understand a concept as well as four days of failing to solve a problem while re-reading the textbook and course notes. The four days may as well have been a week and it wouldn’t have mattered one bit.
Real-world tasks can similarly be knowledge-limited or time-limited. Have you ever written a complex and satisfying piece of code only to waste a lot of time having it repeatedly rejected by a linter? Have you ever tried to schedule an important meeting and found yourself spending hours finding a time that works for everyone? Even in the real world, the easiest way to get better at tackling big challenges is to optimize and perfect the easy tasks first.
Thanks to Hillel Wayne, Jeremy Kun, Lars Hupel, and Ben Brubaker for their support and feedback on early drafts of this post. All mistakes are mine alone.