A documentary rehashes arguments that the SAT is biased and meaningless but overlooks evidence that ... [+]
The SAT and ACT have long been attacked as biased and open to manipulation, and the pandemic has helped weaken their influence. But if we understand what the tests measure, they have some value.
Critics have charged that college admissions tests only favor the wealthy, who can pay big bucks for test prep. Last year’s college admissions scandal, with its tales of test-gaming, cast even more doubt on the tests’ validity. As Covid-19 has made mass testing events untenable, hundreds of colleges have made the SAT and ACT optional. A growing number say they won’t consider test scores at all.
The makers of a documentary film originally released in 2018—The Test and the Art of Thinking—are now trying to capitalize on this ferment by streaming it on digital platforms. The film is artfully made and has an impressive collection of talking heads, along with some compelling personal stories from students and their parents. But it basically repeats arguments that have been made elsewhere, and it overlooks evidence that cuts the other way. Here are some of the film’s key critiques, with responses:
1. There’s no relation between test scores and college performance.
The evidence is mixed, but there’s quite a bit of data showing that SAT scores help predict college GPA and retention and graduation rates. (I’m using “SAT” to refer to both tests.) The predictive power of the test is strengthened when it’s combined with high school GPA.
2. The SAT doesn’t measure anything meaningful, including IQ. It just rewards the ability to avoid traps and game the system.
Actually, the scores are strongly correlated with IQ. While some questions may be designed to trip up careless test-takers, the verbal section of the SAT has a lot in common with standardized reading comprehension tests. And high scores on those tests are highly correlated with general cultural knowledge—even after controlling for the effect of intelligence, using a non-verbal test.
So a large part of what the verbal SAT assesses is how much general knowledge and vocabulary test-takers have—and that, far from being meaningless, is a significant predictor of reading comprehension and the ability to acquire new information.
3. The key to high SAT scores is test prep, which only the wealthy can afford.
Test prep clearly can boost test scores. But only up to a point.
One of the interviewees in the film, Leon Botstein, posits a test question on who the president of the United States was when Pearl Harbor was bombed, with four possible answers:
a) Franklin D. Roosevelt
b) Franklin D. Roosevelt, Jr.
c) Harry Truman
d) George Washington
Botstein’s point is that while there’s only one right answer, (a), (b) and (c) are a lot closer to being correct than (d). If you answered (b) or (c), you should get some credit. If you answered (d), you’re really lost.
Consider that question in connection with test prep. Maybe prepping could somehow help you distinguish between (a), (b), and (c). But if you think George Washington was president when Pearl Harbor was bombed—or you don’t know what Pearl Harbor was—then test prep is unlikely to do you much good.
Unfortunately, due to our failure to teach history and other subjects effectively, quite a few high school students might choose (d). There’s no way to know what specific information you’ll need to do well on any given administration of the test, and it’s virtually impossible for students who are lacking a lot of general academic knowledge to acquire enough of it to do well on the SAT in a matter of weeks or even months. (That’s true for the verbal portion of the test. One problem with the film is that it draws no distinction between the math and verbal tests, and math knowledge is far more predictable.)
The film also overlooks an effort by the College Board—launched three years before the film’s release—to provide free online test prep through Khan Academy. The College Board reported that students who took advantage of the test prep got a significant boost in their scores. At the same time, though, scores for students in historically lower-performing groups have declined.
That could be due to the fact that more students in those groups are taking the test, as the College Board argues. Still, the film should have mentioned that the College Board is providing free test prep—and also that test prep alone is unlikely to help many students.
4. SAT questions are culturally biased and high scores only reflect family wealth.
The film doesn’t go into this argument except as it relates to test prep, but others have advanced it vigorously. If the tests are biased against minorities, you would expect that students in those groups would do better in college than the tests predict. In fact, minorities actually earn slightly lower grades than predicted. As for wealth, the relationship between SAT scores and college grades doesn’t change much after controlling for the effect of socioeconomic status.
It’s not wealth or minority status per se that produces high SAT scores. The reason wealthier and white students generally do better on the SAT is largely because those attributes are associated with having a greater amount of academic knowledge and vocabulary. Generally, our education system further disadvantages students who are already disadvantaged, leading to huge gaps in the kind of knowledge that the SAT assumes test-takers possess. If we want to improve the performance of low-income and minority students on the SAT—and in college and life—we need to start using schools to give them access to the kind of knowledge and vocabulary other students acquire at home, beginning in the early elementary grades.
5. An increasing number of states have been opting to use the SAT (or ACT) to satisfy the federal requirement that reading and math tests be given at least once in high school. If this trend continues, the high school curriculum will come to resemble test prep.
That could be a problem. In 2017, a teacher in Florida proudly reported that all students at her high school spent at least 30 minutes a day doing test prep in language arts class, in an effort to boost scores. Still, schools have to administer some standardized test, and the same thing could happen with one that wasn’t the SAT. In fact, especially since the advent of high-stakes reading and math tests 20 years ago, the curriculum at lower grade levels has come to be dominated by test prep.
Given stagnant and declining scores, it’s clear that hasn’t worked, and it won’t work in high school either. One of the film’s interviewees, Erica Meltzer, explains why. The skills the tests purport to measure—things like “identify the supporting evidence for this claim”—are “formal skills,” she says. “Formal skills are not education. They are things that you can do if you have the education. But they’re not actually a substitute for the education and they shouldn’t be mistaken for education itself.”
But this isn’t an argument against the SAT. Rather, it’s an argument for understanding what the SAT measures—and the fact these “skills” can’t be taught directly, in the abstract. They can only grow alongside knowledge.
6. At least some colleges that have made the SAT optional have found that students admitted without it have done well.
That’s great, and perhaps making the test optional should be the wave of the future. The SAT tends to reward speed as well as general knowledge, and—as the film argues—speed isn’t necessarily a reliable criterion. Students who think more slowly may actually arrive at deeper understandings, even if they don’t do as well on timed tests.
But for some—including at least some disadvantaged students—the SAT may be the best chance they have of proving their potential. In states where all students are required to take the SAT, the practice has identified some low-income high-achievers who might otherwise not have gone to college—or at least not to a college that matched their abilities.
And the meaning of high school GPA varies depending on the school. A student who gets an A at a low-performing school might be dismissed by a college admissions officer who knows the school’s reputation—even if that A actually reflects her ability to do college-level work.
As one former student has argued from personal experience, test scores can also provide a corrective for teachers’ assumptions about a student’s capabilities. That student has said that educators at her school saw her as needing remedial education until she scored in the 99th percentile on “a high-stakes test.” Her name? Alexandria Ocasio-Cortez.
