Fact Checking Paul Tough's "The Years That Matter Most"

Scaling the Hoxby-Turner Intervention to Advance College Going by Low-Income Students

Distortion: "The College Board's effort to replicate Caroline Hoxby's packet experiment was in part an attempt by the organization to correct some of the inequities and imbalances in higher education. But the project had a second purpose as well, arguably just as important, if less high-minded: it was part of a large-scale exercise in corporate rebranding." (p. 71)

Facts: In 2013, Stanford University economist Caroline Hoxby and University of Virginia economist Sarah Turner published promising research about the potential of well-designed informational packets to encourage more low-income students to enroll in selective colleges.

The research suggested that this simple intervention could do a lot of good, so the College Board made a significant investment to replicate the experiment on a much wider scale; we sent every low- and moderate-income high achiever we could identify customized information to help them apply to college. As a nonprofit with the reach and resources to replicate promising social science, we felt compelled to share this promising intervention with as many students as possible.

Over the next several years, as Tough reported in his book, our research team tried different approaches to replicate the impact of Hoxby's study.

The results have been disappointing. But it's false to suggest we hid the results. On the contrary, we published a research paper in 2019 that was covered in the media. "The College Board tried a simple, research-backed way to guide low-income kids into better colleges," Chalkbeat reported. "It didn't work." It's a major and material omission that Tough did not include this fact in the book.

So to be clear, the issue is not that we didn't share this publicly—we did. Where we fell short was not sharing this information in a timely way, despite stating we would. Our uncertainty about the evidence made us reluctant to enter a public dispute until we were more sure of the facts. We accept that criticism and will do better.

That's not unusual in social science, and we continue to analyze this data and try new approaches. We learned valuable lessons and have integrated several components of the initiative into our work, including unlimited and free SAT score sends for low-income students.

It is demonstrably false to say that the College Board pursued this intervention or shaped the findings to expand the SAT. In reality, the SAT grew rapidly because three large states switched from giving all students the ACT to the SAT: Michigan, Colorado, and Illinois. Public documents show that the major factors in awarding these contracts to the College Board were:

• the redesign of the SAT to better align with state standards
• lower price
• better reporting tools
• higher quality service

The information packet intervention played no role in these decisions by key states that dramatically expanded the SAT's reach, nor was it ever intended to.

It's hard to understand Tough's claim that our work and investment here is just a "large-scale exercise in corporate rebranding," or that we've somehow buried the results. We took on this work—and shared the outcomes—in good faith. And we'll keep working to scale the best of education research, seeking every opportunity to propel students forward.

Official SAT Practice on Khan Academy

Distortion: "Official SAT Practice wasn’t helping struggling students catch up—it was allowing students who were already testing well to increase their advantage over the rest of the pack." (p. 92)

Facts: For decades, wealthy families have paid for test preparation programs and private tutors to improve their children's chances on tests like the SAT. So, over the last six years, we've invested heavily to ensure all students, regardless of family resources, have the opportunity to practice the skills they'll need to succeed on the SAT and in college.

In 2015, we partnered with Khan Academy to create Official SAT Practice, the most comprehensive SAT practice available anywhere, personalized for each student, and made it completely free to everyone. More than 8 million users have signed up for the program—three times more than all other test preparation programs combined—which has helped countless students from all backgrounds improve on the SAT and prepare for college.

There are tens of thousands of low-income students every year who can testify to the power of this free resource. Rather than write about them for The Years That Matter Most, Tough instead spotlights the services of a $400-per-hour private tutor in one of the country’s wealthiest neighborhoods. The mere existence of such high-dollar tutors is presented as evidence of College Board complicity in perpetuating social inequity. He goes even further, falsely claiming that the existence of free SAT practice on Khan Academy actually increases inequality. This is absurd; the world before Official SAT Practice on Khan Academy was one in which only wealthy students could pay for expensive test preparation and low-income students had no equal opportunity to practice and improve.

Tough's charges are further contradicted by data we released in 2018 in our Delivering Opportunities report, which shows that students with lower PSAT/NMSQT scores experience larger score gains for the same amount of time spent on Official SAT Practice than students with higher PSAT/NMSQT scores. It is extremely disappointing that Tough falsely claims the College Board obscured information about student practice on Khan Academy while ignoring our seminal report—Delivering Opportunities—that reported on these exact data.

Tough also takes issue with the fact that students whose parents are more highly educated spend somewhat more time—on average—practicing on Khan Academy. We are alert to this reality and shared the finding publicly.

We also took action in response to this data. We forged partnerships with urban school districts across the country to encourage more students from all backgrounds to practice, resulting in students completing over 36 million practice problems during more than 800,000 hours of practice on Khan Academy. We created a scholarship specifically to incentivize practice on Khan Academy as well as other key steps on the path to college. And we trained near-peer advisors in college access organizations to support students using Khan Academy. It is striking that none of these actions merit a mention in Tough's book.

Grade Inflation

Distortion: "In 2017 the College Board launched a public relations campaign to counteract and critique the movement toward test-optional admissions and to call into question the credibility of students' high school grades." (p. 177)

Facts: There is little dispute among researchers and policy analysts that grade inflation is a serious problem. High school grades are rising much faster than any standardized assessment, including ours, would suggest is plausible.

We shared data with Tough, as we have in many public venues—such as a recent Atlantic essay written by our CEO, David Coleman—showing that the problem of inflated grades is worse in wealthier schools and districts. Our analysts have reached that conclusion, as have multiple outside researchers. Instead of engaging with that independent research, Tough distorts a finding from the College Board—that steadily rising high school grades were disproportionately benefiting wealthier students—and implies that our discussion of the SAT as a check against grade inflation was tantamount to a "big tobacco"-style disinformation campaign—an absurd and offensive charge.

As our own research shows, grades are slightly more predictive than SAT scores overall. Taken together, however, grades and SAT scores are much more predictive than either alone—15% more on average. And as Tough’s own reporting on the University of Texas (UT) illustrates, the SAT gives colleges critical insight into the level of preparation students are bringing to college, allowing campuses to target support programs and resources where they’ll do the most good. UT requires all incoming students to take the SAT in part for this reason.

We have repeatedly and publicly stated that college admissions needs checks and balances—grades, independent test scores, and other application materials, all considered together to prevent any one factor from exerting too much influence. This view isn’t just ours—it's widely shared by the general public: "Grades, test scores top list of factors Americans say should be considered in college admissions," Pew Research reported in 2019.

The Utility of Standardized Tests

Distortion: "The threat that the College Board… face[s] from test-optional admissions is more existential than economic. The SAT was designed, in part, to help colleges keep out exactly the kind of students that DePaul's test-optional policy is now allowing in: students with high grades and low test scores." (p.177)

Facts: The College Board believes that there are many situations in which students with lower SAT scores but high grades and other achievements should be admitted to selective colleges. Our support for holistic admissions has been continuous and unreserved. In our annual meeting with our member colleges, our CEO underscored that low test scores "must not be a veto on any student’s life." He has also said colleges should actively look for students who may "have scored less, but achieved so much more."

The College Board's clear position is that grades and test scores together give more insight than either measure alone. As our own research shows, grades are slightly more predictive than SAT scores overall. Taken together, however, grades and SAT scores are much more predictive than either alone—15% more on average. And as Tough points out through his reporting on the University of Texas, the SAT gives colleges critical insight into the level of preparation students are bringing to college, allowing campuses to target support programs and resources where they'll do the most good.

The College Board's 2019 National Validity Study is among the largest SAT validity studies ever conducted, based on data from more than 223,000 students across 171 four-year colleges and universities. The study concluded that SAT scores are strongly predictive of college performance—students with higher SAT scores are more likely to have higher grades in college.

The SAT gives colleges critical insight into the level of preparation students are bringing to college, allowing campuses to target support programs and resources where they’ll do the most good. It also serves as a check against widespread abuse of other aspects of the college admissions process.

Tough notes in his book that test-optional policies have not increased diversity on campus. Some test optional colleges have made real progress in diversifying their student bodies, as have some that do require exams. All of us working in education would like to see a world where achievement gaps don't exist. But the great question we face today is what to do about those gaps.

The College Board works with test-optional institutions—they are our members and have served on our board. Our mission isn’t to give tests—it’s to provide opportunity. And we believe no single test score should be a veto on any student’s future.

Tough makes a further claim here that the SAT favors the rich. This causal claim is inaccurate. It is almost universally understood by researchers that the relationship between income and SAT scores is correlational, not causal. Family income is correlated with access to good pre-K educational experiences, well-resourced schools, good medical care, safe living environments, and more stable family structures. All of these factors combined lead children to greater academic achievement when measured in a standardized way across these different environmental factors. All standardized test scores are correlated with income for these reasons. Every respected researcher will make this distinction.