Season 5: Episode 9 Transcript
Interpreting Testing: Your Scores May Be Stronger Than You Think
Lee Coffin:
From Hanover, New Hampshire. I'm Lee Coffin, Dartmouth's dean of admissions and financial aid, and this is the Admissions Beat.
(music)
Anyone paying attention to the admissions beat—little "a," little "b"–over the last month or so has noticed lots of headlines about standardized testing. And those headlines were generated by the reactivation of a testing requirement at Dartmouth and then at Yale, and maybe by the time we air, by another place or two. There are a bunch of places that are still studying it and there are the majority, which are still optional.
The headlines have often used the word "trend," and in fact this may be, but it's way too soon to know if this is a trend or if a few very selective places have made a decision that testing should be part of their undergraduate admission processes. I think it's too soon for all of us to say testing is back, but for some, testing IS back. So today, I thought, let's have a conversation that digs into the Dartmouth example. Again, this is not a podcast about Dartmouth admission, but a group of faculty at Dartmouth studied this question. This college reenacted testing as a requirement. And so I'm going to be joined by two of the four faculty who wrote the study to think about what does testing tell us?
And for those of you who are juniors in high school, looking ahead and thinking, "Do I need to take the SAT or the ACT? Do I need to be prepared to have testing as a required element?" the answer is for a few places, yes. For many places it's still optional. And for those of you looking at an optional environment, here's the question: when are your scores worth sharing? And if I can do anything as we have this conversation about testing, it is to debunk the idea that any given score is high or low. All testing is contextualized. The score is reviewed from the place where it was generated. So what are the norms where you live, where you go to school? How did you do in relation to those norms?
So when we come back, recurring co-host Jacques Steinberg, formerly of the New York Times, will rejoin Admissions Beat to lead a conversation on this topic. And I'll be joined by Michele Tine of Dartmouth's sociology department and Bruce Sacerdote from Dartmouth's economics department, and we'll have a conversation about standardized testing. What does a score mean? When does a score add value to a holistic undergraduate application? When does it not? And how can you be an educated consumer about the score you have and whether it's worth sharing or not? So when we come back, we'll dive into this topic of standardized testing as an element of holistic admission.
(music)
Jacques, you're back. Welcome.
Jacques Steinberg:
Good to be here.
Lee Coffin:
It's always good to have you. For those listeners who've not met Jacques, Jacques Steinberg is former New York Times reporter and editor, author of The Gatekeepers and The College Conversation, two really terrific books about selective college admission. And if you're new to this space, go find them and read them and you'll be better for it. Jacques and I are joined by Bruce Sacerdote, the Braddock professor of economics at Dartmouth, and Michele Tine, associate professor of sociology and Chair of our education minor. Hi, Michelle. Hi, Bruce.
Michelle Tine:
Hi, great to be here.
Bruce Sacerdote:
Good to be here. Thanks for having us, Lee and Jacques.
Lee Coffin:
So Michele and Bruce were two of four faculty members who completed a study on behalf of our college president, Sian Beilock, to look at testing and the role testing plays both in our admission process but also as a predictor of performance once students enroll at the college and move through it.
So Jacques, I'm going to pass the mic to you, and Michele and Bruce and I are happy to go wherever you'd like to take us on this topic.
Jacques Steinberg:
I'm pleased to play that role for listeners, particularly students and parents. As Lee said, this is going to be a very practical conversation. We're trying to imagine you thinking about applying to schools that, like Dartmouth, as of the next admission cycle, are test-required or the many, many other colleges that are either test-optional or test-blind. So we hope this conversation will be helpful to you in any of those situations and also helpful to your counselors and to admissions officers at other institutions.
Lee, first for you, can you just help give us some grounding? Turn back the clock to just before the pandemic. How was Dartmouth considering testing in the cycle, say, immediately before the pandemic, and then, what changed?
Lee Coffin:
That's an important first question, Jacques. So if I go back to February 2020, if you had said to me as the dean of admission in that moment, will Dartmouth be test optional? I would've said no. There was no conversation happening on the campus where I work to change what had been a longstanding part of our holistic admission review where testing was one factor among many, not THE factor among many, but an important piece of our assessment of a student's academic credentials, with the transcript then and always being the most important part of our review and testing helping to understand what that meant.
Jump a couple months forward, from February to June 2020, the pandemic had now become a much more all-encompassing public health crisis than we understood in February. The testing agencies contacted the deans of admission and said they were having trouble securing sites, guaranteeing public safety in the sites that they had, and recommending "flexibility" in the way we upheld requirements as we moved into the upcoming admission cycle. And so in June 2020, we paused the required policy and said to applicants, "You can't take it without putting yourself at risk. It is not required for this cycle, and we will read your file with whatever information is in it."
Initially, we assumed it was a one-year pause that ended up being a three-year pause. And for the current cycle we shifted towards "recommended", so saying to applicants, "If you have scores, we'd like to see them, and we recommend or encourage their inclusion." And for next year we will shift all the way back to where we were four years ago with testing as a required element. But what I think you'll hear my faculty colleagues say is "required but contextualized" in ways that deepened as we move through the pandemic admission cycles. And to me, taught me something important about reading scores in the local environment from which they were generated.
Jacques Steinberg:
So in a moment, we're going to turn to Michele and Bruce to talk about their research and help listeners translate it and apply it to their own applications. But a little more background from you, Lee, as you think about using testing going forward or how you used it pre-pandemic, can you talk about how it is one factor among many and just a little bit about the many?
Lee Coffin:
Yeah. I've been an admission officer for over 30 years at several places, and if I had a dime for every time someone asked about testing—and the question was always framed with, "it is the most central part of my application, so therefore I'm worried about it"--and how many times I had to reassure everyone, testing is part of it, not the leading factor in your holistic review, I would have enough dimes to take a nice vacation. It is the question that lingers the most in the work I do. And when we talk about one factor among many, the idea that a college is holistic in the way we meet each applicant means the application has many components. Some of them are quantitative. I think our eyes tend to go in that direction because numbers seem to mean something people can interpret. So transcript, grades, the GPA from those grades, the rigor of the curriculum, and then testing, once upon a time subject testing, more recently SAT, ACT. That's the data that is part of holistic review.
In addition to that, we have essays and recommendations and interviews and extracurriculars, and more broadly, the story of person tells about herself as an applicant that helps us not just assess the candidacy on its academic marriage, but imagine the community we're building on the campus where I work. So when it comes together in September, the class and the broader community is representative of lots of different backgrounds and points of view and ways of being that make this the place that it is. "Holistic" is all of those things wrapped up together. Not one element more important than the other academics foundational, but in the mission process of the place like Dartmouth, most of the applicants are high achievers. So the transcript and the testing are foundational, but often not determinative, if that makes sense, Jacques. You don't get in because your GPA and your scores hit a certain mark, because most of the applicants are at that mark. And so the winning combination is that academic preparation and achievement twinned with the story you tell through all these other parts of the file.
Jacques Steinberg:
So you talked about Dartmouth's announcement on February 5th that it was reinstating the SAT. Can you talk about that decision, how it got made and why, and then we can transition to how critical the research was to the making of that decision?
Lee Coffin:
Sure. So as the pause became a two, three and then ultimately four-year journey, I knew a decision needed to be made around one of two paths. Do we remain an institution with an optional testing policy or do we want to end the pause and resume including testing as a required element? And the early conversation I had with the president who started in June last year was we ideally should make a decision by early winter when the high school juniors were gearing up their search so that if testing were to be a part of the required application, they had time to prepare and start to take the exams. And President Beilock, as a cognitive psychologist, said, "Let's gather some data and look at our experience with testing and not just make an emotional decision, but frame our decision to reactivate or remain optional based on some evidence." And so she in turn invited Bruce and Michele and others to study this with her. So probably a good segue to have them chime in on what their study reflected.
Jacques Steinberg:
Yeah. So Bruce and Michele, picking up the story, imagining our audience of students and parents and counselors and peers, how did you frame this as a research question and what did you find that could be helpful to those who are listening as they think about their own applications?
Bruce Sacerdote:
Michele, would you like to go first?
Michele Tine:
Sure. I'll pick up the thread and then pass the baton to you, Bruce.
We were brought in and President Beilock was very transparent with us and said, "As faculty researchers who have expertise in social sciences and data analysis, we would appreciate if you would empirically look at the role that SATs play in admissions under test-optional and test-required policies. So I'd like you to do some unbiased research, but I want to make evidence-based decisions. So here's the data, take the look." And we reported back to her continuously throughout the project, but she was very good at allowing us to show her what the data showed.
We had four main findings. The last one is the biggest, and maybe I'll pass that one to Bruce. The first was, to be totally honest, fairly expected, which is just that SAT scores are highly predictive of academic achievement at Dartmouth. And so when we looked at things that can predict your first year GPA once you are here at Dartmouth, SAT all by itself was the best predictor. It explains statistically about 22% of the variation. I don't know if we want to get into the stats. High school GPA all by itself could also predict it, but not nearly as well. All by itself, only about 9% of the variation. And when you took them together, they explain about 25%. What that means broken down is that GPA above and beyond SAT is really only adding a little bit of predictive power to how well you're going to do at school. And SAT, more than anything else, is a really good predictor of academic achievement. That wasn't a huge surprise.
The second thing we found was that this, and this was important, is that it was a similarly strong predictor of academic success at Dartmouth for all different demographic subgroups, right? So there was no evidence that SAT scores systematically underestimated the college performance of a disadvantaged group in some way.
Bruce, do you want to take the baton on the big one now?
Bruce Sacerdote:
Thank you. So the thing that's most important about SAT-optional policies is that we found that they can be quite harmful to less advantaged students because students—and it's pretty understandable when you get into it—students don't always know when they should submit and when not. It's not like students are perfectly strategic. And in particular, less advantaged students don't know, don't fully understand the way in which their scores can be used in context and how a given score can be enormously strong given the background that student comes from. They could be a score that might be at the median or even the 25th percentile at Dartmouth might be at the 99th percentile of their high school and their neighborhood, and a very strong signal of how well that student is going to do and what they bring to the table. And so students, particularly disadvantaged students, understandably don't know that because that information is not out there in the public domain and they under-submit.
And so to put it bluntly, we discovered that we're missing out on hundreds and hundreds of really talented kids from less advantaged places that weren't submitting their scores. And in some cases, we as the researchers could actually view their scores because even though they weren't used in the admissions process, they had ended up via either transcripts that were not viewed by admissions or via College Board transmission of scores to Dartmouth. We got to see some of those scores and realize, "Boy, there's some great kids we're missing out on." And so, long story short, we found that test-optional policies can be a real hindrance to people identifying themselves to Dartmouth as really talented.
Jacques Steinberg:
So Lee, as this research was shared with you as an admissions officer, what was your takeaway particularly as it pertains to applicants wondering how to size up their scores?
Lee Coffin:
I think that last point that Michele and Bruce were raising is the most important one as we go forward because it has been my lived experience as an admission officer that that happens that the way the college-oriented media, whether it's a guidebook or rankings or the college itself, reports data. It says typically, here's our mean, here's a mid 50% range. And students look at those two data points in a very flat way and make a decision about including or not including their scores without appreciating that we do go school by school and look at the local norms. And I see that data through a school profile, through census information, and there's this misperception that scores are low or high based on their number between 800 and 1,600. When in fact, the 1,200, the 1,400, the 1,190, whatever it is, is considered in the school that the student attends.
So a 1,200 on the SAT at a school where the mean is an 850 is a remarkably high, high in this example, score from that place, and we read it accordingly. There are other schools where there is a lot of preparation, there's more cultural alertness to testing as an element of admissions, and the means are high. And so a 1,400 in that high school might be less noteworthy because everybody's at or above that data point and we're not discounting it, but in and of itself doesn't tell us the same thing.
So I think the opportunity I see based on this research is, how do college admission officers reimagine the story we tell about testing? I'm committed to redefining what that looks like. So the trick is, how do I find a simple way that a student from any high school can look into a database of some type and say, my score is X, does that score put me in range for Dartmouth's admission process or wherever, perhaps based on these other variables around my school? I just don't think a lot of people know we do it and they don't have in an easy-to-use- tool to do it themselves. And so they go back to "my 1,200 is below what Dartmouth is telling me is their average. I'll withhold it." And in doing so, as Bruce said, they missed the chance to put a really important data point in play for themselves. Does that make sense, Bruce?
Bruce Sacerdote:
Yeah, absolutely. And one thing I wanted to add that Lee and Michele and I have talked about a lot is that a lot of the mechanism for this... The difficulty in evaluating students without testing comes through high schools that are in some cases small and some cases rural, and in all cases high schools that admissions does not have a lot of experience with. And so when we separate the data by higher and lower income students, when we look at high income students, even when they don't submit testing, their probabilities of admissions on average are quite similar with and without the testing because Dartmouth in many cases knows their schools so well and knows exactly how to read the transcript. It's the less advantaged students from schools where Dartmouth doesn't have a lot of experience that testing is really useful where that testing can be used in context and really help that applicant's probability of admission. And as a result, that's why the test-optional thing is particularly harmful for the less advantaged students.
Lee Coffin:
And Jacques, just to channel your news you can use, if you're hearing this, 11th grader, and you just took a PSAT or you're about to take the SAT in March, go with that score report to your guidance counselor or a teacher, but start with the guidance counselor and say, "Okay, what's the norm in this school?" So give yourself that analytical framing. The school knows that what's the average score from the junior class, the senior class at this high school. And if your scores are above that, you're in a really good spot. And if you're above the 75th percentile, you're in an even stronger spot that we see. And so the school profile that they share with us usually gives us that range. And so as readers we're able to say, "This 1,350 from that high school is 410 points higher than the mean. That's a really powerful stat inside our evaluation process. And that will empower you to know that, "Oh, my scores really are strong coming from the school where I am a student."
Jacques Steinberg:
And so to pick up on Lee's point, Bruce and Michele, if you could imagine yourselves as college counselors for a moment, we're talking in this conversation about what will be a very narrow band of selective colleges next fall that will be test-required. But many listeners will be wondering whether to submit their scores to schools that are test-optional. Many counselors will be wondering how to counsel a student whether to submit scores to a school that is test-optional. What you've come up with here is something very fresh and relevant for the profession. How would you help translate it to the kitchen table?
Michelle Tine:
I'll take a stab at it. Okay, so you're asking just to clarify, not so much about Dartmouth, but other institutions that remain test-optional. How does a student know what to do? And the truth is that's the hardest part, and that's what we all need to educate around because I'll tell you in my short time being part of this study, I will say I have never met a group of individuals who are more intentional and thoughtful and think about things in context.
And so gathering information, not just about your own achievements and academic preparedness, but also how it fits into your lived experience and who you are and what you can bring to Dartmouth and what perspective you can bring is a really important part to understand. And I agree that talking to your guidance counselor at your school and pushing them to help you understand, and if you can't find the information, ask them to reach out to those schools and say, "How do you interpret a student with this type of background?" Because I have learned that it is very rarely just the raw number that's taken into some algorithm. That's just not how it works. And I think a lot of people think that that is how it works. And our data suggests that a lot of people think that way because we have a lot of disadvantaged students that aren't submitting scores that would've really, really helped their chances of getting in.
Lee Coffin:
And Jacques, I would even broaden it a step wider. It absolutely helps disadvantaged students, but ther eare middle-income kids who go to public high schools where they're making the same, I'm going to call it, a miscalculation, not because they mean to, but it's just not clear. And I think the question you're asking about places that remain test-optional and the decision to include or not include testing is a really important one because my sense is more places will remain optional than not next year.
And so student by student, you have to look at a score and say, "Should I share this?" And the answer is, yes, when your score lands on the higher end of your local distribution. And they're going to be some students... I've had this question come up a couple of times where they're a student at a well-resourced school, and they may be an under-resourced family in a highly resourced school…and they say, so that distribution's not going to help me. And the way of thinking about that is, but the school, the quality of the high school and its curriculum is not unknown to us.
So to Bruce's observation about familiarity, as colleges like Dartmouth see more and more applicants from different parts of the U.S. and around the world, these are new schools that are sending applicants for the first time. And so more information is valuable as we read holistically and both focus on whose position to be successful on the campus we represent in the curriculum it offers. But also, what sometimes gets lost in this conversation is the way admission officers use testing to help sort a cohort of high achievers, many of whom have A's in their local environments. And all by itself, that's not enough information to make an informed decision.
Jacques Steinberg:
So Bruce, if you were sitting with a family at their kitchen table, or if you were sitting with a counselor in a high school and they were talking both about a school that is test-required and whether to apply with testing as a factor or a school that was test-optional and whether to submit a score, what practical advice for them would you draw from the research?
Bruce Sacerdote:
Yeah, I guess the overarching theme is that you're stronger than you think you are and do not count yourself out. And then, secondarily to that, take to heart Lee's point about, if it's test-optional school, think about where that falls in context, and to Michele's point, where that falls in the context of your application, your other accomplishments, the other things that really show who you are as a person. And so I think that's the most important point.
Michele Tine:
I just thought of one last point: I think test-optional policies are going to muddy the water and make it a little difficult in the future for students moving forward as researchers looked very much at how admissions officers can use SATs in context. But SAT scores are also a very important metric for students to use to determine if a school is a good match for them. Rather, is this a reach school? Is this a solid match or is this a safety school? And students need to keep in mind for those test-optional schools that when they're reporting their median scores on the website, it's just the median scores of those submitted. So there's going to be this inflation that happens year after year after year. And I do worry a little bit in five years out that those test-optional schools median scores are going to just get so high that it's going to be really difficult for a student to determine and calibrate if it's a good match for them. And so in some ways, those test-optional policies are tricky in that way.
Jacques Steinberg:
And correct me if I'm wrong, what I'm hearing you all saying is that your score relative to the rest of your high school may be more important than that median score of the college that's reporting. Is that fair or not?
Bruce Sacerdote:
Yeah, I think that's absolutely right. I think that's... And don't get too obsessed about even that. Now we've given you a more sophisticated statistic, but don't obsess about that. Write the best application you can. I think you should be... If you're going to be biased, be biased towards submitting scores because as I say, have some faith in yourself, have some faith in the process, write the best application you can. And other than the early decision round, cast a broad net and find places, as Michele said, that you think would be a good match. There are many wonderful colleges and universities out there. I absolutely love Dartmouth, but it is conceivable that there are other great places out there in the world and be open to them.
Lee Coffin:
That's spoken like a faculty member, not the dean of admission. But the other thing to just point out to high school juniors is the danger of adding what I still call the verbal section to the math section, so evidence-based reading and writing. That's a mouthful. That to me just says verbal. And math adds up to 1,600. But as admission officers, we use each of those sections differently. One, as the name suggests, looks at evidence-based reading and writing, and one is more quantitative. And those tell us different things about the way we understand your academic achievement, the kind of courses you're taking, what the grades on the transcript mean. So step two of looking at your scores is also saying, are they the same? Is the verbal and the math syncing up? Is one higher than the other because your academic interests flow in that direction? That makes sense to us. One more way of taking a pause and not just every number all by itself is not as meaningful as a lot of people think of this.
Jacques Steinberg:
Can you characterize a little bit the feedback that you received? In part, this test, the SAT, has long been regarded in counseling circles, for example, as a test that is biased, that favors some over others, including those who can afford prep or can be correlated in part to the academic achievements of one's parents or the level of language spoken in one's home.
Michele Tine:
I'm really glad you brought that up because the SAT score is not a bias statistical predictor of academic achievement in college. That is true. But there are income-related and racial differences in average SAT scores that are profound and large in the United States of America and on almost every college campus. On average, higher income students score higher than low- income students, white and Asian students, higher than Black and Latino students. And this is the very main point that critics of the SAT make, and they are accurate. Those gaps exist.
What's a little less accurate is linking these very real concerning gaps to the conclusion that the SAT therefore must itself be a biased measure. Evidence suggests that those gaps are more of a reflection, a snapshot of very deep systemic inequity in our K through 12 educational system and society at large as opposed to the SAT being a bias test.
So when we look at other tests that tons and tons of students in the US take and don't prep for, the one I'm most familiar with is called the NAEP test. It's elementary and middle school students take it. They don't prepare for it because it's not about the individual. It's even very rarely about the school. It's about states and how the nation is doing on math and reading. The size and the nature of the gaps on that NAEP test and elementary school students and middle school students is almost identical to the sides and the nature of the gap on the SAT score. So that isn't saying that these tests are biased, it's saying these tests are reflecting these inequities. We know those inequities because we know and we understand those. That's why Dartmouth reads test scores in context. And so peeling those two pieces apart I think is really important.
Lee Coffin:
And what Michele said is exactly what we do. And I have been struck over and over and over throughout my career as you move just in one state from town to town to town, the public high schools that are probably playing each other in high school sports have very different opportunities in the high schools themselves. And we go into each one and we say, what's happening here and how do we look school by school for opportunity or the challenges that exist there? And we read accordingly.
I think the other question, Jacques, that's popped up in some of the interviews I've done with media has been, is this study saying that the decisions we made during the pandemic were somehow flawed because we were test-optional? And my answer there is no. We were moving through a set of months where everything was scrambled, especially in my work. We weren't traveling. We didn't do campus tours. The high school transcripts had passes that they were calling A's. Testing was very unevenly available. And as admission officers, not just at Dartmouth but across all of our selective peers, we made the best decisions we could make with the information we had in those moments. And the students did the same thing.
So I would not characterize the study or the reactivation as a critique of those cycles, but to say the pool we have today and as we look to the future, it's larger. It's grown in really interesting geographic ways that weren't true even five years ago. How does testing as an element give us more information? More information is good, period. The way we read, and I really appreciate having members of the faculty be able to look behind the curtain and say they're reading really in a pretty sophisticated way that allows us to be responsive to environmental factors that a student navigates. Sometimes those environmental factors are lots of resources and it's called test prep, and that produces a score that looks strong. We see it. We know it. We don't spend a lot of time digesting it. In other places, a score that might be a couple hundred points lower than that one is also strong. And that's not saying we adjust, it's just saying we look for the truth where it is. And that's how we read.
And I'm proud of that, Jacques. I think we take great care in going file by file and giving every student their best shot at making their case. And I've said before, I'm not the dean of denial. Feel like my goal is to find a path towards a yes and more information helps me do that.
Jacques Steinberg:
I like that note, Lee, to end on as I pass the microphone to you for final thought. If there's one thread that runs through the Admissions Beat, it's often a note of reassurance, of realistic reassurance. As we land this plane, is there a note of reassurance that you can help us end this conversation on?
Lee Coffin:
I hope so. I hope that the conversation had many moments where someone said, "Hadn't thought of it that way." And I find when I talk about college admission, whether it's testing or recommendations or selectivity or pick a topic, there's endless chatter. A conversation often leads someone to say, "This isn't as problematic or scary as I thought." And I think the proof point for me is we move forward into the next cycle with testing as an element. That article is really important. I am not saying as the dean of admission, your testing is the focus of evaluation. It is one of, it is an element in combination with the record we have from your high school.
And I think if you're in 11th grade, what I've said in other podcasts when we talked about testing during the optional phase is if you can take the test, take the test. See what your scores are. It's a way for us to meet you. We go to College Board and ACT and say, "Introduce us to students who have this range on the test they took," and that's a way of expanding the talent into parts of the country where we want to be present. And I can say to the search parameters, "Show me anybody with this grade point average who has a score within the top 25% at that high school."
It's a more sophisticated way of imagining testing as a recruitment topic than I think a lot of people understand. But I think the topic for the high school class of '25 as it moves forward is other colleges may jump into this required space. That's part of a search. Looking at the way colleges request information. We have a peer recommendation. Most places don't. We have an interview that's still evaluative. A lot of places have moved away from that. What are the other pieces of this admissions puzzle that the college is also inviting you to share? Those count. Those are elements, qualitative elements, but nevertheless important ones that help you tell your story.
And Jacques, I would just say back to you, as a journalist, I've been surprised by the headlines and how many of them have sprouted over the last couple of weeks. I said to one of my friends, "It's like we landed on Mars." There's been a lot of typing related to this reactivation, and it's not a new policy for us. We are doing what we've done for most of our history. And in the classes that were admitted and enrolled during those required years, exciting, interesting, dynamic, diverse groups of people emerged and we'll continue to do that.
But I think the other thing to end on, Jacques, is just with my social scientists here on the pod with me, evidence-based policy continues to be evaluated. We don't just say "we're done." We're going to keep looking at it and seeing how this story plays out and we'll shift as we need to shift if we need to shift, or I think a lot of people are going to be relieved to see like, okay, this is settled back into inorganic orbit.
Michele and Bruce, thanks for coming on Admissions Beat and sharing your research with us. Jacques, always a pleasure to have you as my co-host. To everyone listening, thanks for joining us. We'll be back next week with another conversation. For now, I'm Lee Coffin from Dartmouth College. Thanks so much for joining us.