Some Random Thoughts on Going Test-Optional

Now that we’ve begun to receive applications for 2012, a few interesting things cross my mind:

  • First, so far, only about five percent of our applicants are choosing to apply test optional.  That’s lower than I thought it would be, but perhaps it shows that people know we’ve always emphasized GPA more than test scores, anyway.  (As we should, I might add).
  • Second, I still wonder why so many people are so adamantly opposed to what we’re doing.  Pencilnerd rants regularly, tweeting the same inflammatory and self-serving articles from his blog, none of which contains a shred of data except College Board studies that no one (to the best of my knowledge) has ever been able to come close to duplicating.  But as Upton Sinclair said, “It is difficult to get a man to understand something when his job depends on not understanding it.”
  • Regarding the test prep industry: If you believe that the tests measure something real and academically predictive, are you thus saying that your services make high school students into better college students?  Hmmm.
  • It really is very liberating to think of students as students, and not as test-taking robots.
  • I’ve never said the ACT and SAT are bad.  I’ve never really even thought that. Certainly, students who score well have something on the ball, and that’s probably important for extraordinarily selective places who want to make fine (even meaningless) distinctions between a bunch of 3.9999 and 4.0 students.  But everyone knows really smart people who aren’t great on multiple choice, standardized tests.  Missing out on the opportunity to educate them would be a shame.  Education has been taken over by a test with a low incidence of false positive, because the Ivy League institutions need it.  And too many places want to be like the Ivy Leagues.
  • A colleague sent me his copy of “College Admissions and the Public Interest” by B. Alden Thresher, the former dean of admission at MIT.  It was written in 1966, but pretty much predicted everything we’re experiencing today.  I’m sorry I never had a chance to meet him.
This was all focused by our dinner conversation last night.  We talked about high school students (I have two of them in my house) and how it seems they sometimes lack some critical thinking and writing skills.  (I know this is not a novel opinion for a 52-year old man to have.) I posited after attending college prep sessions at our local high school that our national focus on standardized testing, i.e. choosing correctly on a multiple choice test, has broken down our tolerance for ambiguity, discussion, and real learning. Call it the Fox News Phenomenon, if you like.
And, in some sense, I guess I actually do understand why some people are so opposed to what we’re doing.
(If you’d like to see my slides outlining our thinking on test-optional: Why we did it, and how it will work, you can see a presentation I gave to High School Counselors here.)

A Summary of the DePaul Test-optional Decision

If you’re interested, you can view or download the presentation I gave to High School Guidance Counselors, explaining DePaul’s decision to become the nation’s largest private, not-for-profit university to eliminate an SAT or ACT requirement for freshman admission.

Click here.

What I Never Wanted to be When I Grew Up

Anyone who knows me knows I’m not an evangelist.  I’m not a persuader.  I generally allow people to have their own opinions about things.  If you ask me mine, I’ll tell  you, or if the need is especially pressing, I may  jump in.  “Get the hell out of the way of that truck,” is probably good advice, even unsolicited. But in general, if you think the earth is flat, or one of the two political parties is the one that will save us, or that the Sox are better than the Cubs, go ahead.  Those are things that might be disprovable by others, but not by me.  I have better things to do.

Recently, Beckie Supiano of The Chronicle of Higher Education posted, correctly, I might add, that I love to have fun with data.  She linked to my blog  (I mean, my other blog) where I visualized 2010 IPEDS data.  But I also like to work with data.  I do it all the time.

When DePaul announced last February that we’d become the nation’s largest private four-year university to become test optional, we’d done our homework.  We know our stuff. We have lots of evidence, and collectively, hundreds of years of experience and hundreds of thousands of impressions on the wisdom of doing so.  This was not a spontaneous decision.  And it wasn’t done to raise test scores, increase minority enrollment, game US News and World report, or any of the other reasons the cynics have suggested.  Believe me, in a cynical contest with me, you would lose. Badly.  I’m a pro. But even if the amateur cynics were right,  it doesn’t really matter. DePaul is a university of 25,000 students, and we’re much more than an average test score in the freshman class of 2,500.  Raising that score for the sake of raising it is nowhere to be found on our internal documents or our strategic plans; and it’s not in our thinking.

We knew then, and we know now, for instance, that:

  • If you want to find someone who’s going to be a good college student, look first for good high school students.  And that means looking first and foremost at the high school GPA in tough, college-prep courses.  It’s always been that way; test-optional won’t change that. And, to the surprise of even me, in general it means regardless of the high school you attend.  Cream always rises to the top, it seems.
  • Lots of other institutions we respect–Bates, Holy Cross, Loyola of Maryland, Wake Forest, and even lots of large, public, research universities–have eliminated or reduced the need for standardized tests, and have found that the quality of their classes, as measured by performance and graduation rates, has improved.
  • Even the testing agencies themselves know that there are big differences in test score performance, associated with race or ethnicity, and especially with family income and educational opportunity.  Put another way, standardized tests work best with standardized populations.  And if fifteen hours of coaching can raise your scores from the 70th to the 90th percentile, I’d suggest coaching might be best used elsewhere.
  • The real value in standardized tests is for highly selective institutions who need to make fine (even meaningless) distinctions between otherwise very well qualified applicants.  It’s because the tests have a very low rate of false positives.  In other words, if you have a high score, you’re probably reasonably accomplished.  If you have a low score, it may, but doesn’t necessarily, mean you’re not.  When your admission slots are fixed and small relative to the number of applications you receive, you’re not going to take many chances.  You’ve already got enough students with high scores angry at you for rejecting them.
  • There are other characteristics, including those not measured on a high school transcript, and certainly not by a four-hour test, that determine whether or not someone is going to be successful in college.  In some sense, these things are almost as important as your grades.

So, I figured that we could make a policy decision–a reasoned, data-driven  policy decision–based on research and experience, and other people, including those who might not agree with it, would go along for the ride.  After all, we knew that most students who applied to DePaul would still submit an ACT or SAT score.  Lots would apply who thought “test-optional” was the same as “ability-optional.” (I need to thank my friend Chris Lydon at Providence College, another successful test-optional institution, for giving me that line.)  They’d be denied admission, as they always have been, with or without test scores. But in the end, we’d admit many good, accomplished students who just happened to have test scores lower than they thought they should.

I was wrong. Seven months after the story broke, the Sun-Times in Chicago picked it up.  Seemed odd, but when the story ran, the media jumped all over it, as if they had missed it earlier. I thought it unfortunate that a 45 minute talk with a reporter was condensed to a couple of sentences, one of which was taken totally out of context.   I thought the article missed the point of what we were doing and why we were doing it; I know this stuff inside out, and I couldn’t do it justice in a couple paragraphs (Q.E.D.) But it’s happened before.  Not a big deal.

Pencilnerd  saw it and blasted us, too.  But hey, when you make your living telling people that you can improve their scores and their chances for success in life, I guess you are expected to do so.  We’ve disagreed, and we’ve done so civilly.  I respect his right to have a differing opinion.  And it seems he respects mine.

However, Esther Cepeda used my words (I never spoke to her) and wrote a piece that is wrong on almost every level, and filled bad information and conclusions she’s jumped to based on God-knows-what.  And on top of it all, the headline was offensive and inflammatory, probably on purpose.  She seemed to be proud of her poorly formed opinions. I got angry.  I wrote another post but some thought it a little too blunt.  I took it down, and now I’m writing this.  I hope it makes sense and puts this topic into perspective.

So, for now, at least, I’ve had to turn into something I never wanted to be: A Crusader for the Test-Optional movement.  So be it.  I don’t think the Esther Cepedas of the world will ever get it.  And I know you should never argue with someone who buys ink by the barrel.  But my worst habit is my inability to tolerate pride in ignorance.  Maybe that’s not such a bad thing.

A Response to Esther Cepeda

I’ve been asked to remove this post, as some people think it’s too inflammatory.  While I don’t think so, I’ll allow cooler heads to prevail for now.

For now, suffice it to say that Ms. Cepeda’s recent article in the Sun Times was mean-spirited, wholly uninformed, and devoid of rational thinking. Other than that, it was fine.

And a Final Note: I”ve taken a different approach on my response.  It’s here.  I hope you enjoy it.

How to sell to Not-for-Profits

I’ve worked in higher education for almost all of my 30 years after college, but I did spend some time in for-profit companies. As a result, one of my interests is the differences between the two. I’d guess, based on my interactions in the past, that we on the not-for-profit (NFP) side understand sales people way more than they understand us; I see it every time I get a sales call. In the interest of making those exchanges better, I offer some advice. If you sell to NFP’s, these tips might be instructive:

First, read Corporate Cultures by Deal and Kennedy. You work in a low-risk, fast feed-back industry. I work in a low-risk, slow feedback industry. You may think I need to or want to decide RIGHT NOW, but I don’t. In other words, your sense of time is not ours, for another, many big decisions in NFPs are layered in collaboration and process bureaucracy. In general, things in higher education take lots of time: WAY more time than you think.

Second, your calendar is not our problem; it’s not even on my radar. If the end of the quarter is here and you need to make your quota, well, maybe I can help you, but there has to be something in it for me. Empathy ain’t free.

Third, many of the terms you use are offensive to us (well, not to me, but many of us). Bottom line, profit, leads, and solutions all have layers of meaning to me that they don’t mean to you. Clean it up.

Fourth, it’s likely everyone selling something like you’re selling thinks their product is the best. You’d better be able to tell me why yours is even better.

Fifth, funding models are different. WAY different. I probably won’t be the recipient of the ROI you think your product will generate for me; someone else at the university will. And the cost (and opportunity cost of what I can’t do) is all mine.

Sixth, I can tell how many times you call me. If you don’t listen to my voicemail that says I don’t answer numbers I don’t recognize, well, I’m wondering how smart you might be. Send me an e-mail, like the message says.

Seventh, don’t ask me to call you back so you can sell to me. Ever.

Eighth, don’t use snarky techniques to get through to me, like telling someone you’re returning my call. Your chance goes from slim to none real fast.

Ninth, don’t pretend you’re not going to try to sell me something by indicating you just want to “pick my brain,” or “get my impressions of the market.” I was born at night, but not last night. I know you’re gearing up to sell me something. And it’s going to take more than 30 minutes, no matter what you say.

Finally, I know that theoretically, software can do almost anything if you throw enough development time at it. But when I ask you, I want an honest answer.

More on Going “Test-Optional”

In the weeks since DePaul announced that we were going “test optional” for freshman admission, I’ve received quite a few phone calls and emails with questions. Fortunately, most of them have been very positive, and many of the questions have been thoughtful and engaging.  As is often the case with Internet discussions, the ones who jumped into the fray early were negative, aggressive, and almost always ill-informed.  It’s clear that most had not read the announcement in its entirety.

The biggest sticking point seems to be our collective societal belief that the ACT and SAT are intelligence tests.  Even the testing agencies would tell you they’re not.  So, I’m re-posting an edited version of an earlier blog post I wrote, hoping it clears things up.  This is considerably shorter than the original.

The most fundamental problem in college admission is definitional: No one in college admission can really tell you what they’re looking for in applicants. Oh, we have a general idea, of course: Academically accomplished, intellectually curious, quick on the uptake, creative, a great writer or artist or musician.  You get the picture.  But we can’t really define “it.”

And yet, we have a test to help us find it.

We have other tests, of course: Suppose you are testing for leukemia in children. You know what the “it” that you seek is.  However, the test you design is very different than the one you might design for the academic “it.”  Let me elaborate.

Imagine a grid like this.  For each test you take, there are four outcomes: Either you have the disease or you don’t; and the test comes back either positive or negative.

If you don’t have the disease, but the test comes back positive (Square 1), you usually take another test, or have some additional medical exploration.  This false positive is inconvenient and costly, but not horrible.  Square 2 is bad news, but at least you know about the disease, and can seek treatment.  Square 3 is the most common, and of course the best news: You get a clean bill of health, and in reality, you aren’t sick.  But look at square 4: You have the disease, but the test indicates you don’t.  The “false negative” leads you to believe you’re  healthy, but you’re not.  It is the worst of all outcomes, and people who create medical tests strive to make this outcome as unlikely as possible, even at the expense of accuracy in the other squares.  You can thank our legal system for that.

Because  the academic “it” is nebulous, the creators of standardized tests, of course, have no need to minimize false negatives (presumably, it’s hard to sue when a test doesn’t detect something you can’t define in the first place). But this nebulous thing that the test can’t measure, and that we can’t define, is precisely the thing we’re trying to reward. So, college admissions officers attempt to minimize the risk of a false positive (Square 1): That is, selecting a student who doesn’t have “it,” whatever that “it” happens to be. After all, it’s possible you’ve faked your way through four years of high school by just acting smart, or so some people seem to think.  The test will tell us if you’re the real deal or not.

This is confirmation bias in operation.  Because the tests have low false positives, we see high scorers and believe they must have “it.”  Usually, we’re right (in other words, we don’t know a lot of high scorers who seem intellectually or academically challenged, so a high score confirms the GPA).  The problem comes when we extend that logic, and say that if you don’t score well on standardized tests, you must not have  “it.”  This is called the contra-positive, and it causes people to make logical errors all the time.

Think of it this way:

You can say, “If that’s a poodle, it must be a dog.”  Does it then follow that,”If that’s not a poodle, it can’t be a dog?”  Of course not.  Replace “poodle” with “high scorer” and “dog” with “smart.”  See?  Simple.

And there you have it: The SAT is prized simply because admissions officers believe–and perhaps because daily experience has confirmed to our satisfaction–that it has a very low incidence of false positives. But in trying so hard to avoid false positives, colleges and universities miss an awful lot of talented students who should be in college.

And we think that’s wrong.