After running a quick analysis by year of all enrolling freshmen for the baseline, I decided to create a “Lake Wobegon” model, where everyone in our freshman class is above average.  I re-calculated the class ACT average, using only the scores of students who scored at least 21 on the ACT.  (If you’re from the East or West Coast, here is an explanation of the distribution of ACT Scores.)  The median score is between 20 and 21.

Here is what I got, using Excel.  The actual scores of our freshmen in the given year is in blue; the Lake Wobegon scores are in red.  You can click on these images to see them larger if you’d like:

WOW.  What a dramatic effect, right?

Not really.  Here is where Excel’s most annoying tendency comes in: It thinks it’s smarter than you, and shortens the y-axis, because it thinks you want to make very small differences noticeable to the viewer.  Note that in this chart, the bottom is set to 24.

In this case, I don’t want to exaggerate differences.  I want to show reality.

How to fix?  Let’s try labels.  I ran the decimal places way out so you could see for yourself the actual numbers:

Yet, this is even worse: Your eye is telling you one thing, while your brain tells you another.  What are actually very small numeric differences appear to be large.

So, do this: Always (well, almost always, unless you have a really good reason not to) set your y-axis to zero.  Now, look at the visualization, and tell me if it lines up with the numbers.

I like this demonstration for several reasons: First, it shows with some certainty the futility of attempting to manipulate average test scores by going test-optional.  As I’ve also said before, if that were our intent, there are many far-easier ways to do so.

But equally as interesting to me is looking at the way our most common data visualization tool, Excel, so frequently gets it wrong on so many levels.  I know better, but too many people doing this don’t and it contributes to bad analysis and often bad conclusions.  Don’t let Excel do your thinking for you.  Tell the story you want to tell, not the one Excel thinks you want to tell.

While we were doing research on our decision to become the nation’s largest private, not-for-profit university that is test-optional in freshman admission, I was doing some work studying patterns in ACT data.  With the help of some folks in one of the nation’s largest schools of Computer Science I was able to machine extract ACT data from several years of administrations and create a small database of about 11 million records.  I then began to visualize it.

I came up with several ways to look at it, but eventually wanted to see if there was a way to show two things at once: The relationship between ethnicity and test scores, and the relationship between income and test scores.  For those of you who live on the east or west coast, here is an explanation of ACT scores.  For this purpose, I’m only visualizing Composite Scores.  If you still need more help, check out this concordance with SAT Scores.

I created the y-axis as the ACT composite score, and then used self-reported income bands as the x-axis.  Both are discrete, categorical values, even though ACT is numeric.  ACT increases bottom-to-top, and income bands increase left-to-right.  At the intersection of each variable is a pie chart, sized by the number of students in that group, and colored by ethnicity, using this legend:

I call the result a feather chart, for obvious reasons, and while it’s probably not very practical (and while lots of people hate pie charts for good reasons), it’s very instructive in this instance. We’re looking at overall trends of 11 million records: We don’t need to know in this instance that 25 ACT with $80K of income is slightly larger or smaller than 23 ACT with $100K of income.

On first glance, you can see a couple of things: The fattest part of the feather–the median test score–increases with income.  The higher-income pies are more purple (that is, more Caucasian).  And moving from left-to-right, you get the double-whammy of the relationship between income and ethnicity in the US.

It’s important to note here that being poor or being Hispanic, for instance, doesn’t cause you to score lower on the test (or at least this wouldn’t prove that hypothesis); rather this shows that there is a correlation between the two.  Causality is best left to statisticians (who, of course, will only say that we cannot disprove that there is causality.)  Close enough, I guess.

The lesson? Play around with your data, and try things you normally wouldn’t think of trying.  You might get lucky, even if impractically so.

Inflammatory and illogical posts like this are good to drive traffic to your site, but really, doesn’t some editor owe the reading public a laugh test before publishing?

The College Board/ETS and ACT run a variety of businesses: Testing, of course; curriculum programs like AP; prediction services; net-price calculators; financial aid analysis, and even non-cognitive variable assessment.  And by doing so, they have become the de facto brokers of student lists in the nation.  Criticizing colleges for using this service is like criticizing Boeing for using GE engines on their aircraft while using Sylvania for their on-board light bulbs.  It just doesn’t follow.

If it did, you’d have to then argue that if you bought ACT names, you’d have to require the ACT of every applicant; or if you bought the College Board names, you’d have to require the SAT for every applicant.  In fact, no one does (that I know of) and no one requires the PSAT at all, despite the fact that the PSAT search is the most widely-used search in college admissions.

You may get some traction on this argument if you demonstrate that a college is only buying the names of high-scoring students while saying publicly the tests aren’t valid predictors.  But even the most ardent of test critics admit that at the top end, the tests do work due to their low rate of false-positives (I’ve written about this earlier).

However, we’ve done the opposite; purchasing more names of students with high GPA and lower test scores. We can’t buy the name of every student, of course, because, well, we have a budget.

It’s dangerous for non-practitioners to wander into higher education and start criticizing the way we do things, yet it’s become sport for some.  I recommend Bloomberg stick to Wall Street in the future.

• How could they turn my kid down, with her GPA and her test scores?
• What does it take to get into that place, anyway?
• They took him but deferred her?  How could that be?

The answer, of course, is that there is no answer; college admissions does not operate like the rides at Disney World, where if you’re tall enough to ride, you ride.  Think of it more as the line at the hottest L.A. nightclub: There are some things that increase your chances for admission, some things that reduce it, but in the end, it’s a judgment call after all the factors are considered. (This is, of course, dramatically oversimplified, and should appropriately bruise the egos of all admissions officers, even those with bulging biceps and black t-shirts).

But as I was doing some analysis on our last five admissions cycles using my favorite data visualization tool, something occurred to me that is very obvious to most admissions people, but maybe not anyone else: Admission is easy, and denial is hard.  That is, if you look back at a year, you can extract a more logical pattern from those whom you admitted than from the stack of denials.

Let me explain this visualization (the patterns tell their own tales, but because this is live student data, I can’t let you look at the actual visualization, so I have to explain it using a screen shot.)

Each dot is a person.  The dots are colored by ethnicity.  On the left are the 6,000 or so students we denied last year; on the right are the 10,000 or so we admitted to get our freshman class of 2,458.  On the x-axis is the best-ACT score (either the student’s best ACT or the SAT converted to ACT equivalent using a concordance.)  On the y-axis is the student’s GPA, converted to a 4.0 system.  The gray bars represent the width of one standard deviation either side of the mean, roughly 68% of the middle, leaving  about 16% on either side.)

See how tightly and nicely the admits line up?  And how scattered the denials are?  That’s my point.  If you think of admissions as just looking at ACT and GPA, it makes no sense.  But when you understand that there are many other factors you can’t see here–all tied to a real human being, with all his or her complexities–it makes sense.

The 32 ACT with the 3.9 GPA denied? Applied to the DePaul School of Music, but is not as good on the trumpet as  in World History.  How about 29 ACT with a 3.95?  Suspended in the senior year for serious disciplinary reasons.

The 18 ACT with the 3.0 admitted?  A recent immigrant, who did not speak English until 9th grade, but who got all A’s in junior and senior year in a challenging curriculum, including solid IB grades.  And his first term at DePaul was very successful.

Everyone has a story, and admission goes beyond your record in high school to learn yours.  We’re eager to hear what you have to say.

There’s this guy out there, Douglas Groene.  He calls himself the Pencil Nerd, as if he’s never really looked up the definition of nerd.  Or perhaps he has, and thinks that wishing can change the definition of words.  Or perhaps more accurately, trolls and nerds are somehow related.  I’m not sure, but I do wonder about that nom de plume of his; it sounds to me like someone with an unusual fetish or an image problem.  Whatever.

Anyway, he makes his living selling test prep services, and he has a blog, PencilNerd.com where he preaches about the value and importance of standardized tests in college admissions.

For the last year or so, ever since DePaul announced that we’ve decided to go test-optional in freshman admission, Mr. Groene been tweeting ad nauseum his blog post about our change in policy, calling it a “horrible decision.”  He is of course entitled to an uninformed opinion, and far be it from me to disabuse him of it.  After all, a man is entitled to earn a living, and, as Upton Sinclair wrote, “It is difficult to get a man to understand something when his job depends on not understanding it.”

In the last day or so, though, he has labeled me as a member of the “anti-SAT activist” group along with Bob Schaeffer at FairTest, and Joseph Soares, a professor at Wake Forest .  A couple of things, Mr. Groene:

• I’m not anti-test.  Tests do measure some things.  But they don’t predict anything of much value by themselves, or even in addition to high school GPA.  If you have done any research at all, you know this.  But I doubt you have, and I doubt you are truly interested in the truth.
• You suggest that I have an agenda.  I don’t.  I started with the facts and came to some reasonable conclusions, unlike you who spout your opinions and then argue them as facts: A logical fallacy referred to as “begging the question.”  Your website says you are an attorney. I think lawyers call it submitting facts not in evidence.
• I’m only trying to do what is best for students: To find a way to select those who are going to be successful at my university.  You, on the other hand, depend on the value of the tests (and students’ fears about them) to make a living.   Perhaps you might concede that you are far less disinterested than I?
• You make blanket statements about why DePaul made this decision, but you didn’t talk to me.  I don’t think you read anything we wrote, in fact.  You have only blanket, unproved aphorisms on your side of the argument; not a shred of evidence about whether this decision is right for us.
• To that point about our motivation: We’re reporting all the scores of all the students who enroll; but if we wanted to inflate scores, there are far easier ways to do it, such as reporting “Super Scores.”  We never have; we don’t, and we won’t.  We could also inflate application numbers by using VIP Apps, Fast Apps, and other methods.  We don’t do that, either.  In fact, none of the reasons you listed apply to us. Yet you persist.
• You say we (our newly-named group) use “distorted statistics and lies,” even though we cite scads of unbiased research, conducted over decades.  You cite studies conducted by the College Board.  See above point about neutrality.

If you’re interested, here’s a little presentation I did to High School Counselors.

I understand that you are probably a pro at taking standardized tests, and you thus may believe you are smarter than the rest of humanity.  But this fact should, prima facie, support the point that they are not measures of critical thinking skills, and certainly not of intelligence.

So, you keep making your points, if you must.  But the more you rant and the more you post, the more you expose just how hollow and vapid your opinions are.

Both seem eerily similar: One law school and one undergraduate college have admitted that for years they, um, well, lied about admissions data.  They changed LSAT/ACT/SAT/GPA averages, inflated application and yield numbers, and depressed admit numbers, among other things.  It’s one thing to do this for USNWR, I suppose, but quite another when you submit to state and federal reporting agencies, or worse yet, bond rating agencies.  “Pecuniae obedient omnia.”

It’s not hard to hypothesize why this happened, of course; a lot of people are under a lot of pressure to make numbers look better.  Strategic plans call for an increase in these measures as a testament to academic quality, and to appeal to more students with similar credentials.

It happens with teachers or whole school districts and state-mandated standardized testing on occasion.  And it’s not hard to see some parallels in things that happen regularly in our profession and are passed off without a second thought: Using “click here to apply” emails that raise application numbers to increase the appearance of selectivity; and reporting “super scores” that don’t exist anywhere but in the imagination of some analyst.  Sometimes, as in the case I’m purposely ignoring because it’s not about admissions, overlooking things like this can be a form of complicity.

But what’s most interesting to me about all this is a simple little fact: As far as I can tell, no one on the faculty came forward and said, “Hey, this entering class with an average X of Y really seems more like their average should be Z. Something must be up.”

Instead, in one case it was an auditor, and in another, an inside snitch.  What does that say about the way we use quantitative averages to measure and compare things?

• First, so far, only about five percent of our applicants are choosing to apply test optional.  That’s lower than I thought it would be, but perhaps it shows that people know we’ve always emphasized GPA more than test scores, anyway.  (As we should, I might add).
• Second, I still wonder why so many people are so adamantly opposed to what we’re doing.  Pencilnerd rants regularly, tweeting the same inflammatory and self-serving articles from his blog, none of which contains a shred of data except College Board studies that no one (to the best of my knowledge) has ever been able to come close to duplicating.  But as Upton Sinclair said, “It is difficult to get a man to understand something when his job depends on not understanding it.”
• Regarding the test prep industry: If you believe that the tests measure something real and academically predictive, are you thus saying that your services make high school students into better college students?  Hmmm.
• It really is very liberating to think of students as students, and not as test-taking robots.
• I’ve never said the ACT and SAT are bad.  I’ve never really even thought that. Certainly, students who score well have something on the ball, and that’s probably important for extraordinarily selective places who want to make fine (even meaningless) distinctions between a bunch of 3.9999 and 4.0 students.  But everyone knows really smart people who aren’t great on multiple choice, standardized tests.  Missing out on the opportunity to educate them would be a shame.  Education has been taken over by a test with a low incidence of false positive, because the Ivy League institutions need it.  And too many places want to be like the Ivy Leagues.
• A colleague sent me his copy of “College Admissions and the Public Interest” by B. Alden Thresher, the former dean of admission at MIT.  It was written in 1966, but pretty much predicted everything we’re experiencing today.  I’m sorry I never had a chance to meet him.

Click here.

Recently, Beckie Supiano of The Chronicle of Higher Education posted, correctly, I might add, that I love to have fun with data.  She linked to my blog  (I mean, my other blog) where I visualized 2010 IPEDS data.  But I also like to work with data.  I do it all the time.

When DePaul announced last February that we’d become the nation’s largest private four-year university to become test optional, we’d done our homework.  We know our stuff. We have lots of evidence, and collectively, hundreds of years of experience and hundreds of thousands of impressions on the wisdom of doing so.  This was not a spontaneous decision.  And it wasn’t done to raise test scores, increase minority enrollment, game US News and World report, or any of the other reasons the cynics have suggested.  Believe me, in a cynical contest with me, you would lose. Badly.  I’m a pro. But even if the amateur cynics were right,  it doesn’t really matter. DePaul is a university of 25,000 students, and we’re much more than an average test score in the freshman class of 2,500.  Raising that score for the sake of raising it is nowhere to be found on our internal documents or our strategic plans; and it’s not in our thinking.

We knew then, and we know now, for instance, that:

• If you want to find someone who’s going to be a good college student, look first for good high school students.  And that means looking first and foremost at the high school GPA in tough, college-prep courses.  It’s always been that way; test-optional won’t change that. And, to the surprise of even me, in general it means regardless of the high school you attend.  Cream always rises to the top, it seems.
• Lots of other institutions we respect–Bates, Holy Cross, Loyola of Maryland, Wake Forest, and even lots of large, public, research universities–have eliminated or reduced the need for standardized tests, and have found that the quality of their classes, as measured by performance and graduation rates, has improved.
• Even the testing agencies themselves know that there are big differences in test score performance, associated with race or ethnicity, and especially with family income and educational opportunity.  Put another way, standardized tests work best with standardized populations.  And if fifteen hours of coaching can raise your scores from the 70th to the 90th percentile, I’d suggest coaching might be best used elsewhere.
• The real value in standardized tests is for highly selective institutions who need to make fine (even meaningless) distinctions between otherwise very well qualified applicants.  It’s because the tests have a very low rate of false positives.  In other words, if you have a high score, you’re probably reasonably accomplished.  If you have a low score, it may, but doesn’t necessarily, mean you’re not.  When your admission slots are fixed and small relative to the number of applications you receive, you’re not going to take many chances.  You’ve already got enough students with high scores angry at you for rejecting them.
• There are other characteristics, including those not measured on a high school transcript, and certainly not by a four-hour test, that determine whether or not someone is going to be successful in college.  In some sense, these things are almost as important as your grades.

So, I figured that we could make a policy decision–a reasoned, data-driven  policy decision–based on research and experience, and other people, including those who might not agree with it, would go along for the ride.  After all, we knew that most students who applied to DePaul would still submit an ACT or SAT score.  Lots would apply who thought “test-optional” was the same as “ability-optional.” (I need to thank my friend Chris Lydon at Providence College, another successful test-optional institution, for giving me that line.)  They’d be denied admission, as they always have been, with or without test scores. But in the end, we’d admit many good, accomplished students who just happened to have test scores lower than they thought they should.

I was wrong. Seven months after the story broke, the Sun-Times in Chicago picked it up.  Seemed odd, but when the story ran, the media jumped all over it, as if they had missed it earlier. I thought it unfortunate that a 45 minute talk with a reporter was condensed to a couple of sentences, one of which was taken totally out of context.   I thought the article missed the point of what we were doing and why we were doing it; I know this stuff inside out, and I couldn’t do it justice in a couple paragraphs (Q.E.D.) But it’s happened before.  Not a big deal.

Pencilnerd  saw it and blasted us, too.  But hey, when you make your living telling people that you can improve their scores and their chances for success in life, I guess you are expected to do so.  We’ve disagreed, and we’ve done so civilly.  I respect his right to have a differing opinion.  And it seems he respects mine.

However, Esther Cepeda used my words (I never spoke to her) and wrote a piece that is wrong on almost every level, and filled bad information and conclusions she’s jumped to based on God-knows-what.  And on top of it all, the headline was offensive and inflammatory, probably on purpose.  She seemed to be proud of her poorly formed opinions. I got angry.  I wrote another post but some thought it a little too blunt.  I took it down, and now I’m writing this.  I hope it makes sense and puts this topic into perspective.

For now, suffice it to say that Ms. Cepeda’s recent article in the Sun Times was mean-spirited, wholly uninformed, and devoid of rational thinking. Other than that, it was fine.

And a Final Note: I”ve taken a different approach on my response.  It’s here.  I hope you enjoy it.