What causes what? The human brain is programmed to answer this question constantly. This how we survive. What made that noise? Bear made that noise. What caused my hand to hurt? Fire caused my hand to hurt.
We are so eager to figure what causes what — that we often get it wrong. I wore my lucky hat to the game. My team won. Therefore, my lucky hat caused my team to win.
On today's show we dive deep into the world of correlation and causation with Charles Wheelan, author of the new book, Naked Statistics: Stripping the Dread from the Data.
ROBERT SMITH, HOST:
Ten years ago, Charlie Wheelan was living in Chicago, and he and his wife faced a classic dilemma for parents in cities. They were trying to figure out where to send their daughter to kindergarten.
CHARLES WHEELAN: And then we got wind that there was a Chicago public school in our neighborhood that was doing some interesting things, and that seemed like it would be a great place to send our oldest daughter to school.
JACOB GOLDSTEIN, HOST:
Seems like an easy choice. But Charlie was raised out in the suburbs of Chicago, and to people in the suburbs of Chicago, the city's public school system seemed like a horror, a terrible place to send your kids.
WHEELAN: So my mother - I think the best metaphor I can offer is that she was stricken to the point that she kind of had to lie on a couch with a compress on her head at the prospect of her grandchildren, her beloved grandchildren, spending their formative years in a Chicago public school.
SMITH: Now, Charlie Wheelan was teaching public policy at the time at the University of Chicago, so he had something that a lot of us wish we had when we were having fights with our parents - access to data and a grasp of statistics. And Charlie started to tell his mother that she was falling prey to one of the classic fallacies of our time. Sure, the Chicago public school system might have lower test scores, higher dropout rates than private schools or even schools in the suburbs, but that doesn't mean that the Chicago public school system caused those low test scores or high dropout rates.
WHEELAN: The important thing in figuring out whether a school is good, not good, somewhere in between is how much value is added to the kids who walk in the front door. And in suburban areas, and in public schools and in some charter schools, the kids who walk in the front door have motivated, highly educated, affluent parents. So those kids are going to do very well no matter what happens. And in the Chicago public schools, you had a really tough demographic, so they walked in the front door with a lot of disadvantages.
GOLDSTEIN: This argument Charlie was having with his mom is over this really basic question that we all struggle with all the time, whether we know it or not. That basic question - what causes what?
WHEELAN: And the human brain is really programmed to answer this question constantly. I mean, this is how we have survived to dominate this planet, right? What made that noise? Bear made that noise.
GOLDSTEIN: What caused my hand to hurt? Fire caused my hand to hurt.
WHEELAN: Were so eager to figure out what causes what that we often get it wrong. Ooh, I wore my lucky hat had to the game, and my team won, so I am never taking off this hat.
GOLDSTEIN: Hello, and welcome to PLANET MONEY. I'm Jacob Goldstein.
SMITH: And I'm Robert Smith. Today on the show, what causes what? Or if you want to be fancy about it, correlation and causation.
(SOUNDBITE OF BOOKA SHADE'S "NUMBERS")
GOLDSTEIN: Our guest today, Charlie Wheelan - you may know him as the author of a book called "Naked Economics." We've talked about it before on the show. Charlie has a new book out. It's called "Naked Statistics."
SMITH: And Charlie, did you choose the title "Naked Statistics" so that people wouldn't really notice the statistics part of things?
WHEELAN: We did. In fact, I did a book signing in Hanover, and I saw a woman walking out of the store with "Fifty Shades Of Grey" and "Naked Statistics." And I thought, OK, she may be sorely disappointed with one or the other.
GOLDSTEIN: So the book sort of covers a lot of ground in statistics. We're not going to try and cover it all today. We're really going to focus on this one thing, on this one question - what causes what? And, really, this question is at the core of how everybody makes all kinds of decisions. I mean, if you think about it, every decision a government makes, every decision a parent makes with their kid, almost any decision you can think of, it's trying to cause some outcome. And statistics can be very helpful in trying to figure out what causes what, but as you show in the book, statistics can also totally screw you over.
SMITH: Yeah, we wanted to start with a specific example in your book, Charlie. And it involves a really serious issue, which is that older women, especially after menopause - they have an increased risk of certain diseases. And of course, researchers are fascinated by this, and they wanted to look into it. So tell us what happened.
WHEELAN: Well, there was a huge longitudinal study. It was a study of nurses over many, many years - decades, even. It was found that women, as they approached menopause, who were given hormone replacement therapy, which is estrogen, had lower rates of heart disease and other diseases that are prominent killers of women.
GOLDSTEIN: And, you know, to point out, this wasn't just a random connection. This wasn't, boy, people wearing purple shoes have less cancer. There was at least some biological corroboration or at least a theory that would make sense, which is, look, what happens as you age as a woman, your estrogen levels go down. We're seeing this association between lower estrogen and bad outcomes, so then estrogen therapy, if it shows that it has positive effects on things like heart disease, would appear to be a beneficial strategy.
SMITH: And this was, like, a big, serious study. I mean, this was done by Harvard. People looked at this. It was a large sample size. And doctors saw this, and they said, well, if there is a correlation between reduced risk of disease and estrogen, we're going to start prescribing estrogen.
GOLDSTEIN: Right. It made medical sense. So it got to the point where not only were doctors comfortable prescribing estrogen for thousands and thousands of women, it kind of became what you did when you got older, but there was even a movement to start giving estrogen to men.
SMITH: Now, scientists looked at this and they said, OK, there may be a correlation between taking estrogen, having reduced heart disease, but there could be a lot of other factors here. Maybe the people who are more likely to go their doctor and ask for estrogen are just healthier people. Maybe there are other reasons we don't know about. And so the scientists did something scientists do all the time, and it's called a randomized trial. Now, explain to me what they did.
WHEELAN: They took another large group of women, and they randomly assigned them to one of two groups, just like your seventh grade science project, the treatment group or the control group. And one group got the estrogen; the other group did not.
SMITH: And what they found was really scary.
WHEELAN: Scary and tragic. They found that the opposite appears to be true, which is that the women who got estrogen therapy in this controlled experiment actually had higher rates of heart attack, stroke, breast cancer and other things that were quite damaging. And then when they looked back at the number of women who had been prescribed estrogen, they assumed that thousands of women most likely had these events or died prematurely because of what had been really fundamentally a correlation and not a causal relationship. And in fact, to the extent there was a causal relationship, it went the other direction, that estrogen was actually bad for the health of women.
GOLDSTEIN: And this is obviously a particularly large, particularly tragic case of seeing correlation and mistakingly thinking that there is causation. But this kind of error, it happens all the time. It happens everywhere. I mean, one we talk about a lot on the show is the stock market, or to the extent we talk about the stock market, you know, the daily moves. The stock market is up today or it's down today. Basically, every news story about the stock market, if it's up, they just find some correlation, find some good news and assume that that good news is causing it to go up. On the same day, if the stock market were down, they would find some other news, some bad news and assume that bad news is causing it to go down.
WHEELAN: Yeah. On any given day, thousands of things are happening. So I read a story not long ago in The Wall Street Journal - admittedly, it was a little tongue in cheek, but in the run-up to the Masters, it pointed out that when Tiger Woods was winning tournaments, the Dow Jones Industrial Average consistently did better.
SMITH: Well, then I should be tracking his golf scores and not earnings reports.
WHEELAN: So watch the Masters and other tournaments, you know, with your broker on the line, and of course that is clearly just correlation.
GOLDSTEIN: And I feel like when we make this jump, when we want to go from finding correlations to actually figuring out causes, that's when we sort of get to the edge of statistics, right? It seems like statistics is amazing at figuring out correlations and even, you know, getting rid of extraneous variables and whatever. But as far as I can tell, statistics on its own is not enough to get us to causation. To get to causation, it seems like people who are studying this stuff ultimately need to come up with sort of clever tricks.
SMITH: And, Charlie, in your book, you have a number of experiments that researchers have done to get to this question of what causes what. And one of the ones I loved was about crime and specifically crime in D.C., in the District of Columbia, because there's this huge policy question about cops and crime. If you just hire more cops, will crime go down? And it's really hard to tease out because obviously high-crime cities have an incentive to hire a lot of cops. And so the data's really messy, but you describe a way in which they figured out this problem. They figured it out in the District of Columbia.
WHEELAN: Right. So there were some very clever researchers, and they said, all right, well, we need to find an example where we get a lot of police for reasons unrelated to crime. Now, that you would think is a relatively rare occurrence. Why in the world would you create a lot of police if you didn't have a lot of crime? And they happened upon - and this is what great researchers do - the terrorism alert system. So you know all those colors, orange, red, whatever else they are. By law, because Washington, D.C., is likely to be a terrorism target, when the terror alert level goes to orange, then extra police are put on the Mall and other parts of Washington to protect against terrorists. It has nothing to do with street crime or things like that.
So they could then ask, on orange alert days when there are extra police on the streets for reasons unrelated to street crime, what happens to street crime? Does it go down? And in fact, it does. It turns out that when you have the extra police there for terrorism-related reasons, they're on the streets, they make the streets safer, and things like murder, robbery, assault go down. And that to my mind is a very clever way of establishing a causal relationship between - at least in Washington, D.C. - between more police and less crime.
GOLDSTEIN: As long as we assume that the robbers aren't hiding in their rooms because they're afraid of the elevated terror level.
WHEELAN: It's entirely possible. You know, actually, in that vein, one of the things they considered was is it possible that tourists were less likely to visit Washington or to go out and about on...
SMITH: So there's less victims on the street.
WHEELAN: So there are fewer victims, and you know what they did? They checked that hypothesis by looking at ridership levels on the Metro system, and they actually were not diminished on high-terror days, so they suggested the number of victims was largely unchanged.
GOLDSTEIN: So even in this very clever universe of finding this very particular time, there are ways you could trip up. I mean, the fewer tourists hypothesis seems plausible, and they tested it, and no, there are the same number of tourists.
WHEELAN: Yes. So, you know, even then, it's tough to establish causality, but you're building a case. It's almost like circumstantial evidence in a courtroom where you're getting closer to the point where you can be persuaded that, yes, in fact this person did that crime or A causes B or that police reduce crime.
SMITH: So, Charlie, how old are your kids now?
WHEELAN: They're now 15, 12 and 10.
SMITH: So pretty soon you're going to be facing a question that a lot of people face, which is colleges and where to send your kids to college. And when we were talking about clever ways to tease this out, you talked in your book about basically schools like Harvard and Yale, these big-name schools. Is it worthwhile to send your kid there? Will it make their life? Will it cause great success if you send them to those schools rather than other schools? And this was another one of those problematic issues like cops and crime.
WHEELAN: Right. And you can imagine why this is so hard to study. People are not randomly assigned to Harvard versus a local community college. The kids who get into Harvard are highly talented. That's why they got into Harvard. So when you compare the wages or life outcomes of people who went to Harvard versus those who went to lesser schools, it's - that is the classic apples to oranges comparison. There's really no legitimacy to that comparison whatsoever.
On the other hand, it's a very difficult thing to study. Neither Harvard nor most families would agree to participate in an experiment where you had a 50 percent chance of being sent to Harvard and a 50 percent chance of being sent to a community college or not being able to go to college at all. So it's hard to tease out the value added at these highly selective schools.
GOLDSTEIN: But you describe in your book a way that some researchers managed to actually if not do that at least come close.
WHEELAN: Right, with a very clever experiment, and this was done by two researchers, Stacy Dale and Alan Krueger. They found a large group of students who had been accepted to a group of schools that they described as the highly selective schools, so it's the Ivy League and a whole bunch of others you would recognize. And those same students had also been selected or admitted to a group of schools that were less traditionally competitive, less highly selective.
So they were good enough to get into Harvard. Some of them subsequently went to the highly selective schools and some of them didn't. And what that sets up is a nice comparison group where everybody in the sample is smart enough to get into what we describe usually as the very best schools. Some went to those schools, some didn't, so we can then compare their life outcomes 10 or 15 years later.
WHEELAN: It turns out that there doesn't appear to be any great advantage - in fact, I think no statistically significant advantage at all - in terms of wages for most students going to the highly selective schools, with one notable exception, which is, for minority students, there does appear to be an advantage. But for the vast majority of folks in the sample, there was no real huge advantage to getting into and going to one of these highly selective schools.
SMITH: Well, so you have three kids. You are a professor of public policy. You know statistics. You wrote an entire book on statistics. Are you now the dad that's saying you don't need to go to Harvard? You don't need to go to Yale or University of Chicago.
WHEELAN: Well, given that I taught at the University of Chicago, I now teach at Dartmouth, I went to Princeton, I went to, you know, I'm...
SMITH: Come on.
WHEELAN: ...Hardly - yeah, I know, I know. I think that...
SMITH: You're still pressuring them to go to an Ivy League school.
WHEELAN: I'm not going to pressure them at all. I think that's the takeaway is that I do think there are a lot of benefits in terms of network effect and things like that that are a positive outcome related to going to those schools. But I'm certainly not going to put pressure on them. And I'm certainly not going to cling to the delusion that the thing that's going to shape the trajectory of their lives most is whether they get in or don't get in because really - and this goes all the way back to the way I felt about the Chicago Public Schools - it's going to be the talent that they bring to the front door that matters most.
GOLDSTEIN: So what wound up happening with your kids going to Chicago Public Schools? And how did your mother end up feeling about it?
WHEELAN: Well, there's a nice little interlude here in that we got an objective measure of how they were doing in the Chicago Public Schools because after they'd been in school for six years - so my oldest started in the Chicago Public Schools in kindergarten. I had a sabbatical, and we moved to Hanover, N.H. And Hanover, N.H., is a privileged little oasis that spends an ungodly amount on its public schools. They would qualify as very good schools by any superficial measure. So these are the good schools - certainly in my mother's eye, this is exactly where they should have been in the first place.
So for one year, all three kids in first grade, fourth grade and sixth grade transferred to the Hanover Public Schools so we can now see whether they're illiterate and have all kinds of other problems. And in fact, they did extraordinarily well in the Hanover schools, which suggested that they were getting a very fine education in the Chicago Public Schools and that the value added there was excellent.
GOLDSTEIN: Did you convince your mother about the Chicago Public Schools?
WHEELAN: You know, to her - you know, first of all, she was always a little worried about safety. That was not really an issue either. But it wasn't just about the academics. I would say in her defense that she really did come around and in part because of our experience and because, you know, it was a genuine concern on her part - you have to grant her that - that when she realized the kids were turning out OK, she has actually shifted her opinion. And she is a little more skeptical of people who pay extraordinary amounts or devote great effort to try and get into a private school when the kid's 3 years old in the thinking that this is going to radically change that child's life. So I do think - you know, I'm not sure that she'd be eager to have Bee (ph) send - she still didn't want my daughter going to one of the highly selective high schools in Chicago, but she did come around to some extent, so I'll give her credit for that.
SMITH: That seems like a perfect place to wrap this up. Charlie Wheelan, thank you so much for walking us through the world of statistics.
WHEELAN: Wow. It was good to be back with you. Thank you.
SMITH: I should say that we have established that there is a high correlation between appearing on PLANET MONEY and professional success.
WHEELAN: (Laughter) Exactly, right. And I'm glad to be a part of that.
SMITH: It's all causality.
WHEELAN: Thank you for having me.
SMITH: It's all causality, Charlie.
(SOUNDBITE OF MUSIC)
SMITH: As always, we'd love to hear what you think of the show. Give us a shout - firstname.lastname@example.org.
GOLDSTEIN: You can also visit us on the blog at npr.org/money where we will also post links to Charlie's book and some of the articles that we talked about on the show today. I'm Jacob Goldstein.
SMITH: And I'm Robert Smith. Thanks for listening. Transcript provided by NPR, Copyright NPR.