Ben Goldacre: Battling Bad Science


So I’m a doctor, but I kind of slipped
sideways into research, and now I’m an epidemiologist. And nobody really knows
what epidemiology is. Epidemiology is the science
of how we know in the real world if something is good
for you or bad for you. And it’s best understood through example as the science of those crazy,
wacky newspaper headlines. And these are just some of the examples. These are from the Daily Mail. Every country in the world
has a newspaper like this. It has this bizarre,
ongoing philosophical project of dividing all the inanimate
objects in the world into the ones that either cause
or prevent cancer. Here are some of the things
they said cause cancer: divorce, Wi-Fi, toiletries and coffee. Some things they say prevent cancer: crusts, red pepper, licorice and coffee. So you can see there are contradictions. Coffee both causes and prevents cancer. As you start to read on, you can see that maybe there’s some
political valence behind some of this. For women, housework
prevents breast cancer, but for men, shopping
could make you impotent. (Laughter) So we know that we need to start
unpicking the science behind this. And what I hope to show is that unpicking
the evidence behind dodgy claims isn’t a kind of nasty, carping activity; it’s socially useful. But it’s also an extremely valuable
explanatory tool, because real science is about
critically appraising the evidence for somebody else’s position. That’s what happens in academic journals, it’s what happens
at academic conferences — the Q&A session after a postdoc
presents data is often a bloodbath. And nobody minds that;
we actively welcome it. It’s like a consenting
intellectual S&M activity. (Laughter) So what I’m going to show you
is all of the main things, all of the main features of my discipline,
evidence-based medicine. And I will talk you through all of these
and demonstrate how they work, exclusively using examples
of people getting stuff wrong. We’ll start with the absolute weakest
form of evidence known to man, and that is authority. In science, we don’t care how many letters
you have after your name — we want to know what your reasons are
for believing something. How do you know that something
is good for us or bad for us? But we’re also unimpressed by authority
because it’s so easy to contrive. This is somebody called
Dr. Gillian McKeith, PhD, or, to give her full
medical title, Gillian McKeith. (Laughter) Again, every country
has somebody like this. She is our TV diet guru. She has five series
of prime-time television, giving out very lavish
and exotic health advice. She, it turns out, has a non-accredited
correspondence course PhD from somewhere in America. She also boasts that she’s a certified
professional member of the American Association
of Nutritional Consultants, which sounds very glamorous;
you get a certificate. This one belongs to my dead cat, Hettie.
She was a horrible cat. You go to the website, fill out the form, give them $60, it arrives in the post. That’s not the only reason
we think this person is an idiot. She also says things like
eat lots of dark green leaves, they contain chlorophyll
and really oxygenate your blood. And anybody who’s done
school biology remembers that chlorophyll and chloroplasts
only make oxygen in sunlight, and it’s quite dark in your bowels
after you’ve eaten spinach. Next, we need proper science,
proper evidence. So: “Red wine can help
prevent breast cancer.” This is a headline
from The Daily Telegraph in the UK. “A glass of red wine a day could help
prevent breast cancer.” So you find this paper, and find
that it is a real piece of science. It’s a description of the changes
in the behavior of one enzyme when you drip a chemical
extracted from some red grape skin onto some cancer cells in a dish on a bench
in a laboratory somewhere. And that’s a really useful thing
to describe in a scientific paper. But on the question of your own personal
risk of getting breast cancer if you drink red wine, it tells you absolutely bugger all. Actually, it turns out
that your risk of breast cancer increases slightly with every amount
of alcohol you drink. So what we want are studies
in real human people. And here’s another example. This is from Britain’s “leading”
diet nutritionist in the Daily Mirror, our second-biggest selling newspaper. “An Australian study in 2001
found that olive oil, in combination with fruits,
vegetables and pulses, offers measurable protection
against skin wrinklings,” and give the advice: “If you eat olive oil and vegetables,
you’ll have fewer wrinkles.” They helpfully tell you
how to find the paper, and what you find
is an observational study. Obviously, nobody has been able
to go back to 1930, get all the people born
in one maternity unit, and half of them eat lots
of fruit and veg and olive oil, half of them eat McDonald’s, and then we see how many wrinkles
you’ve got later. You have to take a snapshot
of how people are now. And what you find is, of course: people who eat veg and olive oil
have fewer wrinkles. But that’s because people who eat
fruit and veg and olive oil are freaks — they’re not normal, they’re like you;
they come to events like this. (Laughter) They’re posh, they’re wealthy,
less likely to have outdoor jobs, less likely to do manual labor, they have better social support,
are less likely to smoke; for a host of fascinating, interlocking social, political and cultural reasons, they’re less likely to have wrinkles. That doesn’t mean
it’s the vegetables or olive oil. (Laughter) So ideally, what you want
to do is a trial. People think they’re familiar
with the idea of a trial. Trials are old; the first one
was in the Bible, Daniel 1:12. It’s straightforward: take a bunch
of people, split them in half, treat one group one way,
the other group, the other way. A while later, you see
what happened to each of them. I’m going to tell you about one trial, which is probably
the most well-reported trial in the UK news media over the past decade. This is the trial of fish oil pills. The claim: fish oil pills improve
school performance and behavior in mainstream children. They said, “We did a trial. All the previous ones were positive,
this one will be too.” That should ring alarm bells: if you know the answer to your trial,
you shouldn’t be doing one. Either you’ve rigged it by design, or you’ve got enough data so there’s
no need to randomize people anymore. So this is what they were going
to do in their trial: They were taking 3,000 children, they were going to give them these huge
fish oil pills, six of them a day, and then, a year later, measure
their school exam performance and compare their performance against what they predicted
their exam performance would have been if they hadn’t had the pills. Now, can anybody spot
a flaw in this design? (Laughter) And no professors
of clinical trial methodology are allowed to answer this question. So there’s no control group. But that sounds really techie, right?
That’s a technical term. The kids got the pills,
and their performance improved. What else could it possibly
be if it wasn’t the pills? They got older; we all develop over time. And of course, there’s the placebo effect, one of the most fascinating things
in the whole of medicine. It’s not just taking a pill
and performance or pain improving; it’s about our beliefs and expectations,
the cultural meaning of a treatment. And this has been demonstrated
in a whole raft of fascinating studies comparing one kind of placebo
against another. So we know, for example, that two sugar pills a day
are a more effective treatment for gastric ulcers than one sugar pill. Two sugar pills a day beats one a day. That’s an outrageous
and ridiculous finding, but it’s true. We know from three different studies
on three different types of pain that a saltwater injection
is a more effective treatment than a sugar pill, a dummy pill
with no medicine in it, not because the injection or pills
do anything physically to the body, but because an injection feels
like a much more dramatic intervention. So we know that our beliefs
and expectations can be manipulated, which is why we do trials
where we control against a placebo, where one half of the people
get the real treatment, and the other half get placebo. But that’s not enough. What I’ve just shown you are examples of the very simple
and straightforward ways that journalists and food supplement
pill peddlers and naturopaths can distort evidence
for their own purposes. What I find really fascinating is that the pharmaceutical industry
uses exactly the same kinds of tricks and devices, but slightly more sophisticated
versions of them, in order to distort the evidence
they give to doctors and patients, and which we use to make
vitally important decisions. So firstly, trials against placebo: everybody thinks a trial
should be a comparison of your new drug against placebo. But in a lot of situations that’s wrong; often, we already have a good treatment
currently available. So we don’t want to know
that your alternative new treatment is better than nothing, but that it’s better than the best
available treatment we have. And yet, repeatedly, you consistently
see people doing trials still against placebo. And you can get licensed
to bring your drug to market with only data showing
that it’s better than nothing, which is useless for a doctor like me
trying to make a decision. But that’s not the only way
you can rig your data. You can also rig your data by making the thing you compare
your new drug against really rubbish. You can give the competing drug
in too low a dose, so people aren’t properly treated. You can give the competing drug
in too high a dose, so people get side effects. And this is exactly what happened with antipsychotic medication
for schizophrenia. Twenty years ago, a new generation
of antipsychotic drugs were brought in; the promise was they would have
fewer side effects. So people set about doing trials
of the new drugs against the old drugs. But they gave the old drugs
in ridiculously high doses: 20 milligrams a day of haloperidol. And it’s a foregone conclusion
if you give a drug at that high a dose, it will have more side effects,
and your new drug will look better. Ten years ago, history repeated itself, when risperidone, the first
of the new-generation antipsychotic drugs, came off copyright,
so anybody could make copies. Everybody wanted to show their drug
was better than risperidone, so you see trials comparing
new antipsychotic drugs against risperidone
at eight milligrams a day. Again, not an insane dose,
not an illegal dose, but very much at the high end of normal. So you’re bound to make
your new drug look better. And so it’s no surprise that overall, industry-funded trials
are four times more likely to give a positive result than independently sponsored trials. But — and it’s a big but — (Laughter) it turns out, when you look at the methods
used by industry-funded trials, that they’re actually better
than independently sponsored trials. And yet, they always manage
to get the result that they want. So how does this work? (Laughter) How can we explain
this strange phenomenon? Well, it turns out that what happens is the negative data
goes missing in action; it’s withheld from doctors and patients. And this is the most important
aspect of the whole story. It’s at the top
of the pyramid of evidence. We need to have all of the data
on a particular treatment to know whether or not
it really is effective. There are two different ways you can spot
whether some data has gone missing. You can use statistics
or you can use stories. I prefer statistics,
so that’s what I’ll do first. This is a funnel plot. A funnel plot is a very clever
way of spotting if small negative trials have disappeared,
have gone missing in action. This is a graph of all of the trials done
on a particular treatment. As you go up towards the top of the graph, what you see is each dot is a trial. As you go up, those are bigger trials,
so they’ve got less error; they’re less likely to be randomly false
positives or negatives. So they all cluster together. The big trials are closer
to the true answer. Then as you go further down at the bottom, what you can see is, on this side,
spurious false negatives, and over on this side,
spurious false positives. If there is publication bias, if small negative trials
have gone missing in action, you can see it on one of these graphs. So you see here
that the small negative trials that should be on the bottom left
have disappeared. This is a graph demonstrating
the presence of publication bias in studies of publication bias. And I think that’s the funniest
epidemiology joke you will ever hear. (Laughter) That’s how you can prove it statistically. But what about stories? Well, they’re heinous, they really are. This is a drug called reboxetine. This is a drug which I, myself,
have prescribed to patients. And I’m a very nerdy doctor. I hope I go out of my way to try and read and understand
all the literature. I read the trials on this. They were all positive,
all well-conducted. I found no flaw. Unfortunately, it turned out,
that many of these trials were withheld. In fact, 76 percent of all of the trials
that were done on this drug were withheld from doctors and patients. Now if you think about it, if I tossed a coin a hundred times, and I’m allowed to withhold from you
the answers half the times, then I can convince you
that I have a coin with two heads. If we remove half of the data, we can never know what the true
effect size of these medicines is. And this is not an isolated story. Around half of all of the trial data
on antidepressants has been withheld, but it goes way beyond that. The Nordic Cochrane Group were trying
to get ahold of the data on that to bring it all together. The Cochrane Groups are an international
nonprofit collaboration that produce systematic reviews of all of the data
that has ever been shown. And they need to have access
to all of the trial data. But the companies withheld
that data from them. So did the European Medicines Agency — for three years. This is a problem that is currently
lacking a solution. And to show how big it goes,
this is a drug called Tamiflu, which governments around the world have spent billions
and billions of dollars on. And they spend that money
on the promise that this is a drug which will reduce the rate
of complications with flu. We already have the data showing it reduces the duration
of your flu by a few hours. But I don’t care about that,
governments don’t care. I’m sorry if you have the flu,
I know it’s horrible, but we’re not going to spend
billions of dollars trying to reduce the duration
of your flu symptoms by half a day. We prescribe these drugs. We stockpile them for emergencies on the understanding they’ll reduce
the number of complications, which means pneumonia and death. The infectious diseases Cochrane Group,
which are based in Italy, has been trying to get
the full data in a usable form out of the drug companies, so they can make a full decision about whether this drug
is effective or not, and they’ve not been able
to get that information. This is undoubtedly
the single biggest ethical problem facing medicine today. We cannot make decisions
in the absence of all of the information. So it’s a little bit difficult from there to spin in some kind
of positive conclusion. But I would say this: I think that sunlight is the best disinfectant. All of these things
are happening in plain sight, and they’re all protected
by a force field of tediousness. And I think, with all
of the problems in science, one of the best things that we can do is to lift up the lid,
finger around at the mechanics and peer in. Thank you very much. (Applause)

Leave a Reply