Is Evidence-Based Medicine Only an Illusion?

By Kristina Fiore, Staff Writer, MedPage Today
Published: February 23, 2013

In a system where half of all clinical trials never see the light of publication, doctors are merely "imagining that we're practicing evidence-based medicine," says Ben Goldacre, MBBS, a British physician and science journalist.

Goldacre is among the most vocal critics of drugmakers who refuse to hand over complete clinical trial data, making it impossible for doctors and patients to get the full picture on most of the medicines widely used today. He decries the industry's behavior in his new book, Bad Pharma, which in itself is a review of the evidence on evidence review and the stumbling blocks incurred by researchers who try to dig deeper.

He'd like to see all of the clinical study reports ever completed brought out of "dry storage archive...and everywhere else that people stack their old, crinkly, yellow paperwork" and made publicly available -- ideally on his new website, AllTrials.net, which recently signed on GlaxoSmithKline (GSK) to release every study the company has ever done.

During a promotional tour in the U.S., Goldacre answered questions from MedPage Today, in an abbreviated form on camera and in this edited, in-depth Q&A:

Kristina Fiore: What piqued your interested in the issue of missing data?

Ben Goldacre, MBBS: What's really interesting about the phenomenon of trials going missing in action is that it's not an anecdote; it's not a one-off problem. It's this systemic flaw that's right at the core of evidence-based medicine and it undermines and distorts everything that we do. And it has never been fixed.

That's partly because of industry pressure, but also because it's a cultural blind spot for doctors and academics, and that's really strange. I think future generations will look back at missing trial data the same way we look back at medieval bloodletting. They'll say: 'You spent hundreds of millions of dollars on just one trial that's so carefully designed, you used all of these careful statistical tests to try and detect tiny, tiny differences between one treatment and another, and then after doing all of that, you threw it all out the window by deleting half the trial data, especially the negative stuff.'

It's incomprehensible, it's loony.

Fiore: Do clinicians have the full data on any drug that's on the market today?

Goldacre: It's very hard to know. The best currently available evidence comes from systematic reviews on the problem of missing trial data, which show that about half of all trials conducted and completed for the medicines we use today haven't gone on to be published, and that trials with positive results are twice as likely to be published as trials with negative results.

That's the overall picture. For any one individual study, it's difficult to unpick. Some trials are more likely to go missing than others.

What I find even more amazing is [what happens] when people actively strike out trying to get this information. Tamiflu (oseltamivir phosphate) is the classic example. It's the poster child for missing data. When the Cochrane Collaboration realized that a huge part of clinical trial data hadn't been properly published for Tamiflu, they got in touch with Roche [and] Roche promised in December 2009 that they would share the full clinical study reports, but they still haven't.

Even more peculiar is that Richard Bacon, a member of Parliament in the U.K., asked [Prime Minister] David Cameron if he's aware that half of all clinical trials go missing in action, and that for Tamiflu in particular, half of the results haven't been made properly available. When the prime minister was asked if he would have Roche release full study reports, he dodged the question.

In the U.K. alone, we spend half a billion pounds, that's $800 million U.S. dollars, in 1 year on this one drug. But we don't have all the evidence to make a decision if it's better than aspirin or not, and the prime minister of the country isn't willing to say whether or not he'd ask Roche to hand it over. That's an extraordinary state of affairs. It's bizarre that we've tolerated it for so long.

Fiore: What other observations have you made about the relationships between drugmakers and regulators as you've been reporting this book?

Goldacre: This isn't a book of cheesy conspiracy theories. I don't think that people are bad and evil and wrong and hiding the cure for cancer, or anything like that. What you see are much more gentle distortions, and regulators suffer from a very outmoded form of paternalism. They treat doctors the way that doctors used to treat their patients. There's an idea among regulators that it's okay for them to make decisions behind closed doors and just tell doctors what to do. I think that doesn't work anymore because science isn't about just telling someone the answer. Science is about being clear, open, and transparent about your methods and your results.

If you look at some of the real big problems we've spotted in medicine over the past 10 years, they weren't spotted by regulators. Problems with Vioxx (rofecoxib) weren't spotted by the FDA, they were spotted by an independent cardiologist. Problems with Avandia (rosiglitazone) weren't spotted by regulators, they were spotted by individual academic researchers. Problems with Tamiflu weren't spotted by regulators.

That's not because regulators are stupid. They're smart people, but they're smart people working on difficult problems, and difficult problems benefit from many eyes, which is why regulators should be much more free about sharing all of the information they have with doctors and academics and patients.

Implicit in your question is: Are they too close to industry? And the answer is yes, but not in an outwardly corrupt way. I don't think they're being paid off. There's a revolving door between regulators and industry. In the European Medicines Agency (EMA), we recently had an almost cartoonish example where the head of the EMA told the board he was going to work for industry as a consultant ... with 3 days' notice.

What's amazing is not that he did that, and that he thought that would be okay, but that the board wrote back and said, 'Sure that's fine, no problem.' That kind of thing is really worrying.

Fiore: There have been a lot of efforts to try to get regulators and drugmakers to release all of their data. Why haven't any of those worked?

Goldacre: The things that have been tried to force drug companies and independent academics to share results have failed in two ways. Firstly they've not been enforced, and secondly, they have loopholes by design.

There are two big examples we cite as having fixed the problem. One is the International Committee of Medical Journal Editors' 2005 guidelines that say editors won't publish any trials unless they've been registered in public before they're started. The idea was if that everybody who's got a positive result wants to publish it in a journal, then that's a real carrot that journal editors can use to force people to register their trials.

Unfortunately, we found out this wasn't being enforced. A paper published 5 years after these regulations came into force looked at every trial published in the top 10 journals in the big five fields of medicine. Half of them weren't registered properly, and a quarter of them weren't registered at all. So that didn't work. And anyone who says it did work is just misinformed or is misinforming you.

The second thing that's been tried is the FDA Amendment Act of 2007, which says you have to post the results of your trial within 1 year of completion to ClinicalTrials.gov. Again, everybody ran around saying, 'This is great, the problem of clinical trials going missing in action has been fixed forever.'

But a study in BMJ in 2012 found the rate of compliance with this law was 22%. Four out of five trials ignored this law. And yet no fine has ever been levied by the FDA. Even if one had been levied at $10,000 a day, that's basically a parking ticket to a big drug company making a few billion off one drug.

Crucially, even if these laws had been uniformly enforced, they still would have done nothing to improve the evidence base for medicine today. And that's because ClinicalTrials.gov was only getting access to trials that were completed after 2008. We know that about 80% of all the drugs that are prescribed today came on the market more than 10 years ago. So to improve the evidence base for the medicines we use today, we don't need the results of clinical trials conducted and completed in 2008. That won't fix medicine until 2033.

We need the trials that were conducted and completed 10 or 20 years ago. All of that stuff is available, it's in dry storage archive in the chalk hills of Cheshire and re-commissioned nuclear bunkers, and everywhere else that people stack their old, crinkly, yellow paperwork. We just need to get that stuff out. There's no excuse for withholding it. None of the laws that anyone has ever passed even attempts to achieve that.

Fiore: But you're trying to do that with AllTrials.net.

Goldacre: I believe very strongly that when patients come to realize the extent to which we've failed them by not addressing missing data, they will be, well, disappointed is the most relaxed word I can think of. I think they'll be appalled.

Just getting people to talk about the problem of missing trial data is an important first step, because that gets industry and others who want to defend what they view as their right to mislead doctors and patients to do so in public.

AllTrials.net is a campaign I set up with Sense about Science, a U.K. group, and BMJ and Sir Ian Chalmers, one of the founders of Cochrane Collaboration. We're calling for three things. We want all trials to be registered, including retrospectively, so that we know about all the trials that have been done. We're calling for all results to be reported in brief summary formats. And lastly we're asking for full clinical study reports. What we're not asking for is any patient confidential information.

Since it launched about a month ago, I have been amazed. We are now supported by the great and the good of U.K. medical research groups – the Medical Research Council, the Wellcome Trust. We got support from over 80 patient groups. But also, amazingly, GSK, one of the biggest drug companies in the world, has now signed up. They've been trying to mend their image over the past few years; last year they paid a $3 billion fine for acts of criminal and civil fraud including withholding information on clinical trials. They've now committed to release all trial results and full clinical study reports for all clinical trials they've ever conducted going back to the dawn of the company.

Fiore: And why it will succeed where other efforts have failed?

Goldacre: It's interesting and powerful for two reasons. Firstly, in the same 24 hours that GSK committed to doing that, the Pharmaceutical Research and Manufacturers of America (PhRMA) put out a lurid and colorful denunciation of me and BMJ saying how dangerous and unreasonable it was of us to ask drug companies to release all of the evidence about how well their drugs work in patients. It's an extraordinary position to even try and maintain. They said it would be impossible because it would restrict innovation [and] it would require releasing individual patient confidential information, which I'm not asking for.

So in the same 24-hour period, you have PhRMA saying it's impossible, dreadful, and unacceptable, but you have GSK, one of the biggest companies in the world, saying yeah, we agree, this is important and needs to be fixed.

Secondly -- and I think this is really interesting -- is the paradox that if not everyone's signed up to AllTrials.net, that may help to create a financial incentive for companies to do the right thing.

If there are two treatments made by two different companies and both seem to have equal benefits, but one of them is made by a company that has made a commitment to sharing all clinical trial results while the other company refuses to make the same commitment, which drug would you use?

I don't think GSK will be the only company to sign up. I think we're going to enter a very interesting era where potentially the market is differentiated by ethical companies who've made a commitment to sharing all their trial results, and unethical companies who are still aggressively defending their ability to withhold information from doctors and patients.

http://www.medpagetoday.com/PublicHealthPolicy/ClinicalTrials/37506