Evidence Based Medicine
Why we can’t trust clinical guidelines
Cite this as: BMJ 2013;346:f3830
This article has a correction
Please see: Why we can’t trust clinical guidelines
On 13 April 1990, in an unprecedented action, the US National Institutes of Health faxed a letter to every physician in the US on how to correctly prescribe a breakthrough treatment for acute spinal cord injury. Many neurosurgeons were sceptical of the evidence that lay behind the new recommendation to give high dose steroids, yet when two respected organisations released a review and a guideline recommending the treatment, they felt obliged to give it. Now, over two decades later, new guidelines warn against the serious harms of high dose steroids. This case and others like it point to the ethical difficulties that doctors face when biased guidelines are promoted and raise the question: why do processes intended to prevent or reduce bias fail?
Doctors who are sceptical about the scientific basis of clinical guidelines have two choices: they can follow guidelines even though they suspect doing so will cause harm, or they can ignore them and do what they believe is right for their patients, thereby risking professional censure and possibly jeopardising their careers.1 2 34 This is no mere theoretical dilemma; there is evidence that even when doctors believe a guideline is likely to be harmful and compromised by bias, a substantial number follow it.5
Disturbing precedent
In the early 1990s, high dose steroids became the standard of care for acute spinal cord injury,6 reinforced by a Cochrane review. The Cochrane Collaboration, is widely known to have strict standards concerning conflicts of interest, yet in this case the collaboration permitted Michael Bracken, who declared he was an occasional consultant to steroid manufacturers Pharmacia and Upjohn, to serve as the sole reviewer.7 He was also the lead researcher on the single landmark study, published in the New England Journal of Medicine,8 used to support the Cochrane review.
Neurosurgeons were not convinced. Many expressed concern about high rates of infection, prolonged hospital stays, and death with high dose steroids.9 10 One expert estimated that more patients had been killed by the treatment in the past decade than died in the 9/11 World Trade Center attacks.5
A poll of over 1000 neurosurgeons showed that only 11% believed the treatment was safe and effective. Only 6% thought it should be a standard of care. Yet when asked if they would continue prescribing the treatment, 60% said that they would. Many cited a fear of malpractice if they failed to follow “a standard of care.”5
That standard was reversed this March, when the Congress of Neurological Surgeons issued new guidelines. The congress found that, “There is no Class I or Class II medicine evidence supporting the benefit of [steroids] in the treatment of acute [spinal cord injury]. However, Class I, II, and III evidence exists that high-dose steroids are associated with harmful side effects including death.”11
Manufacturing consensus
Guidelines are usually issued by large panels of authors representing specialty and other professional organisations. While it might seem difficult to bias a guideline with so many experts participating under the sponsorship of large professional bodies, a worrying number of cases suggests that it may be common.
A recent survey found that 71% of chairs of clinical policy committees and 90.5% of co-chairs had financial conflicts.12 Such conflicts can have a strong impact: FDA advisers reviewing the safety record of the progestogen drospirenone voted that the drug’s benefits outweighed any risks. However, a substantial number of the advisers had ties to the manufacturer and if their votes had been excluded the decision would have been reversed.13
Biased guidelines can have powerful and wide ranging effects. Thousands of guidelines have been issued,14 and, when promulgated by highly respected professional societies, they sometimes serve as de facto “standards of care” that may be used to devise institutional protocols, to develop measures of physician performance, and for insurance coverage decisions. Guidelines may influence the medicines selected for inclusion on drug formularies and may be used as a “reliable authority” to support expert testimony in malpractice suits.4 Eighty four per cent of doctors say they are concerned about industry influence over clinical guidelines,12 yet the fear of malpractice suits puts many in an untenable position of following guidelines they believe are flawed or dangerous to patients.
Exuberant claims for alteplase in stroke
A similar scenario may be playing out for the use of the thrombolytic drug alteplase in acute stroke. Earlier this year, the American College of Emergency Physicians with the American Academy of Neurology (jointly)18 and the American Heart Association,19 separately, issued grade A level of evidence guidelines for alteplase in acute stroke. The simultaneous recommendation by three respected professional societies would seem to indicate overwhelming support for the treatment and consistent evidence. However, an online poll of 548 emergency physicians showed that only 16% support the new guidelines.20 Although the poll was not scientific, other surveys show substantial scepticism among emergency physicians and the treatment remains contentious.21 22 23 24 25 26 27
Guideline authors say that opposition to the guidelines is insubstantial. Andy Jagoda, a member of the guideline committee and professor and chair of emergency medicine at Mount Sinai School of Medicine, said that “almost all” resident physicians “believe in tPA [alteplase] for stroke.” Another guideline author, Steven R Messe, assistant professor of neurology at the Hospital of the University of Pennsylvania and the Pennsylvania Hospital, told the BMJ that “only a small, vocal minority [of emergency physicians] are opposed.”
An earlier survey claimed that emergency physicians don’t oppose alteplase for stroke. At a glance, the claim seemed justified: the poll found that 83% of the doctors surveyed said they would give the treatment.21 However, when asked whether “the science supports the use of tPA [alteplase],” only 49% agreed.
Alteplase was approved for acute stroke after the 1995 National Institutes of Neurologic Diseases and Stroke (NINDS) trial showed a 13% absolute reduction in disability.28 Advocates quickly began to promote the treatment with exuberant claims. The American Heart Association said it could “save lives,” a claim the organisation was forced to withdraw in 2002 when it was pointed out that no study had shown a mortality benefit.29 In 2007, leading stroke experts with industry ties repeated the “saves lives” claim in the New York Times, suggesting that far too few stroke patients were receiving the drug, largely because of resistance among emergency physicians. The newspaper later published a brief correction stating there was no evidence to support the claim that the drug saved lives.30
But as with steroids for acute spinal cord injury, claims of benefit rest on science that is contested. Sceptics say that baseline imbalances, the use of subset analyses, and chance alone could account for the claimed benefit.24 26 31 32 33 They also note that only two of 12 randomised controlled trials of thrombolytics have shown benefit and five had to be terminated early because of lack of benefit, higher mortality, and significant increases in brain haemorrhage.33
In addition, the guideline committee did not include the largest study of the treatment to date in its analysis. Messe, who was one of the guideline’s authors and a spokesperson until April 2011 for Boehringer Ingelheim, Genentech’s European marketing partner, told the BMJ that the joint panel did not include the International Stroke Treatment-3 (IST-3) Trial because the outcome “showed a benefit” among subgroups and because the patients treated were not the same population as in the NINDS trial. However, the effect on the primary outcome in IST-3 (treatment of stroke from 0-6 hours) was actually negative, and the claimed benefits were based on secondary, exploratory analyses. When this was pointed out, Messe acknowledged that the primary outcome was negative and said, “No one has claimed, nor do we recommend, treatment up to 6 hours.”
The new grade A recommendation by the American College of Emergency Physicians is seen as particularly surprising because emergency physicians have been the strongest critics of the treatment. In a survey of 1105 emergency doctors, 40% said they were “not likely to use” alteplase for acute stroke even under the ideal conditions recommended by the NINDS protocol.34 Two thirds of those doctors cited the risk of symptomatic intracerebral haemorrhage as the factor that most concerned them. A quarter cited the lack of clear treatment benefit.34 Their concerns seem understandable in light of a Cochrane review of pooled effects that showed alteplase increased fatal intracerebral haemorrhage nearly fourfold, and that thrombolytics overall were associated with a significant increase in mortality by the end of follow-up, representing an extra 30 deaths per 1000 treated patients.35
Curt Furberg, a prominent methodologist and former Food and Drug Administration adviser, told the BMJ: “The most powerful evidence comes from the Cochrane pooled analysis.” Furberg objected to the use of subgroup analyses to prove benefit, saying, “When clinical trial results are heterogeneous, it’s important to look at the totality of evidence. You should never draw firm conclusions from post hoc analyses. You can’t just select data that supports the thesis you like by asking, ‘How do the results look at 2 hours? How about 2 hours and 10 minutes? How about 3 hours?’ By chance alone you will find something that supports your bias.”
Best guidelines influence can buy: how it happens
Proponents of alteplase have launched projects to ensure uptake of the guidelines in the US, such as the development of “stroke certified hospitals,” which require hospitals to commit resources to enable rapid administration of alteplase to eligible stroke patients. Since ambulances divert patients with suggestive symptoms to stroke certified hospitals, the project has substantial financial ramifications. These efforts, and others like the “Brain Attack” campaign, have been actively supported by the American Heart Association and American Stroke Association, which “partnered” with the Joint Commission (a quasi-governmental agency that accredits hospitals) to promote hospital stroke certification. Genentech, Boehringer Ingelheim and Novo Nordisk, which market alteplase, have contributed tens of millions of dollars to the associations.
In its newly released guidelines, the American Heart Association states that it “makes every effort to avoid any actual or potential conflicts of interest that may arise as a result of . . . a business interest of a member of the writing panel.” However, according to their conflict of interest disclosure statements, 13 of the 15 authors had ties to the manufacturers of products to diagnose and treat acute stroke; 11 had ties to companies that market alteplase.19 In 2010, two years after the association launched this guideline panel, it revised its financial conflicts policy; in the future, neither committee chairs nor the majority of its guideline writing members may have any relevant ties to industry.
Concern about the credibility of guidelines led the Institute of Medicine to recommend that ideally no guideline authors should have financial conflicts of interest.14 If individuals who have professional conflicts that can’t be divested (for example, specialists whose career depends on treating a certain condition) are included, the institute recommends that they “should represent not more than a minority” of the panellists.14
In the guidelines issued jointly by the American College of Emergency Physicians and the American Academy of Neurology, three of eight panellists disclosed ties to the manufacturers. However, seven had either direct ties to the manufacturer or indirect ties, knowingly or not, through affiliations with the Foundation for Education and Research in Neurological Emergencies (FERNE), which provides unrestricted continuing medical education grants (table⇓). Guideline readers were unlikely to know that according to its 2008 tax return, 100% of the $97 000 donated to the foundation that year came from drug companies, including $50 000 from Genentech. The foundation president and founder, Edward P Sloan, is an outspoken advocate of alteplase for stroke.36
View this table:
Competing interests of authors of American College of Emergency Physicians and the American Academy of Neurology guidelines on alteplase
For all guidelines, the overwhelming majority of committee chairs and cochairs have ties to industry,12 and selection of panellists with desired viewpoints can make a wished for outcome a foregone conclusion. Committee stacking may be one of the most powerful and important tools to achieve a desired outcome. Seven of the eight panellists had previously published or lectured on the merits of alteplase for stroke. The eighth panellist, Robert Wears, described himself as an “agnostic” but added that he was “surprised” that he was named as an author since he had resigned from the committee six years earlier. Not one sceptic was included on the panel. In response to a question about whether any known sceptics were invited to be on the committee, a spokesperson for the American Academy of Neurology said, “A potential panel member’s opinion on a topic does not determine eligibility for participation on an American Academy of Neurology guideline author panel. The guideline development process is evidence based.”
Wears, a highly respected methodologist and professor of emergency medicine at the University of Florida Health Sciences Center, had been the methodologist for the committee. He told the BMJ that he resigned in part because he was growing increasingly “disillusioned” with the guideline process. When asked why Wears’ name appeared as one of the committee members, Rhonda Whitson, clinical practice manager for the college told the BMJ, “He may have thought his role on the tPA panel ended sooner than it did . . . However, he did participate throughout the project as needed for his role.”
A spokesperson for the Annals of Emergency Medicine, which published the clinical policy, explained how Wears’ name was able to appear in the journal. She told theBMJ that it does not peer review the college’s clinical policies; nor does it vet the authors or members of the development panel.
Widespread problem
Many other conflicted guidelines have come to light in recent years. In 2006, the New England Journal of Medicine published an article warning against aggressive treatment of anaemia with erythropoietin in patients with kidney disease. Patients treated aggressively had increased rates of heart failure and need for dialysis.37Yet guidelines issued in 2007 by the National Kidney Foundation, which received multimillion dollar donations from companies that make erythropoietin, recommended aggressive treatment that would increase the number of patients receiving the drug.38
In 2004, newly issued cholesterol guidelines greatly expanded the number of people for whom treatment is recommended. A firestorm broke out when it was learnt that all but one of the guideline authors had ties to the manufacturers of cholesterol lowering drugs.39
Yet these and other guidelines continue to be followed despite concerns about bias, because as one lecturer told a meeting on geriatric care in the Virgin Islands earlier this year, “We like to stick within the standard of care, because when the shit hits the fan we all want to be able to say we were just doing what everyone else is doing—even if what everyone else is doing isn’t very good.”
Notes
Cite this as: BMJ 2013;346:f3830
Footnotes
- Competing interests: I have read and understood the BMJ Group policy on declaration of interests and have no relevant interests to declare.
- Provenance and peer review: Commissioned; not externally peer reviewed.
References
- ↵Cundiff D. The story behind a whistleblower doctor license reinstatement hearing. KevinMD.com, 6 January 2010. www.kevinmd.com/blog/2010/01/story-whistleblower-doctor-license-reinstatement-hearing.html.
- ↵Garber AM. Evidence-based guidelines as a foundation for performance incentives. Health Aff (Millwood) 2005;24:174-9.
- ↵Timmermans S, Mauck A. The promises and pitfalls of evidence-based medicine. Health Aff (Millwood)2005;24:18-28.
- ↵Molloy S, Middleton F, Casey AT. Failure to administer methylprednisolone for acute traumatic spinal cord injury—a prospective audit of 100 patients from a regional spinal injuries unit. Injury2002;33:575-8.
- ↵Lenzer J. NIH secrets: study break. New Republic2006 Oct 6. www.newrepublic.com/article/national-institutes-health-fred-geisler-spinal-cord.
- ↵Canadian Association of Emergency Physicians. Steroids in acute spinal cord injury. CJEM2003;5:7-9.
- ↵Bracken MB. Steroids for acute spinal cord injury. Cochrane Database Syst Rev2009;1:CD001046.
- ↵
- ↵Nesathurai S. Steroids and spinal cord injury: revisiting the NASCIS 2 and NASCIS 3 trials. J Trauma1998;45:1088-93.
- ↵Coleman WP, Benzel D, Cahill DW, Ducker T, Geisler F, Green B, et al. A critical appraisal of the reporting of the National Acute Spinal Cord Injury Studies (II and III) of methylprednisolone in acute spinal cord injury. J Spinal Disord2000;13:185-99.
- ↵Hurlbert RJ, Hadley MN, Walters BC, Aarabi B, Dhall SS, Gelb DE, et al. Pharmacological therapy for acute spinal cord injury. Neurosurgery2013;72:93-105.
- ↵Kung J. Failure of clinical practice guidelines to meet institute of medicine standards: two more decades of little, if any, progress. Arch Intern Med2012;172:1628-33.
- ↵Lenzer J, Epstein K. The Yaz men. Washington Monthly2012 Jan 9. www.washingtonmonthly.com/ten-miles-square/2012/01/the_yaz_men_members_of_fda_pan034651.php.
- ↵Institute of Medicine. Clinical practice guidelines we can trust. National Academies Press, 2011.
- ↵Ransohoff DF, Pignone M, Sox HC. How to decide whether a clinical practice guideline is trustworthy. JAMA2013;309:139-40.
- ↵Guyatt G, Akl EA, Hirsh J, Kearon C, Crowther M, Gutterman D, et al. The vexing problem of guidelines and conflict of interest: a potential solution. Ann Intern Med2010;152:738-41.
- ↵Gale EA. Conflicts of interest in guideline panel members. BMJ2011;343:d5728.
- ↵Clinical policy: use of intravenous tPA for the management of acute ischemic stroke in the emergency department. Ann Emerg Med2013;61:225-43.
- ↵Jauch EC, Saver JL, Adams HP Jr, Bruno A, Connors JJ, Demaerschalk BM, et al. Guidelines for the early management of patients with acute ischemic stroke: a guideline for healthcare professionals from the American Heart Association/American Stroke Association. Stroke2013;44:870-947.
- ↵Poll on support for new guidelines on tPA for stroke. SurveyMonkey2013. www.surveymonkey.com/s/CBF2ZDB.
- ↵Scott PA, Xu Z, Meurer WJ, et al. Attitudes and beliefs of Michigan emergency physicians toward tissue plasminogen activator use in stroke: baseline survey results from the Increasing Stroke Treatment through Interactive behavioral Change Tactic (INSTINCT) Trial hospitals. Stroke2010;41:2026-32.
- ↵
- ↵
- ↵Hoffman JR, Schriger DL. A graphic reanalysis of the NINDS Trial. Ann Emerg Med2009;54:329-36.
- ↵Klauer K. History repeating: questioning tPA. Emerg Physicians Monthly, 2012 Nov. www.epmonthly.com/columns/in-my-opinion/history-repeating-questioning-tpa/.
- ↵Newman D. The guideline, the science, and the gap. Scientific medicine and research translation.17 April 2013. http://smartem.org/content/guideline-science-and-gap.
- ↵Newman DH, Shreves AE. Thrombolysis in acute ischaemic stroke. Lancet2012;380:1053-4; author reply 1054-5.
- ↵National Institute of Neurological Disorders. Stroke rt PASSG. Tissue plasminogen activator for acute ischemic stroke. N Engl J Med1995;333:1581-7.
- ↵Lenzer J. Alteplase for stroke: money and optimistic claims buttress the “brain attack” campaign. BMJ2002;324:723-9.
- ↵Kolata G. Lost chances for survival, before and after stroke. New York Times2007 May 28. www.nytimes.com/2007/05/28/health/28stroke.html.
- ↵Radecki RP. Skepticism about thrombolytics in stroke is not unreasonable. Nat Rev Neurol2011;8:176.
- ↵Fatovich DM. Response to the recent best evidence topic on the use of thrombolysis in stroke. Emerg Med J2012;29:82.
- ↵Newman D. Thrombolytics for acute ischemic stroke. TheNNT.com 2012. www.thennt.com/nnt/thrombolytics-for-stroke/.
- ↵Brown DL, Barsan WG, Lisabeth LD, Gallery ME, Morgenstern LB. Survey of emergency physicians about recombinant tissue plasminogen activator for acute ischemic stroke. Ann Emerg Med2005;46:56-60.
- ↵Wardlaw JM, Murray V, Berge E, Del Zoppo GJ. Thrombolysis for acute ischaemic stroke. Cochrane Database Syst Rev2009;4:CD000213.
- ↵Sloane EP. Clinical Use of tPA in acute ischemic stroke. www.uic.edu/com/ferne/pdf/chile1103/sloantpa_acep_chile1103_lecture.pdf.
- ↵
- ↵Armstrong D. Medical journal spikes article on industry ties of kidney group. Wall Street Journal 2006 Dec 26. http://online.wsj.com/article/SB116709354076459273.html.
- ↵Abramson JE, Barnard RJ, Barry HC, Bezruchka S, Brody H, Brown DL, et al. E.petition to the National Institutes of Health seeking and independent review panel to re-evaluate the national cholesterol education project guidelines. 2004. http://cspinet.org/new/pdf/finalnihltr.pdf.
No comments:
Post a Comment