Negative studies in a positive light

Archives of Internal Medicine, Editorial, Volume 169(11), 8 June 2009, p 1022–1023

Editorial Policies and Publication Bias: The Importance of Negative Studies

Author(s):Sridharan, Lakshmi BA; Greenland, Philip MD

Correspondence: Dr Greenland, Departments of Preventive Medicine and Medicine, Northwestern University, Feinberg School of Medicine, 750 N Lake Shore Dr, 11th Floor, Chicago, IL 60611 (p-greenland@northwestern.edu).

Financial Disclosure: None reported.

Additional Information: Dr Greenland was the previous Editor of the Archives.

Publication bias is the tendency for certain kinds of studies, typically those showing a significant positive result in a clinical trial or an observational study, to receive more favorable publication decisions than equally well-conducted studies that report a negative or null result.1, 2, 3, 4 Authors, editors, and peer reviewers all participate in this favoritism toward publication of positive results. An obvious outcome of the bias toward overpublication of positive results is that many treatments or exposures are overrated in the published literature. Some critics have gone so far as to claim that publication bias results in “most published research findings” being “false.”5 Although most researchers, reviewers, and editors would probably believe that such a claim is far too harsh, an unquestioned result of the overwhelming bias to publish mostly positive studies is that subsequent meta-analyses are distorted and result in promoting existing scientific biases. The Cochrane Collaboration admits the existence of this bias in the systematic reviews it publishes and suggests attenuating strategies such as probability models and funnel plot techniques.6

See also page 1028

In this issue of the Archives, the editors selected for publication a “negative trial” in which Etter et al 7 found that starting smoking cessation treatment with nicotine gum 4 weeks prior to the cigarette smoking quit date did not improve long-term smoking abstinence compared with treatment beginning in the standard way on the actual quit date. The editors decided to accept this particular negative trial because the publication of well-conducted negative or neutral studies is important to overcome the problems caused by publication bias. Publication of this particular trial also prompted us to comment herein on our recent efforts to understand and try to reduce publication bias within the editorial policies of the Archives.

The overall industry of medical research has begun to address the issue of publication bias in various ways. For example, clinical trials registration has been a notable recent step in combating underreporting of negative trials.8 However, this measure alone cannot effectively address the issue of bias based on funding of trials that are virtually programmed for positive results owing to the use of placebo-comparison groups,8 nor can trial registration assure that reports of negative trials are ultimately sent by the authors for journal submission, or even if submitted that they are eventually accepted by high-quality journals following peer review. As an example, a recent analysis 1 shows that positive clinical trials registered with the US Food and Drug Administration are 5 times more likely to get published than their negative counterparts. It was unclear whether this distortion resulted from authors or sponsors shelving negative trials or from editors and reviewers rejecting them after submission.

Even a stalwart of medical publishing—the peer-review process—has been the subject of much attention, with recent exposure of abuses among peer reviewers.9 Peer review relies on reviewers to report their biases and try to avoid them, but the tendency to favor positive over negative studies is a bias that is almost never explicitly acknowledged by reviewers. In addition, most often, peer review is single-blinded with reviewers unknown to authors, although many have argued for open review to mitigate potentially biased reviewers.10 Whether open reviews would reduce publication bias against negative studies has not been evaluated, but the idea seems both feasible and might reduce publication bias. In a prior editorial,11 the Archives' editors expressed strong support of a rigorous peer-review process. In part, that support for peer review focused on the importance of self-regulation in professionalism, particularly in medicine, where the professionals operate under an oath of ethical conduct to do no harm.11, 12 Part of doing no harm would involve the attempt to reduce one's biases against negative studies so that the totality of the medical literature would be more true to the actual results from all studies that were properly performed.

With this guiding principle of reducing publication bias, the Archives editors recently explored mechanisms that might identify and reduce biases at this journal. A journal's editorial board does not have direct control over registered clinical trials or which studies an author submits for publication. However, it does have control over its own editorial and peer-review processes. First, to understand the tendency of authors to submit mostly positive studies, we assessed the percentage of positive articles that authors submitted to the Archives. Of 100 consecutive submitted manuscripts assessed in June and July of 2008, 77% reported significant primary results, based on editors' assessments of the results. If the articles had been categorized based on the authors' interpretation of their analyses, a higher percentage of manuscripts would have fallen into the positive category. Of the manuscripts sent out for external peer review, over 83% of positive studies were accepted by the Archives. Only 3 negative studies were sent to external review, of which only 1 was ultimately accepted. Overall, only 5.3% of all negative studies that were submitted were accepted.

Recognizing that publication bias can result from reviewers' enthusiasm for positive results, we next evaluated the willingness of our 58 most highly rated and prolific peer reviewers to participate in an alternate peer-review process. The proposed hypothetical alternate process involved 2 steps. First, peer reviewers would have access only to a modified abstract containing no mention of results, the full introduction describing the nature of the research question, and a complete “Methods” section to allow an evaluation of the quality of the research. With this information available, the reviewers would be asked to provide a preliminary assessment of the manuscript in the absence of the “Results” section. Following this preliminary assessment, we proposed that reviewers would gain access to the full article, including the “Results” section, and be asked to make a final evaluation of the manuscript. We hypothesized that this 2-stage procedure would force peer reviewers to make an initial evaluation solely based on the quality of the methods and that the result would be a more equitable consideration of well-performed negative studies. Of the 43 respondents, 37 (<86%)>

We then turned to an assessment of the role of the editorial board. Prior to peer review, editors may decide to reject articles on their face value. Furthermore, editors assign reviewers and render final decisions after receiving reviewer comments. At the Archives, an editorial estimate of study rejection without any external peer review was roughly 70% of all submissions, whereas a JAMA study reported a 50% editorial rejection rate at that journal.2 These substantial figures suggest that any investigation of publication bias at the journal level ought to begin with, or at least include, the editors. Consequently, the aforementioned alternate review process was applied to the editorial review that occurred prior to outside peer review. In a pilot study, among a selection of submitted articles, a study was characterized as positive if an author's conclusion about his or her primary outcome was portrayed as such. Of the 46 articles examined, 28 were positive, and 18 were negative (with an explicit attempt to oversample negative studies in this pilot research). Ultimately, 36 of the 46 articles (<77%)>

Although this provides some comfort, it is important to look at not only the majority of manuscripts but also the tail ends of the curve, because this is most likely where any bias would lie. In doing so, we found that over 7% of positive articles benefited from editors changing their minds between steps 1 and 2 of the alternate review process, deciding to push forward with peer review after reading the results. By contrast, in this small study, we found that this never occurred with the negative studies. Indeed, 1 negative study, which was originally queued for peer review after an editor's examination of the introduction and “Methods” section, was removed from such consideration after the results were made available.

We admit that these findings are neither conclusive nor definitive but rather a descriptive analysis from a pilot study. Certainly, it is reassuring that the editors were mostly consistent in their opinions regardless of the results. However, in the minority of cases in which bias matters, the influence of the results on the editor's decision to move to peer review and ultimately to publication is still uncertain. There is a dearth of rigorous research on editorial bias and the possible interventions that may attenuate it. The alternate review process piloted at the Archives has never been performed before, to the best of our knowledge, although it has been suggested.13 Importantly, such a mechanism can be implemented both with editors and peer reviewers, addressing 2 sources of potential bias over which a medical journal can have the most direct impact. The negative trial by Etter et al 7 published in this issue of theArchives was a part of our pilot study. Obviously, the editors supported peer review and publication of this study based on the rigor and quality of its methods alone, and that decision was sustained even when the negative results were revealed to them.

The importance of self-regulation cannot be overstated in the field of medicine. The ultimate voice for the research that guides life-or-death clinical decisions—the medical journal—must staunchly reflect the professionalism that guides medicine. This rings true particularly in the present climate of repeated challenges to the integrity of medical research. The Archives is proud to have taken an initial step in examining our own practices. In this spirit, we encourage other journals to consider a similar approach. In the end, there is little to lose and much to gain from strengthening the rigor, and reducing the biases, of our work.

REFERENCES

1. Olson CM, Rennie D, Cook D, et al. Publication bias in editorial decision making. JAMA. 2002;287(21):2825-2828.

2. Callaham ML, Wears RL, Weber EJ, Barton C, Young G. Positive-outcome bias and other limitations in the outcome of research abstracts submitted to a scientific meeting. JAMA. 1998;280(3):254-257.

3. Falagas ME, Alexiou VG. The top-ten in journal impact factor manipulation. Arch Immunol Ther Exp (Warsz). 2008;56(4):223-226.

4. Rising K, Bacchetti P, Bero L. Reporting bias in drug trials submitted to the Food and Drug Administration: review of publication and presentation. PLoS Med. 2008;5(11):e217.

5. Ioannidis JP. Why most published research findings are false [published online August 30, 2005]. PLoS Med. 2005;2(8):e124.

6. Publication bias: the Cochrane Collaboration open learning material. http://www.cochrane-net.org/openlearning/html/mod15.htm . Accessed January 9, 2009.

7. Etter J-F, Huguelet P, Perneger TV, Cornuz J. Nicotine gum treatment before smoking cessation: a randomized trial. Arch Intern Med. 2009;169(11):1028-1034.

8. De Angelis C, Drazen JM, Frizelle FA, et al; International Committee of Medical Journal Editors. Clinical trial registration: a statement from the International Committee of Medical Journal Editors. N Engl J Med. 2004;351(12):1250-1251.

9. Vastag B. Reviewer leaked Avandia study to drug firm. Nature. 2008;451(7178):509. 10. Godlee F. Making reviewers visible: openness, accountability, and credit. JAMA. 2002;287(21):2762-2765.

11. Emanuel LL, Greenland P. Peer review and professionalism at the Archives of Internal Medicine. Arch Intern Med. 2005;165(22):2559-2560.

12. Hayden SR. Have you read the Hippocratic oath lately [published online December 19, 2008]? J Emerg Med. doi:10.1016/j.jemermed.2008.07.024.

13. Glymour MM, Kawachi I. Review of publication bias in studies on publication bias: here's a proposal for editors that may help reduce publication bias.BMJ. 2005;331(7517):638.

Hat tip: http://pharmacritique.20minutes-blogs.fr/