This Article Full Text (PDF) Alert me when this article is cited Alert me if a correction is posted Services Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Download to citation manager Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Google Scholar Articles by Tobin, M. J. Search for Related Content PubMed PubMed Citation Articles by Tobin, M. J.

Rigor of Peer Review and the Standing of a Journal

Shortly after launching The Philosophical Transactions of the Royal Society in 1665, Henry Oldenberg, the first editor, wrote to Robert Boyle that a submitted manuscript needs to be carefully scrutinized "before we give a publick testimony of it to ye world, as is desired of us" (1). Right from the start, peer review was used as an instrument of quality control to distinguish scientific journals from book publishing. Through the emergence of reviewers, journals could insist on the highest standards for excellence before publishing a report of new research findings. This exacting scrutiny by fellow experts is without parallel in any other field, and the reviewer has been described as "the lynchpin about which the whole business of science is pivoted" (2).

For a component of pivotal importance to the progress of science, journals provide scant guidance to reviewers. The confidential and anonymous nature of editorial peer review makes it especially difficult for the novice to learn the skill. In this issue of AJRCCM (pp. 1019–1023), Hoppin (3) describes how he reviews a manuscript. He considers the task one of the utmost responsibility, and writes with wisdom garnered from more than 30 years of experience. His essay is unique: I am not aware of another published description of how to review a manuscript and the cognitive processes involved. All reviewers will benefit from reading Hoppin's essay, with resulting enhancement of the quality of their critiques.

The publication of an article in a journal represents not only the work of the authors, but it also reflects the standards of that journal. The standing of a journal among other journals is determined by the expectations and demands of the scientists who serve as reviewers. Because the reviewers also submit their own research to the journal, they embody the journal's standards for scientific excellence. When reviewers believe that a journal publishes work of a low standard, they will be less demanding in their critiques, the journal will publish manuscripts of lower quality, and its standing will fall. Authors regard reviewers as hurdles to vault before their work gets published and find it hard to see them as offering help. Readers, however, depend on reviewers to scrutinize a manuscript and guarantee them it will be worth the time invested in reading it. Readers also benefit from the improvements made to articles through the intellectual capital added by reviewers; in turn, the effort made by the reviewers depends on their assessment of the role of that journal in the advancement of science. And rounding this positive feedback loop, journals that employ the toughest reviewing procedures are the most attractive to authors who hold themselves to the highest standards.

Editorial peer review has always had its critics. At its dawn, Isaac Newton was complaining that addressing criticisms of his submission to The Philosophical Transactions "had sacrificed my peace (of mind), a matter of real substance" (4). Research into peer review has only gotten underway in the last 10 to 15 years and has been slow to uncover major benefits (5–7). These studies have been interpreted to mean that editorial peer review has no value (8). Absence of proof, however, is not proof of absence. It is not possible to do the definitive outcome study: to randomize two continents to the stricture, or lack thereof, that the results of all experiments be published in peer-reviewed journals and then to follow the progress of science in each continent for 300 years. Moreover, research into peer review has not included rigorous studies of its most critical and ineffable components: the making of subjective value judgments, and the effect on manuscript quality and relevance.

Any discussion of the validity of peer review and science must ultimately grapple with ontology, the branch of philosophy that asks what actually exists, and epistemology, the branch of philosophy that asks what, if anything, we can know (9). Most scientists would be satisfied if they could achieve the epistemologic status of empirical truth revealed by objective measurements in an experiment. In this context, the term "objective" refers to information obtained without human intervention. But the design of an instrument and the interpretation of findings always involve a component of human judgment. In scientific epistemology, the most rigorous standard of empirical objectivity we can hope for is "intersubjective agreement" on the findings (10–13). The objectivity of scientific knowledge resides in its being a social construct: its communal institutions, norms, and activities, including the peer-review system, are not just a background for the logic of the scientific method—they are constitutive of science as we know it (13). Peer review, of course, can never guarantee validity, and being a human enterprise, peer review cannot be free of error, any more than can polygraph tracings or digital readouts. Scientific journals are archives of work done, not of revealed truth (14). It is time that decides validity.

Hoppin touches on the issue of why scientists donate so much time to the review of manuscripts submitted by the fellows of other scientists. This community of scholars recognizes that they belong to an intricate social system in which one has to contribute if one is to receive (15). Scientists also derive several benefits from reviewing: they learn about cutting-edge research long before it is published; they improve their skills as critical appraisers by comparing their critiques against those of other reviewers and seeing the decision letters of associate editors; and they learn how to write more competitive manuscripts by seeing the faults of others. Each year, AJRCCM publishes a list of individuals who reviewed manuscripts for the Journal. And appointment to the Editorial Board of AJRCCM is based solely on the cogency, rigor, and timeliness of the individual's performance as a reviewer. The greatest reward, however, is the self-awareness that an individual is serving as one small cog in the wheel of medical progress. In any social enterprise, the best individuals volunteer their services without caring about payment. They find sufficient reward in knowing that the work of the moment is connected to a much greater process: in this case, science leading to improved care of patients—work of ever-lasting value. You cannot, however, have a social enterprise of this magnitude without some parasites. Every journal has a handful of authors who phone or write about their every submission, demanding expedited treatment, and in turn are delinquent or provide superficial comments when invited to review the manuscripts of other authors.

Online peer review has reduced the time between submission and first decision to 33 days at AJRCCM. More important than speed, the online system has enhanced the rigor of peer review. Formerly, we assigned a manuscript to two reviewers, but now commonly assign four or more reviewers. The descriptor number on a manuscript allows us to instantly identify the reviewers with the greatest expertise in each of the 172 areas covered by AJRCCM. We have more than 5,600 reviewers in our database, and associate editors can assess level of expertise and prior performance by clicking a mouse when assigning manuscripts. New reviewers are constantly added to the database, and we also maintain a list of delinquent and superficial reviewers. The internet is revolutionizing the speed of processing manuscripts and the dissemination of new findings, but the bedrock of science has not changed since the 1660s: experiments are converted into science only after the results have been published in a peer-reviewed journal. This point is well illustrated by Leonardo da Vinci, perhaps the finest scientific mind ever. Leonardo's notebooks are records of the most intense scientific efforts by a single individual. But his research efforts came to nothing, because the stricture for disciplined publication was not formulated until more than 150 years after Leonardo's death.

REFERENCES

1. Zuckerman H, Merton RK. Patterns of evaluation in science: institutionalization, structure and functions of the referee system. Minerva 1971;9:66–100.[CrossRef]
2. Ziman JM. Public knowledge: an essay concerning the social dimension of science. Cambridge: Cambridge University Press; 1968. p. 111.
3. Hoppin FG. How I review an original scientific article. Am J Respir Crit Care Med 2002;166:1019–1023.[Free Full Text]
4. Siegelman SS. The genesis of modern science: contribution of scientific societies and scientific journals. Radiology 1998;208:9–16.[Medline]
5. Lock S. A difficult balance: editorial peer review in medicine. London: British Medical Journal; 1991.
6. Goodman SN, Berlin J, Fletcher SW, Fletcher RH. Manuscript quality before and after peer review and editing at Annals of Internal Medicine. Ann Intern Med 1994;121:11–21.[Abstract/Free Full Text]
7. Pierie JP, Walvoort HC, Overbeke AJ. Readers' evaluation of effect of peer review and editing on quality of articles in the Nederlands Tijdschrift voor Geneeskunde. Lancet 1996;348:1480–1483.[CrossRef][Medline]
8. Smith R. The future of peer review. In: Godlee F, Jefferson T, editors. Peer review in health sciences. London: BMJ Books; 1999. p. 244–253.
9. Magee B. The story of philosophy. London: Dorling Kindersley; 1998. p. 8.
10. Polanyi M. Science, faith and society. Oxford: Oxford University Press; 1946.
11. Polanyi M. Personal knowledge: towards a post-critical philosophy. London: Routledge and Kegan; 1958. p. 225–257.
12. Gross AG. The rhetoric of science. Cambridge, MA: Harvard University Press; 1996. p. 129–143.
13. Ziman J. An introduction to science studies: the philosophical and social aspects of science and technology. Cambridge: Cambridge University Press; 1984. p. 34–57.
14. Relman A. Journals. In: Warren KS, editor. Coping with the medical literature: a primer for the scientist and the clinician. New York: Praeger Publishers. 1981. p. 67–78.
15. Ingelfinger F. Peer review in biomedical publication. Am J Med 1974;56:686–692.[CrossRef][Medline]

This article has been cited by other articles:

				M. J. Tobin Thirty Years of Impact Factor and the Journal Am. J. Respir. Crit. Care Med., August 15, 2004; 170(4): 351 - 352. [Full Text] [PDF]

				M. J. Tobin Mentoring: Seven Roles and Some Specifics Am. J. Respir. Crit. Care Med., July 15, 2004; 170(2): 114 - 117. [Full Text] [PDF]

				M. J. Tobin Assessing the Performance of a Medical Journal Am. J. Respir. Crit. Care Med., June 15, 2004; 169(12): 1268 - 1272. [Full Text] [PDF]

				K. Parameswaran and M. J. Tobin Financial Disclosure of Reviewers Am. J. Respir. Crit. Care Med., October 15, 2003; 168(8): 1010 - 1010. [Full Text]

				M. J. Tobin Writing a Review Article for AJRCCM Am. J. Respir. Crit. Care Med., October 1, 2003; 168(7): 732 - 734. [Full Text] [PDF]

				M. J. Tobin Impact Factor and the Journal Am. J. Respir. Crit. Care Med., September 15, 2003; 168(6): 621 - 622. [Full Text] [PDF]

				H. M. Druce, F. G. Hoppin Jr., and M. J. Tobin Peer Review and Manuscript Triage Am. J. Respir. Crit. Care Med., August 1, 2003; 168(3): 397 - 398. [Full Text] [PDF]

				M. J. Tobin Conflicts of Interest and AJRCCM: Restating Policy and a New Form to Upload Am. J. Respir. Crit. Care Med., May 1, 2003; 167(9): 1161 - 1164. [Full Text] [PDF]

				M. J. Tobin Tuberculosis, Lung Infections, Interstitial Lung Disease, and Journalology in AJRCCM 2002 Am. J. Respir. Crit. Care Med., February 1, 2003; 167(3): 345 - 355. [Full Text] [PDF]

				M. J. Tobin The Journal in 2002 Am. J. Respir. Crit. Care Med., November 1, 2002; 166(9): 1153 - 1156. [Full Text] [PDF]

This Article Full Text (PDF) Alert me when this article is cited Alert me if a correction is posted Services Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Download to citation manager Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Google Scholar Articles by Tobin, M. J. Search for Related Content PubMed PubMed Citation Articles by Tobin, M. J.