Published ahead of print on August 1, 2002, doi:10.1164/rccm.200204-324OE
© 2002 American Thoracic Society
How I Review an Original Scientific ArticleDepartments of Medicine, Memorial Hospital of Rhode Island and Brown University, Pawtucket, Rhode Island Correspondence and requests for reprints should be addressed to Frederic Hoppin, M.D., Division of Pulmonary and Critical Care Medicine, Memorial Hospital of Rhode Island, 111 Brewster Street, Pawtucket, RI 02860. E-mail: Frederic_Hoppin_Jr{at}Brown.edu WHY WRITE THIS? There has been substantial recent interest in the quality of the peer review system in biomedical publication, with several International Congresses (1) and a recent JAMA issue entirely devoted to the topic (2). The quality of reviews of articles submitted for publication varies widely (1, 3–5). Black and colleagues have suggested that their quality might be improved if journals trained their reviewers (6). How do we currently learn the trade? Some of us have learned by doing reviews, by fielding reviews of our own submissions, and by comparing our own reviews with other reviews of the same articles. When editorial consideration of a submission is completed, the editorial offices generally forward all correspondence to the referees. I always look at this correspondence because it often reveals new insights and provides useful feedback on my review. Some, more fortunate, started by drafting reviews for their seniors (with confidentiality strictly protected) and then engaging in an intense tutorial over the science, the presentation, and the review itself. Lock's comprehensive and scholarly review of editorial peer review contains a very useful set of guidelines (4), and there are other relevant publications (3, 5–10). But, to my knowledge, no how-to-review paper has been published. I hope that some of the lessons I have learned over the years as reviewer and onetime Associate Editor and the practices that I follow might be helpful to the novice and might provide affirmation and perhaps a pointer or two for the experienced reviewer. WHAT DOES IT TAKE TO DO A GOOD REVIEW?
Motivation
Scientific Expertise Several different areas of expertise may be relevant for a given submission. A paper that is sent to me, for example, may include elements of clinical and applied science, general pulmonary physiology, basic lung and chest wall mechanics, mathematical modeling, or stereology. Although my expertise is uneven among these topics and a submission often requires significant expertise in disciplines that I cannot cover responsibly, the Associate Editor usually turns out to have selected reviewers to cover all main areas.
Helpful Attitude Yet, an insightful and articulate review can substantially improve the science and clarity of a submitted paper (8) and can advance the authors' knowledge and ability to conduct and report science. The reviewer can be fully as helpful as an involved laboratory colleague or a visiting professor. My approach is to be resolutely respectful. This does not mean watering down the review; downplaying a concern; failing to demand justification, explanation, and clarity; or avoiding a clear recommendation. It does mean (even late at night, after a busy day, with a marginal manuscript) reading with patience, objectivity, and openness to new ideas and approaches, and reporting with complete clarity and without summarily closing off debate. It also means being careful not to give rein to my competitive instincts.
Time Senior reviewers, surprisingly, are reported to do a worse job than their juniors (7). That the seniors also spend less time (7) may be the explanation for the lesser quality! Another possibility is that the seniors are more ready to cut a review short when they determine that a paper has clear, serious, irreparable scientific deficiencies and believe there is no need to detail all deficiencies in a sloppily written paper. Nonetheless, reviewers should be warned that "time is of the essence," in this setting means "spend it, don't hurry it, even if you are senior." How an academician can find the time is a critical issue. Although there are certainly many important intangible benefits to reviewing (e.g., broadening one's scientific knowledge, enjoying the scientific interchange and debate, fulfilling a sense of responsibility), the tangible benefits are limited to the possibility that gaining the respect of the editors might lead to invitations to participate in national societies, an editorial board, or a study section. Furthermore, the job competes with activities that have immediate rewards or accountability, e.g., teaching, preparing grant applications, performing research, seeing patients. It would be very helpful if the academy could be structured to reward this activity more directly (11). HOW DO I PROCEED?
Acceptance
First Reading I then read the article closely, focusing primarily on understanding the science. I stop wherever I do not fully understand the science from what is written, where some aspect of the science is troubling, or where I believe the authors may have failed to put their work into fair and full perspective. I attempt to characterize each such problem in a preliminary fashion. I do not look for specific errors, as from a checklist. The process goes in the other direction, e.g., a question about the science occurs to me, the answer does not, and my task then is to identify the specific error. This last task is not always easy or immediate. I may have to check the literature, consult a colleague, or do some hard thinking. Errors of presentation may be more readily identified than are errors of the science, but it is often unclear whether a problem arises from fuzzy presentation, fuzzy thinking, or both (8). What sorts of problems do I encounter? I will give some categories, descriptions, and a few examples below. My intent, again, is not to provide a checklist but rather to help the reader to characterize the problems he or she encounters.
Problems with the science. Many problems arise from failure to apply available, specific knowledge. The authors have not applied relevant basic scientific principles, have not considered a likely methodological uncertainty, have failed to recognize a confounding factor, have not considered the appropriate statistical power (14). For example, I have seen more than one study in which the authors reported measurements that depended on chest wall configuration made at total lung capacity, without specifying whether total lung capacity had been maintained actively with an open glottis or passively with relaxation against a closed glottis. I had to presume, until I heard otherwise, that the authors, unaware of the substantial difference of configuration or of its implications, had failed to control a potentially confounding variable.
Problems with the ethics.
Problems with the presentation. Redundancies, irrelevancies, and unnecessary excursions are relatively minor sins but may impair communication by boring and distracting the reader. Failures to define terms or to use words with precision are more serious because they can mislead. Noncolloquialisms, common when English is not the authors' native tongue, distract and can mislead. Jargon (by which I mean any nonstandard word, idea, or even argument that has become so familiar to the authors that they neglect to explain and delimit it) is a prevalent and insidious problem. At its most benign, jargon annoys and fudges. This is why, for example, nonstandard abbreviations are strictly limited by many journals. Worse, jargon can mislead. For example, the phrase "inflection point" is used in a number of current clinical ventilator studies to designate the distinct upward deflection or "knee" on the inflation limb of a lung pressure–volume curve. However, to the vast majority of scientists and all lexicographers (so far), the term designates a very different point on the curve, namely where it changes from concave to convex or vice versa. Jargon can be as subtle as the use within a given article of two closely related terms or phrases that the authors may or may not intend to be exactly equivalent, for example, "pulmonary function" and "pulmonary function tests." Very common terms can become jargon when they are not carefully defined for the purposes of the paper; for example, functional residual capacity can differ substantially depending on which of several acceptable definitions is applied: (1) a mechanistic definition—the lung volume where the sum of static lung and passive chest wall recoils is zero; (2) a functional definition—the lung volume at the end of a relaxed, prolonged expiration; or (3) another functional definition—the lung volume at the end of a series of ongoing expirations under any one of a variety of specified scenarios. I am often confused by quite pedestrian errors. For example, without commas to set it off, a dependent phrase may run on to the rest of the sentence, and the reader is interrupted while searching for a contextual clue to the syntax. This is particularly problematic in scientific writing, as it tends to contain long series of nouns, e.g., "... hospital outpatient weight control program standards ..." As another example, compare the statements "It was concluded that ..." and "Our data, however, show that ..."—the identity of the authors who reached the conclusion (the distinction may be important) is clear in the active voice but equivocal in the passive voice. Even spelling mistakes may not be benign—a computer spellchecker will never reject an "ever" that should have been a "never," and a technical editor may not follow the science well enough to catch the error. Many articles are poorly focused. The thrust of a paragraph, for example, should be clear at the beginning, e.g., a "topic sentence." Another example, I often see a set of data strung out in the text of the RESULTS section in a serial recitation of means, standard deviations, and "n." If the message to be drawn from the data resides in comparisons within the set, this practice burdens the text, is less accessible than a figure or table where the reader can readily make the requisite comparisons, and commonly displaces an explicit statement of what the authors want the reader to see. How much more focused, concise, and informative it is to say simply, "[Variable A] increases linearly with [Variable B] (see Figure 3, and Table 2)." It is astonishing how often authors fail to develop their ideas systematically, i.e., to lead the reader through their thinking. For example, the reader needs to know the basis for the experimental design at the outset. Yet I often see an idea that is important to the experimental design postponed (perhaps in a misguided effort to avoid redundancy) until the DISCUSSION, where it is fully developed. Both purposes can be readily accomplished by identifying the idea in the INTRODUCTION, together with an appropriate road sign, e.g., "as is developed in more detail in the DISCUSSION ..." Even worse, an astonishing number of submissions fails to be explicit about the logical structure of the study, for example by failing to specify goals, hypotheses, testable predictions of the hypotheses, and conclusions, perhaps under the illusion that the logical structure of the study is so obvious as to "go without saying." I do not keep a checklist of what must appear in a paper. Instead, I keep asking the general question, "What is missing?" Some examples: Have the authors acknowledged other reasonable hypotheses? For a given argument, have they specified, examined, and assessed the impact of all reasonable assumptions? Have they considered methodological limitations? Does the DISCUSSION address all discrepancies or agreements between their results and those of other workers? This is almost an attitude on my part. Gross errors, such as percentages that do not add up to 100, have often slipped by reviewers into the archival literature. I do look quickly at every datum (in a viable paper), but I cannot take the time to check calculations unless something looks way out of line. I regularly find significantly misleading or inaccurate statements about specific citations—sometimes I knew the citations, sometimes I checked out the citation because the attribution seemed odd or was particularly critical to the science being presented.
Notations. Finally, I return to the front page to list the main issues. This is an ordered list, informed by the broad questions that I have already listed on the front page, by the more substantive notations throughout the text, and by the abstract, which I take as representing what the authors think is important. I then put the manuscript aside for a day or so because important insights and perspectives often occur to me while I am doing something else and because returning to it enforces an initial "view from 40,000 ft."
Second Reading
Criteria for judging the science. I would also like to know if the article is important. Sometimes an article appears to provide a convincing answer to a question of current interest. The absence of such a connection, however, does not preclude ultimate importance. The main reason is the prevalence of serendipity in scientific progress. This was elegantly demonstrated by Comroe and Dripps (15). They selected the 10 most important clinical advances in cardiovascular–pulmonary medicine and surgery over the preceding 30 years. They identified and then examined 529 articles that had important effects on the direction of subsequent research and development, which in turn proved to be important for 1 of these 10 clinical advances. An astonishing 41% of the articles reported work "that, at the time it was done, had no relation whatever to the disease that it later helped to prevent, diagnose, treat, or alleviate." So I look instead for novelty of idea, conclusion, data, or methodology. These criteria are relatively easy to apply. An article that is both new and has scientific integrity has a shot at turning out to be important. I avoid making a judgment on the basis of a particular study being applied as versus basic. Applied studies may have the appeal of practical relevance, and basic studies the appeal of broad relevance, but landmark studies have been published over the full spectrum from applied to basic. I do not consider politics or the reputation and academic status of the authors. The referees' anonymity, incidentally, can help insulate the Associate Editor in that regard. Hesitation to challenge weakness in articles submitted by well-respected scientists and friends would serve them and the journal poorly.
Criteria for judging the presentation.
Recommendations. What do I recommend when an article formulates a relatively compelling question, or puts forward an intriguing idea, but the science is weak? I convey to the authors what might improve the science, and I describe the pros and cons to the Associate Editor, who then has a difficult decision to make. A somewhat mischievous perspective on this issue is given by Julius Comroe, in one article of his delightful "Retrospectroscope" series in this journal. He pointed out how briefly, informally, and even incidentally a number of the truly great advances in science were first introduced (16). In one example, he quoted the 267 words in which Korotkoff described and explained the basis for the now ubiquitous clinical method of determining blood pressure. Comroe concluded with the following mischievous fantasy: "Dear Dr. Korotkoff: I doubt that I would have had the foresight to recommend publication. What do I recommend when the article offers only a minor advance? Authors seem to be in more of a hurry to publish than in years past, perhaps due to (1) a larger cadre of competing investigators, (2) awareness that promotions committees are better at counting papers than they are at evaluating them, and (3) the felt need to establish a track record for funding. I often find it helpful to look at other papers on the topic from the same laboratory, which may show that the submission contributes to an orderly and productive evolution of ideas or turn up a pattern of repetitive "churning" of data and ideas. If this inquiry does not clarify, I recommend asking the authors to specify and defend exactly what is new in their submission; the burden, really, is on them. How does the adequacy of the presentation bear on my recommendation? Mostly as an absolute threshold, namely that the reader must be able to make an independent judgment about the strengths and weaknesses of the authors' data and conclusions from what is presented (8). How to balance high standards against the purpose of the archival literature, which is to enable scientists to communicate? How to avoid being a curmudgeon on the one hand and a soft touch on the other? The Associate Editor brings his or her own calibration into evaluation of my review. Nonetheless, I keep an eye on the severity and content of the other reviewers' comments on the same articles and keep in mind that more than 70% of submissions to AJRCCM, for example, are not accepted. Often my recommendation reflects suspended judgment, pending a response from the authors. I am particularly careful to give them the opportunity to respond in the case of a potential fatal flaw; sometimes they can readily clear up the issue, sometimes not. Once and only once, after prolonged rumination, I concluded that the central reasoning in an article was circular. This was put to the authors, who responded, "You are right. Thanks. We withdraw the paper." THE WRITE-UP
Comments to the Editors The summary, in three or four sentences, identifies the topic of the study, indicates the basic approach, selects the main findings, and paraphrases the authors' main conclusions. The summarizing exercise is important to me for distilling my thoughts, and it provides the Associate Editor with the background for the main criticisms/questions that follow. I then list several main criticisms/questions in descending order of importance. These are selected from among the list on the front page. For each, I summarize its basis, postponing a full explanation to the "Comments to the Authors." I also indicate what I see as the importance of each item and what I think the authors may be able to do in response. Finally, I indicate and characterize my recommendations, e.g., "This is a novel idea, worth inviting major revision (see COMMENTS TO THE AUTHORS)." I also indicate my degree of confidence in my recommendation, e.g., "I suspect that the authors will have difficulty answering Question 1 satisfactorily." I acknowledge any major help I have received–not to absolve myself of any responsibility for the recommendation but to give credit where it is due and to introduce individuals whom the Associate Editor might find helpful as reviewer in the future. I once informed the Associate Editor of a potential conflict of interest when I reviewed a paper that reported the characteristics of a device in which the author had a personal commercial interest and that might have been considered to be less than fully objective.
Comments to the Authors Some ground rules: (1) my observations and analysis must be clear to the authors. The recommendations themselves, however, are private, and should not preempt the Associate Editor; (2) I downplay praise. If the article is accepted, the authors will be pleased enough, regardless of any praise. If the article is rejected and I had praised it, the Associate Editor may have an uncomfortable interchange with the authors at the next national meeting; (3) I avoid censure, even of chronic offenders, as it is unnecessary, belittling, and disrespectful. Each submission is on the order of a man-year's work by my fellow scientists, and the stakes are high for them. The first paragraph is a direct copy of my summary at the beginning of "Comments to the Editor." From this, the authors can learn what I took from their presentation (possibly a surprise to them), which may help them to focus and prepare their responses and revision. If there are aspects of the study that I have not evaluated, I say so explicitly, e.g., "I have reviewed the applied math carefully for its assumptions and physiologic implications, but am not equipped to review the math itself." This is followed by "Major Comments," numbered and starting with the main points made in the "Comments to the Editors." Each criticism/question must be explained to the authors and never left as unsupported, qualitative statements, such as "inadequate controls"—there must always be a "because ..." I have occasionally indicated alternative approaches, either to explain a criticism or question or to let the authors pick up a useful suggestion. This practice carries the risk that the alternatives may reflect my own scientific style or arbitrary choices rather than an objective scientific evaluation. With regard to presentation, I have often heard it said that referees have limited time and responsibility, that deficiencies of presentation are the authors' responsibility, that it is presumptuous for authors to submit articles without first obtaining meticulous, critical reviews from in-house or out-house colleagues, that submissions should always be gone over carefully by someone for whom English is the native tongue, and that any negative consequences of weak and hasty presentation are well deserved. I sympathize deeply with each of these arguments, but the overriding consideration is that good science should be published. One can call the authors on sloppiness by giving them a few egregious examples and generalizing the complaint. It is more challenging to help with poor exposition, e.g., it is difficult to explain why I do not understand what I do not understand! The best path through this particular thicket starts with telling the authors what I did understand from what they wrote, e.g., "this appears to say ... ," or with telling them where I got lost, e.g., "at this point I found myself wondering ..." Once, exasperated with a chronic offender, who routinely presented novel science very badly, I pointedly suggested a rigorous, in-house, line-by-line revision by his senior author. Once, having had to work hard to untangle an argument in an otherwise worthwhile paper, I wrote, "Is this what you are saying?" and outlined my version. The authors accepted it fully. And gratefully (or so they said). I have never come close, however, to doing what Jere Mead once did, namely completely rewriting a foreign paper, believing that otherwise some excellent science would have been lost! For many specific problems with the presentation, it may be useful to refer the authors to online sites (17). The final section is "Minor Comments." This is culled from my notations in the text, (i.e., about redundancies, inappropriate symbols, and the like) listed in the order they appear in the text and identified by page, paragraph, and line. The issue of reviewer anonymity is controversial (11). Most journals presume that the referee prefers anonymity and that this will help ensure relative objectivity. Many of us, however, sign some of our reviews if we think the authors might welcome direct dialog later. FINAL COMMENT I asked a colleague recently how his recent submission had fared in review. "One bad review, one good review," he replied. "The ‘bad’ one liked it but was really superficial—I don't think they understood it. The ‘good’ one didn't like it much, but the review was just wonderful." By which he meant it was really insightful and helpful to the quality of his science. Acknowledgments Uninhibited in-house reviews of an early draft of this article by Robert Banzett and James Butler were very helpful and warmly appreciated. Received in original form April 15, 2002; accepted in final form July 25, 2002 REFERENCES
This article has been cited by other articles:
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
(
