INTRODUCTION

TOP ABSTRACT INTRODUCTION METHODS RESULTS DISCUSSION REFERENCES

To apply the results of health-care research to their practice, clinicians judge the similarity between their own patients and study participants. Reproducible characterizations of patient populations are crucial to this process. Obtaining agreement when determining the presence or absence of clinical syndromes such as acute respiratory distress syndrome (ARDS) may be particularly challenging. Although investigators have offered a number of definitions of ARDS (1), two have gained popularity. The North American-European Consensus Committee (NAECC) statement provides the simpler of the two definitions (2). The NAECC requires an acute onset of respiratory failure; a ratio of partial pressure of oxygen (Pa_O2) to fractional concentration of inspired oxygen (FI_O2) of less than 200 regardless of positive end-expiratory pressure (PEEP); diffuse bilateral infiltrates on frontal chest radiograph; and pulmonary capillary wedge pressure less than 18 cm H₂O or, in the absence of a pulmonary artery catheter, absence of clinical evidence of left atrial hypertension.

The more complex definition, the Lung Injury Severity Score (LISS), requires clinicians to score the degree of abnormality in oxygenation using the Pa_O2/FI_O2 ratio, on chest radiograph (scored as the number of quadrants with alveolar consolidation), and in compliance of the respiratory system, as well as the level of PEEP, each on a five-point scale from 0 to 4 (3). In situations in which compliance is not available clinicians can calculate the score omitting this component. Similarly, PEEP need not be included if patients are not mechanically ventilated. Clinicians add the score, divide by the number of components, and classify the patient as having ARDS if the score is greater than 2.5.

The NAECC definition does not involve two LISS components, respiratory system compliance and PEEP. Although the LISS definition assumes clinicians will exclude heart failure patients, it does not provide explicit guidance for identifying these patients. These differences could potentially lead the two systems to classify different groups of patients as having ARDS. In addition, the different approaches to defining ARDS (using absolute criteria for NAECC and a composite score for LISS) could contribute to differences in patient classification.

Some investigations in acutely ill patients tend to use the LISS definition, and some use the NAECC definition. For example, among studies of pressure and volume-limited ventilation for patients with ARDS, Brochard and colleagues (4) and Amato and colleagues (5) used the LISS score to classify their patients, whereas Brower and colleagues (6) used the NAECC criteria. If the two systems classify patients differently, a number of problems are likely to ensue.

First, clinicians may have difficulty in determining to whom among their patients the results of ARDS studies are applicable. Clinicians concerned about ensuring that care in their intensive care unit (ICU) results in comparable outcomes to other high-quality units may find their attempts to compare ARDS outcome confounded by the use of different criteria. For instance, if different units use alternative definitions, groups of patients with underlying differences in prognosis could all be given the ARDS label. The result would be that differences in underlying patient prognosis would be misinterpreted as differences in quality of care.

These differences in underlying prognosis due to classification anomalies may further result in misleading apparent changes in prognosis over time (1), which would be incorrectly attributed to changing patterns of care. A similar phenomenon would bias observational studies examining the impact of alternative treatments: a group with an underlying better prognosis would result in a spurious attribution of treatment benefit. Finally, clinical investigators in acute lung injury and acute respiratory distress syndrome risk finding that they are unable to achieve their expected rate of patient recruitment or that their anticipated study event rate is grossly inaccurate as a result of assuming that the two definitions of ARDS classify patients in a reproducible manner. A recent review article addresses many of these issues related to ARDS definitions and prognosis (7).

Clinicians and researchers need not be concerned about these issues if alternative definitions of ARDS reproducibly classify patients in a similar fashion. Up to now, investigators have made few head-to-head comparisons of alternative definitions in the same group of patients. To determine the consistency with which they classify patients as having or not having ARDS, and to evaluate the sources of inconsistency and the importance of any inconsistency, we therefore applied the NAECC and LISS definitions to patients enrolled in a randomized trial of a lung-protection ventilation strategy in patients with acute lung injury.

	METHODS

TOP ABSTRACT INTRODUCTION METHODS RESULTS DISCUSSION REFERENCES

We previously conducted a randomized trial comparing two ventilation strategies in 120 adults with acute lung injury (8). Inclusion criteria included intubation < 24 h, peak airway pressures  30 cm H₂O, Pa_O2/FI_O2 < 250, and  1 ARDS risk factor. We excluded patients with anticipated duration of ICU admission < 48 h, very unlikely survival, heart failure, high risk of cardiac arrhythmia or ischemia, acute asthma, intracranial hypertension, or pregnancy. Some patients had ARDS at enrollment, and some developed ARDS during the study.

Study patients had daily chest radiographs. For the 99 patients in whom chest radiographs were available after the trial, a radiologist and an intensivist underwent training to standardize their review methods. They independently determined the number of quadrants with consolidation, and the presence or absence of diffuse bilateral infiltrates, and resolved disagreements by reviewing films together. For the remaining patients, we used the initial interpretations of radiographs that study intensivists had provided.

We collected Pa_O2, FI_O2, PEEP, and pulmonary capillary wedge pressure (PCWP) measurements at 8-h intervals. We calculated respiratory system compliance as exhaled tidal volume divided by (plateau pressure minus set PEEP). The trial excluded patients with heart failure, and we assumed study patients remained free of heart failure (PCWP was measured at least once in 59% of the patients and 779 of 3,014 observations included a PCWP measurement). Using NAECC and LISS definitions (Table 1), we classified each patient as having, or not having, ARDS at any time during the study. Complete data were available for 118 patients.

We calculated the proportion of patients meeting ARDS criteria for each classification, and used McNemar's chi-square test to determine whether the difference in proportions could be explained by chance. To explore the relative contribution of different score components to agreement and disagreement, we conducted a second analysis in which we included only observations when compliance was recorded.

We quantified the agreement between classifications using raw agreement, chance-corrected agreement (kappa), and chance-independent agreement (phi) (9). We interpreted kappa and phi results as follows: values of less than 0poor; 0 to 0.2slight; 0.2 to 0.4fair agreement; 0.4 to 0.6moderate agreement; 0.6 to 0.8substantial agreement; and values of 0.8 to 1.0 represent almost perfect agreement (14).

We recalculated agreement after leaving out individual components of each definition (for instance, leaving out radiographic criteria from each definition). We varied NAECC thresholds for the Pa_O2/FI_O2 from 200 to 300. To further explore the impact of varying the NAECC threshold for Pa_O2/FI_O2 we used the LISS as a reference standard and calculated the sensitivity and specificity of the NAECC. Finally, we varied the thresholds for ARDS diagnosis by LISS from 2 to 4. Because results were very similar whether we used the complete data set or restricted ourselves to observations in which compliance data were available, we present only the latter set of analyses.

Finally, we explored the prognostic significance of an NAECC versus a LISS ARDS classification by comparing patients in these two groups according to baseline characteristics and outcome.

	RESULTS

TOP ABSTRACT INTRODUCTION METHODS RESULTS DISCUSSION REFERENCES

Agreement between the two trained chest radiograph reviewers on the presence of bilateral infiltrates was excellent (raw agreement 0.89, kappa 0.72, phi 0.75) (12). Although not as high, agreement between the trained and untrained reviewers was also acceptable (raw agreement 0.68 to 0.72, kappa 0.38 to 0.47, phi 0.59 to 0.73). We found similar levels of agreement for the rating of the number of quadrants with consolidation (trained reviewers weighted kappa 0.74, untrained reviewers 0.47 to 0.54).

Table 1 presents the incidence of ARDS using the two definitions and presents the extent to which the two classifications agreed. The incidence of ARDS using the NAECC criteria was 55.1% (95% confidence interval [CI], 46.1% to 64.1%), and using the LISS criteria 61.9% (95% confidence interval, 53.1% to 70.6%) (p value on the difference between proportions 0.07).

The raw agreement on the presence of ARDS using the two classifications was 0.73 (95% CI, 0.65 to 0.81). The chance-corrected agreement (kappa) was 0.46 (95% CI, 0.32 to 0.61) and the chance-independent agreement (phi) was 0.63 (95% CI, 0.41 to 0.79).

Compliance was measured at least once in 65% of the patients, and 824 of 3,014 observations included a compliance measurement. When we restricted ourselves to observations when compliance was measured, we found that raw agreement between the two classifications rose to 0.83 (95% CI, 0.76 to 0.90) and the kappa to 0.65 (95% CI, 0.52 to 0.79) and the chance-independent agreement 0.67 (95% CI, 0.50 to 0.81) (Table 2). When we explored reasons for the disagreement we found that removing components either reduced agreement or left agreement unchanged (Table 2). Thus, no component was particularly responsible for the disagreement.

When we used an NAECC threshold of 300 for Pa_O2/FI_O2 raw agreement fell from 0.83 to 0.75. When we used the conventional Pa_O2/FI_O2 of 200 for defining ARDS according to NAECC and used LISS as the criterion standard we found a sensitivity of 0.81 and specificity of 0.87. Using a threshold of 250 sensitivity rose to 0.82, but specificity fell to 0.76. Using the 300 threshold, sensitivity did not increase further (0.82), whereas specificity decreased to 0.62. These results demonstrate that increasing the Pa_O2/FI_O2 threshold leads to minimal gains in sensitivity and diminishing specificity. Thus, agreement will deteriorate with a higher threshold.

When we used a LISS cut of 2 instead of the conventional 2.5 we found raw agreement decreased from 0.83 to 0.73, kappa fell from 0.65 to 0.42, and phi decreased from 0.67 to 0.66. When we used a LISS cut of 3, raw agreement decreased from 0.83 to 0.75, kappa decreased from 0.65 to 0.51, and phi decreased from 0.67 to 0.65. Agreement continued to fall as we tested LISS thresholds above 3 (for instance, at 3.5 raw agreement was 0.63 and kappa was 0.30). Thus, the conventional 2.5 cut optimizes agreement between the two measures.

Tables 3 and 4 describe the patients classified as having ARDS using the alternative criteria. There is no apparent difference in illness severity or mortality between the two groups.

	DISCUSSION

TOP ABSTRACT INTRODUCTION METHODS RESULTS DISCUSSION REFERENCES

We have shown that in the context of this study, the NAECC and LISS criteria led to similar estimates of ARDS incidence. However, we found that agreement between these two definitions of ARDS, both commonly used in clinical trials, was only moderate (using kappa on the full data set) to substantial (kappa on the data set restricted to observations when compliance was measured, phi on both data sets). We have demonstrated that these differences in patient classifications using NAECC and LISS criteria were not prognostically important. The absence of prognostic differences between patients classified by the two criteria suggests that for most purposes, investigators and clinicians may safely use the NAECC and LISS criteria interchangeably.

In a previous study of alternative ARDS classifications schemes Moss and colleagues compared NAECC and LISS criteria with their own classification scheme (15). These investigators found that 31% of 111 patients at risk for ARDS proved positive by NAECC criteria, whereas 27% of the same 111 patients proved positive by LISS criteria. Moss and colleagues did not make direct comparisons between the NAECC and LISS classifications, but found that both systems lead to an accuracy of greater than 90% using their own definition of ARDS as the gold or reference standard. These findings are consistent with ours.

Strengths of our study include careful and rigorous data collection in the context of a multicenter randomized trial; the independent, duplicate review of chest radiographs following formal training and the subsequent demonstration of moderate to high levels of agreement in these interpretations; sample size adequate for reasonably narrow confidence intervals around our estimates of agreement; and thorough exploration of our data.

One limitation of our study is the possible idiosyncracies of our interpretation of NAECC and LISS criteria. For instance, we interpreted the NAECC criteria as requiring diffuse, bilateral infiltrates. Although the authors of the document implied in their discussion that the infiltrates must be diffuse, this was not absolutely clear from their formal list of criteria (2). Because we believe it is important not to classify patients with subsegmental opacification as having ARDS, we insisted the infiltrates be diffuse. Given limitations in the explicitness of the NAECC statement of their criteria, an alternative interpretation would accept bilateral infiltrates of any character, including subsegmental consolidation, as supportive of a diagnosis of ARDS. More work is needed to standardize the interpretation of chest radiographs for the purpose of diagnosing ARDS. An additional issue that our investigation did not address is the inclusion in the LISS of an assessment of nonpulmonary organ failure. We did not address this issue explicitly because it is not addressed in the NAECC deinition of ARDS, and because a large majority of patients included in our database had nonpulmonary organ dysfunction.

Our population is not completely representative of all patients with ARDS. We excluded patients with underlying malignancies, those with heart failure, and those with very high peak pressures on the ventilator at the time of screening for enrolment. Agreement for scoring systems will always be higher when extreme patients (those who clearly do not have ARDS, or those very sick patients who clearly do have ARDS) represent a large proportion of the population. To the extent that we excluded sicker patients with more obvious ARDS, we would see lower agreement between the scoring systems.

We explored possible sources of disagreement between the two systems. We found that omission of individual components either had minimal impact or appreciably reduced the level of agreement. Thus, no component adds more to disagreement than to agreement, and moving to a more restricted set of components will not improve reliability of classification.

We found that increasing the Pa_O2/FI_O2 NAECC threshold above 200 leads to decreased agreement. We also found deterioration in agreement when we increase the LISS threshold for ARDS to 3.0, or lowered it to 2.0. Thus, we cannot explain disagreement between the systems on the basis of suboptimal choice of thresholds.

Agreement between any two classification schemes may be limited by the reliability of each one. Therefore, separate assessments of our study population using just one definition for ARDS would disagree to the extent that methods of data recording, data entry, and interpretation of chest radiographs are not reproducible. We have demonstrated limitations in the reproducibility of chest radiograph interpretations in the context of a clinical trial in acute lung injury; the limitations can be overcome through standardized training of study investigators (12). Similarly, Rubenfeld and colleagues demonstrated high interobserver variability among a panel of international investigators in this field (16). Interobserver variability in applying the NAECC and/or LISS classifications for ARDS may be an important source of the disagreement we have observed between the two classification schemes in this study, and we continue to recommend more attention to issues related to interobserver variability in the application of future classifications, particularly for the purpose of application in clinical trials.

In the meantime, our findings suggest that the choice of ARDS definition is unlikely to explain differences in apparent incidence of ARDS in different populations. This is reassuring for investigators studying changes in the prognosis of ARDS, or alternative management approaches for patients with ARDS, through evaluations of population outcomes across studies using either the NAECC or LISS definitions. Similarly, clinicians can safely apply to their practice the results from clinical studies or from cross-institution comparisons utilizing these two alternative classification schemes for ARDS.