Published ahead of print on September 14, 2006, doi:10.1164/rccm.200602-177OC
© 2006 American Thoracic Society doi: 10.1164/rccm.200602-177OC
Development and Validation of a Clinical Prediction Rule for Severe Community-acquired PneumoniaService of Pneumology, Department of Emergency Medicine, and Research Unit, Hospital de Galdakao, Galdako; and Basque Foundation for Health Innovation and Research (BIOEF), Sondika, Bizkaia, Spain Correspondence and requests for reprints should be addressed to Pedro P. España, M.D., Service of Pneumology, Hospital de Galdakao, 48960 Galdakao, Bizkaia, Spain. E-mail: pespana{at}hgda.osakidetza.net
Rationale: Objective strategies are needed to improve the diagnosis of severe community-acquired pneumonia in the emergency department setting. Objectives: To develop and validate a clinical prediction rule for identifying patients with severe community-acquired pneumonia, comparing it with other prognostic rules. Methods: Data collected from clinical information and physical examination of 1,057 patients visiting the emergency department of a hospital were used to derive a clinical prediction rule, which was then validated in two different populations: 719 patients from the same center and 1,121 patients from four other hospitals.
Measurements and Main Results: In the multivariate analyses, eight independent predictive factors were correlated with severe community-acquired pneumonia: arterial pH < 7.30, systolic blood pressure < 90 mm Hg, respiratory rate > 30 breaths/min, altered mental status, blood urea nitrogen > 30 mg/dl, oxygen arterial pressure < 54 mm Hg or ratio of arterial oxygen tension to fraction of inspired oxygen < 250 mm Hg, age Conclusions: A simple score using clinical data available at the time of the emergency department visit provides a practical diagnostic decision aid, and predicts the development of severe community-acquired pneumonia.
Key Words: emergency department • prediction rule • severe community-acquired pneumonia
Community-acquired pneumonia (CAP) is a common and potentially serious illness. Management decisions regarding site of care, extent of assessment, and level of treatment are based primarily on disease severity. Severe CAP (SCAP) is a life-threatening condition that requires intensive care. Estimates of the frequency of SCAP range from 5 to 35% (1), with mortality ranging from 20 to 50% (2). These relatively wide ranges indicate disparities between definitions of SCAP. There is no universally accepted definition of SCAP. During the last decade, the term has been used for cases that ultimately result in death (3, 4), and/or patients requiring admission to an intensive care unit (ICU). Such practical definitions seem to be insufficient because the risk of death from CAP is not the same as the need for inpatient care. On the other hand, the decision to admit a patient to the ICU depends on the clinical judgment of individual clinicians and the local practices of their hospitals, differences that could account for much of the variability regarding ICU admission (3–8). Studies focused on the evaluation of patients admitted to the ICU (9–11) mix some variables evident at the time of admission with other potentially evolutionary criteria, which are not applicable to early hospital admission. Other criteria, such as mechanical ventilation and septic shock (11), are less subject to interpretive variability and better reflect illness severity. Identification of patients likely to have a major adverse outcome is a key step in reducing the mortality rate of SCAP. Unfortunately, many patients may not appear severely ill at initial presentation. In these patients and others, suboptimal management in the period between hospital admission and ICU referral has been associated with increased mortality (12). Early identification of the sickest patients or those with rapidly progressing CAP may allow for earlier intervention and thus potentially improve outcomes. Approximately 75% of CAP cases are initially evaluated and treated in hospital emergency departments (EDs) (13), making these centers the ideal location for the establishment of a medical practice guideline to evaluate SCAP. The identification of patients at risk for adverse outcomes can greatly aid clinical decision making in the ED. The main goal of the present study was the development and validation of a prediction rule for SCAP used in the ED. We used hospital mortality, mechanical ventilation, and/or septic shock as dependent variables to define SCAP, then compared the results of applying this prediction rule with those derived from the application of three other validated prediction rules: the modified American Thoracic Society rule (m-ATS) (9), the British Thoracic Society's CURB-65 (confusion, urea, respiratory rate, and blood pressure at age 65 years or older) (3), and the Pneumonia Severity Index (PSI) (4, 14).
Selection Criteria Patients included in the study were nonimmunosuppressed adults aged 18 yr or older with a pulmonary infiltrate on chest X-ray not known to be old and with symptoms consistent with pneumonia, including cough, dyspnea, fever, and/or pleuritic chest pain. (Additional details are provided in the online supplement.) Patients with an expected terminal event, defined as any patient with metastatic cancer, advanced dementia, or a disease or condition with a high likelihood of predicted fatality during the next 30 d, were explicitly included in the study.
Derivation and Internal Validation Cohort
External Validation Cohort An external validation cohort was formed with patients admitted to four other hospitals in the same health network. Patient records were randomly selected and retrospectively analyzed for two periods, March 1998 to March 1999 and March 2000 to September 2001. This cohort had been used in a previous study (6). Sample design and hospital characteristics, as well as additional details regarding identification, evaluation, and exclusion criteria, have been described previously elsewhere (15). (Additional details are available in the online supplement.) Local ethics committees approved the project.
Data Collection A protocol was defined for the collection of data and definition of variables such as septic shock, severe sepsis, and mechanical ventilation. (Additional details are in the online supplement.)
Statistical Analysis
To identify the variables correlated with SCAP, we performed univariate analyses using the We performed logistic regression models to select the variables for our prediction rule. We provide the area under the curve (AUC) of the final model and the odds ratios (OR) and 95% confidence intervals (CI) of all selected variables. No assumptions were made in relation to missing values. We also estimated the same parameters for other prediction rules (m-ATS, CURB-65, and PSI) as applied to the three cohorts. All effects were considered significant at p < 0.05, unless otherwise noted. (Additional details are available in the online supplement.)
A total of 1,940 episodes of CAP diagnosis were recorded in the ED; 1,776 satisfied all criteria for inclusion in the outcome analysis (Figure 1). Of these, 46 episodes were classified as an expected terminal event at the time of diagnosis. A total of 1,057 patients were randomly assigned to the derivation cohort and 719 to the internal validation cohort. The external validation cohort was formed by 1,121 patients admitted to four other hospitals. The sociodemographic features of the derivation, internal validation, and external validation cohorts are described in Table 1. The average age was 64.7 yr (SD, 19.7 yr). Some variables were missing in the external validation cohort, although only respiratory rate, arterial pH, and PaO2/FIO2 < 250 mm Hg showed missing percentages higher than 5%. Patients in the derivation and internal validation cohorts were managed following a clinical guideline, so they did not show any missing values. Pulse oximetry was performed on 4.1% of patients, whereas the rest underwent blood gas analysis. The rate of SCAP among admitted patients was 11.5% in the derivation cohort, 9.8% in the internal validation cohort, and 12% in the external cohort, whereas in-hospital mortality was 9.1, 8.2, and 9.7%, respectively. Of the total number of patients evaluated at the ED, 3% were admitted to the ICU; among inpatients, 4% were admitted to the ED. The sociodemographic characteristics of admitted patients in the derivation and validation cohorts are described in Table E1 in the online supplement.
Table 2 shows the 14 variables analyzed in the derivation cohort associated with SCAP in the univariate analysis. Of those, two were sociodemographic, one was comorbidity, five involved the physical exam, four were analytical, and two were related to X-ray results.
In the multivariate analyses, eight independent predictive factors were correlated with SCAP: systolic blood pressure < 90 mm Hg, arterial pH < 7.30, respiratory rate > 30 breaths/min, blood urea nitrogen (BUN) > 30 mg/dl, oxygen arterial pressure < 54 mm Hg or PaO2/FIO2 < 250 mm Hg, altered mental status, age  80 yr, and multilobar/bilateral lung affectation in X-rays. Although living in a nursing home was also correlated with SCAP (OR, 2.4; 95% CI, 1.1–5.5), it was not included in the predictive model. Results from the multivariate analyses were used to develop a clinical prediction rule. From the  parameter obtained in the multivariate model, a score was assigned to each predictive variable (Table 3). By adding up the points assigned to each predictive variable, a score was given to each patient, with a higher score corresponding to a higher likelihood of SCAP. This enabled us to group patients in different risk categories (Figure 2). The prediction rule showed good discriminating power, as measured by the AUC of 0.92. Applying this prediction rule to the cohorts yielded an AUC of 0.92 for the internal cohort and 0.80 for the external validation cohort. The prediction rule also showed good discriminating power when applied to in-hospital mortality (see Figure E2).
When applying a cutoff point of 10 or greater, our prediction rule showed an AUC of 0.83 for the derivation cohort, 0.86 for the internal validation cohort, and 0.72 for the external validation cohort. Values of sensitivity, specificity, and positive and negative predictive values for our rule and three other predictive rules are shown in Table 4. Both m-ATS and CURB-65 had low sensitivity (51.3 and 68.4%, respectively), whereas PSI risk class IV–V and adjusted PSI demonstrated poor specificity (68.1 and 57.5%) for the derivation cohort, a trend lessened in the validation cohorts. The performance of the different rules was consistent across cohorts.
When applied to the entire prospective cohort (derivation and internal validation cohorts), our prediction rule did not detect eight false negatives out of 1,250 patients. All eight died, although only one patient with a history of congestive heart failure was admitted to the ICU on the third day after admission, requiring mechanical ventilation and vasopressors. Another patient died of complications from lung thromboembolism, diagnosed via computerized axial tomography, without requiring any modification of the antibiotic treatment. Another patient suffered from neurologic degenerative disease and was treated with home-administered noninvasive mechanical ventilation; this patient presented with a pH of 7.35 and a PCO2 of 69 mm Hg at the time of admission. Three patients were between 76 and 80 yrs of age, and fulfilled, in addition, one of the other criteria in the rule (either PO2 or BUN). The other two were 80 yr or older. One of these patients had advanced-stage neoplastic disease and the other presented with three criteria (BUN of 29 mg/dl, PO2 of 59 mm Hg, and respiratory rate of 30 breaths/min, just under the limit of the cutoff for this variable) at the time of diagnosis (see Table E2). Of the 526 patients positively identified as having SCAP by the prediction rule, there were 414 false positives. Fourteen required admission to the ICU for monitoring. The contributors to the false positives were systolic blood pressure < 90 mm Hg (32 cases), arterial pH < 7.30 (21 cases), BUN > 30 mg/dl (204 cases), respiratory rate > 30 breaths/min (121 cases), PaO2/FIO2 < 250 mm Hg (162 cases), altered mental status (102 cases), multilobar/bilateral lung affectation in X-rays (157 cases), and age > 80 yr (262 cases). Of the 414 patients with false-positive results, 82% presented with severe sepsis early in the hospital course, defined as infection plus new-onset acute organ dysfunction using consensus criteria. Among the 80 patients without severe sepsis, the relevant variables were age, BUN, multilobar lung affectation, and respiratory rate. After excluding patients with expected terminal events, who were more likely to receive aggressive treatment, we evaluated the rule for predicting SCAP in the entire prospective cohort. The AUC was 0.92; when a cutoff of 10 or greater was used, the AUC was 0.84.
Our findings suggest that a simple model that uses only eight variables, which are easily accessible and interpretable at the time of ED presentation, can identify seriously ill patients with CAP who are at risk for an adverse outcome and can provide a practical diagnostic decision aid. Using data initially collected in the ED, our rule can identify at admission patients developing SCAP within and beyond the first 24 h. To our knowledge, this is the first such prediction rule that achieves this goal. This rule achieved high sensitivity and high AUC, which were maintained in validation samples. It was derived from a significant number of consecutive patients from the ED of a single hospital and validated in two cohorts with a wide sample of patients, thus suggesting that such a rule could be applied with some confidence in medical practice while undergoing future prospective evaluations. The model has a higher discriminatory power for predicting SCAP than do the m-ATS, CURB-65, and PSI. Instead of using ICU admission, which is highly variable according to individual ICU admission practices (3–8), as a criterion for determining SCAP, we considered in-hospital death, mechanical ventilation, and/or septic shock as endpoints, given their more objective nature as variables (11). The main value of this prediction rule is its ability to identify patients who need additional monitoring and more aggressive treatment after the first evaluation in the ED either at the ICU or other alternative settings, such as intermediate care units or a specialized regular ward, depending on the severity of the patient's condition. Multiple logistic regression results identified eight independent predictive factors that correlated with SCAP. As stated earlier, the "nursing home" variable, although significantly associated with SCAP, was excluded from the prediction rule because cultural and social differences regarding the use of nursing homes could affect the rule's generalizability. The strongest clinical predictors of SCAP were pH < 7.30 and systolic pressure < 90 mm Hg. A depressed pH, which is likely a side effect of metabolic acidosis derived from sepsis (17), is not included in other prediction rules such as CURB-65 or m-ATS (3, 9). On the other hand, it is given the highest score in the PSI (4) and also appears as a clinical characteristic associated with mortality in patients with SCAP admitted to the ICU (18). All of the variables used to create our score are included in the PSI, except for multilobar/bilateral lung affectation in X-rays, which has recently been related to treatment failure and poor prognosis (7). Our rule presents a simplified and reweighted PSI with a different cutoff for pH, PO2, and age. Dealing with only eight variables instead of 20 offers greater simplicity and applicability. The covariates in our rule are readily obtainable in the normal course of a patient's ED work-up. An overall score was computed from the scores assigned to each of the eight predictive variables. It must be noted that having eight variables with different scores makes memorization difficult. For this reason, the score may be used as a continuous variable or with a cutoff of 10 or greater, with easy-to-use major and minor criteria that do not require any tool. Thus, our score includes variables present in other well-known prediction rules, such as the CURB and PSI, and recalculates the relative weight of these variables and comes up with a simple, easy-to-memorize algorithm that can easily identify patients at risk of an adverse evolution. Our rule is less complicated than the PSI; it adds variables to the CURB-65 that have been demonstrated to improve the ability to predict severity; and it is more useful than the m-ATS for patients in the ED. It was not our goal to prove or disprove the validity of these rules when they are applied in a way consistent with their original purposes. The variables we used are also included in the CURB-65, although this rule uses an age cutoff of 65 yr, which is lower than our cutoff of 80 yr. In addition, we included three more variables—pH, PaO2/FIO2, and multilobar/bilateral lung affectation in X-rays—that the authors of the CURB-65 had pointed out would likely be necessary to consider (19). Hypoxemia has been associated with impending respiratory failure, subsequent ICU admission (4), and mortality (20) in patients with CAP, reflecting the severity of primary organ impairment in this illness. Furthermore, the identification of arterial hypoxemia has direct treatment implications, which include the delivery of supplemental oxygen, ventilatory support (21), as well as hospitalization for more intensive clinical observation. Arterial blood gas provides useful information about arterial pH and PaO2, although in patients with hemodynamic stability, it could be avoided with a pulse oximetry finding of greater than 90%. With respect to the m-ATS criteria, our score shares some variables with this system. In our score, however, the two major m-ATS criteria, septic shock and mechanical ventilation, were defined as endpoints because they are potentially evolutionary criteria. Consequently, they are not applicable to early hospital admission.
The variables can be grouped into major (pH < 7.30 and systolic pressure < 90 mm Hg) and minor (respiratory rate
At a cutoff of 10 or greater, the prediction rule showed 92.1% sensitivity and 73.8% specificity, which proved superior when compared with the application of m-ATS or CURB-65 with three or more factors, although it showed less sensitivity than PSI classes IV and V and the adjusted PSI. The specificity findings suggest that our rule has the potential to identify a larger proportion of patients as low risk than the PSI. We chose a cutoff of 10 or more because it presented good sensitivity while showing the best balance between sensitivity and specificity. The low sensitivity of m-ATS is probably due to the use of data available at initial evaluation. On the other hand, the low sensitivity of CURB-65 may be due to the fact that the class with two factors had a 7% mortality rate and that the cutoff for this rule is established with three or more factors. Due to the substantial proportion of outpatients included in our study, the prevalence of SCAP in the prospective cohort was lower than in the external derivation cohort and in other studies primarily focused on inpatients (3, 5, 8–11). This explains why the negative predictive values for all prediction rules were uniformly high, whereas the positive predictive values were generally low in our study cohort. The low positive predictive value of the rules often failed to select the patients at risk for subsequent SCAP. Therefore, its best use as a stratification tool may be as an adjunct to clinical judgment. The negative predictive value was excellent for all rules, which means that all patients who do not fulfill the criteria could be safely managed without intensive care. Approximately 80% of the patients with false-positive results presented with severe sepsis. Severe sepsis is common in CAP, occurring in approximately one-half of all hospitalized patients with CAP and early in the course of the hospital stay (22). It has been shown that early intervention in subjects with incipient severe sepsis may improve outcome (23). Our rule did not detect a small population of eight patients who developed SCAP, only one of whom required admission to the ICU. This patient was admitted with associated comorbidity and evolved to shock and respiratory failure on the third day after admission, thus underscoring the importance of evolution during the first hours of acute-onset diseases such as CAP. Given that pneumonia is a dynamic process, a small yet important number of patients who might not fulfill severity criteria on hospital admission may nevertheless be at risk of developing SCAP in the following days. In connection with this, Ewig and colleagues (5) observed that the m-ATS rule predicted 100% of ICU admissions later in the course of hospitalization (within 48–72 h). Although the aforementioned prediction rules currently established by scientific societies (24–28), such as CURB-65 and PSI, adequately predict the risk of death and therefore the decision for hospital admission, they do not specify the level of care most likely required. In an effort to fill that gap, our study focused on the identification of patients whose disease can suddenly take a turn for the worse, and thus who may require additional monitoring and care (hemodynamic monitoring, ventilatory support, and nursing observation). For that reason, despite the high number of false positives, our rule is adequate for identifying seriously ill patients with CAP at the time of ED evaluation who are at risk for an adverse outcome, which is a more efficient tool from a clinical point of view. It is important to increase the degree of awareness about signs of severity in CAP, because it has been demonstrated that treatment for patients with obvious signs of severity, such as confusion and hypotension, often does not measure up to treatment guidelines (29). Exclusion from the derivation and internal validation cohorts of the 46 patients considered to be having terminal events maintained the model's predictive capability. It is difficult to predict the terminal phase of many diseases (30), which is why these patients were not excluded initially. In either case, this rule is valid for identifying all at-risk patients in the ED as well as those suitable for more aggressive treatment. This study has four main limitations:
The clinical heterogeneity of CAP is a global limitation when attempting to create a scoring system capable of separating patients into appropriate management groups. This limitation compels the notion that clinical judgment, which is difficult to define in objective terms, must remain an indispensable part of decision making. Our proposed score aims to identify as early as possible patients at increased risk of complicated CAP evolution. The use of this score in EDs may help in triage decisions. It may also be used to design future studies and health care strategies aimed at reducing CAP mortality. We suggest that patients scored as high risk (score In summary, a prediction rule based on eight variables that are easily accessible at the time of ED admission is able to identify patients whose CAP may substantially worsen. The similarity of these prognostic variables to those of previously published models suggests that the results could be generalized to other institutions. The value of this rule is supported by the small number of false negatives and the fact that it can be used in any type of population visiting an ED.
The authors appreciate the support of the Direction of Osakidetza, staff members of different services, and the medical record sections of the participating hospitals. They also thank Mr. Patrick J. Skerrett for his assistance in the editing work of the manuscript and Dr. A. Torres for the review of the manuscript.
This article has an online supplement, which is accessible from this issue's table of contents at www.atsjournals.org Originally Published in Press as DOI: 10.1164/rccm.200602-177OC on September 14, 2006 Conflict of Interest Statement: None of the authors has a financial relationship with a commercial entity that has an interest in the subject of this manuscript. Received in original form February 7, 2006; accepted in final form September 12, 2006
This article has been cited by other articles:
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
TOP
ABSTRACT
AT A GLANCE COMMENTARY
2 and Fisher's exact tests were used to test for statistical significance among proportions. For continuous variables, the Student t test or analysis of variance was performed when appropriate in the univariate analysis of the three cohorts. 
