Published ahead of print on January 26, 2006, doi:10.1164/rccm.200508-1338PP
© 2006 American Thoracic Society doi: 10.1164/rccm.200508-1338PP
Statistical Treatment of Exacerbations in Therapeutic Trials of Chronic Obstructive Pulmonary DiseaseDivision of Clinical Epidemiology, Royal Victoria Hospital; McGill University Health Centre; and Departments of Epidemiology and Biostatistics and Medicine, McGill University, Montreal, Canada Correspondence and request for reprints should be addressed to Samy Suissa, Ph.D., Division of Clinical Epidemiology, Royal Victoria Hospital, 687 Pine Avenue West, Ross 4.29, Montreal, PQ, H3A 1A1 Canada. E-mail: samy.suissa{at}clinepi.mcgill.ca ABSTRACT Randomized trials and a meta-analysis suggesting that inhaled corticosteroids reduce exacerbation rates in patients with chronic obstructive pulmonary disease (COPD) show major discrepancies that may be due to different approaches to data analysis. These trials used statistical techniques that were either weighted or unweighted for follow-up time, with p values and confidence intervals estimated with or without accounting for between-patient variability in exacerbation rates. We illustrate the validity of these methods using data from a cohort of 5,454 patients with COPD structured to emulate a randomized trial. The "reference" group was defined as patients with a history of exacerbations before cohort entry (n = 1,137), whereas the "treated" group included an equal number (n = 1,137) of patients with no prior exacerbation. Random samples of 100 and 200 subjects were selected three times from each of two groups to further illustrate the variability in the findings. Exacerbations during follow-up were identified from prescriptions for systemic antibiotics. The correct rate ratio of 0.75 estimated by the weighted approach was underestimated as 0.57 by the unweighted approach. When the weighted approach did not, however, also account for between-patient variability, the p value was greatly underestimated (e.g., rate ratio, 0.79; p = 0.0007 instead of p = 0.12) and confidence intervals were much narrower than after properly accounting for this variability. In conclusion, the reports from randomized trials and the meta-analysis that inhaled corticosteroids reduce COPD exacerbation rates are the result of improper statistical analysis techniques. The only two studies that used the correct statistical approach found insignificant effects with these drugs.
Key Words: biases • data analysis • drug effectiveness • inhaled corticosteroids • methods The role of inhaled corticosteroids in the treatment of chronic obstructive pulmonary disease (COPD) is highly debated (1, 2). Oddly, while inhaled corticosteroids seem to have only minor effects on lung function, they appear to significantly reduce COPD exacerbation rates in several trials (3–6). A meta-analysis of several major trials reported a 30% reduction in the rate of COPD exacerbations with inhaled corticosteroids, with impressive statistical significance (rate ratio, 0.70; 95% confidence interval [CI], 0.58–0.84) (7). Surprisingly, this meta-analysis also suggested that, for some studies, the difference in the rate of exacerbation between inhaled corticosteroids and placebo had been statistically significant, whereas the individual studies had in fact originally reported that these differences were nonsignificant. For example, the Lung Health Study reported a nonsignificant p value of 0.07 for exacerbation rates, defined as hospitalizations for respiratory conditions. For this same outcome, the meta-analysis calculated a 95% CI of 0.26 to 0.80 for this study, which corresponds to a highly significant p value of 0.005 (4). Such contradictions in estimation from the same data were not discussed. A review of the basic principles behind the statistical analysis of exacerbation rates is therefore necessary to clarify these discrepancies. This article reviews the diverse approaches to data analysis used in the randomized trials of inhaled corticosteroids in the treatment of COPD and their meta-analysis. We then use data from a cohort of patients with COPD, structured to emulate a randomized controlled trial, to illustrate the impact of these methods on the bias and precision of the estimates. APPROACHES TO DATA ANALYSIS The rate of exacerbation is intended to measure the probability that a patient who is currently exacerbation-free will experience an exacerbation within a small time interval (e.g., in the next day). In the published randomized trials of inhaled corticosteroids in COPD, the mean rate of exacerbation for a group of patients was estimated in two ways and four different approaches to compute p values and CIs were used. The first approach was to initially estimate the rate for each patient individually by dividing each patient's number of exacerbations during follow-up by the length of time each patient was monitored. The mean rate for the group was then estimated by taking the average, or in some cases the median, of these individual patient rates (3–5). This approach, based on the simple average of individual rates, is called unweighted since each patient contributes equally to the group mean irrespective of his or her length of follow-up time. With this unweighted approach, the mean rates and rate ratios are thus estimated directly and compared between groups receiving inhaled corticosteroids or not, using a t test or linear regression methods. Some studies estimated median rates and rate ratios and therefore used the Wilcoxon test, which compares ranks of the rates from highest to lowest, instead of the actual numeric values as would a t test. For some studies, the unweighted approach was used, reporting only the numbers of events in each group and not the actual rates (6, 8, 9). The second approach was to estimate the mean rate directly by pooling all patients in a treatment group and dividing the total number of exacerbations in the pooled group by the total follow-up time of the pooled group (10–12). This approach is called weighted since, by dividing the total number of exacerbations by the total duration of person-time of follow-up of the group, the resulting mean rate is equivalent to weighing each patient's individual rate by their follow-up time. This approach produces the correct and best estimate, or, in statistical terms, the maximum likelihood estimate, of the rate since it is unbiased for the probability of an exacerbation in a small time interval (e.g., 1 day) (13). For this weighted approach, the CI and p value of the rate ratio were estimated in two different ways. First, a Poisson distribution was assumed for the total number of exacerbations in each group. Such a distribution correctly assumes that exacerbations can occur repeatedly but that they happen randomly and independently of each other over the follow-up time. The rate ratio of exacerbation between the treatment groups is then estimated using a Poisson regression model, which requires two modifications. First, a logarithmic link is introduced, which is equivalent to taking the logarithm of the rate instead of the rate itself, as the dependent variable in a regression analysis. This link must be used to provide the rate ratio instead of the rate difference. Second, to account for the different lengths of follow-up, the Poisson regression model uses what is called an "offset," the logarithm of the follow-up time. These modifications are standard defaults in any statistical package that carries out Poisson regression analyses. This default Poisson regression technique, however, assumes that all patients are homogeneous with respect to their rate of exacerbation. It does not recognize that for some diseases, such as COPD, there may be more patients than expected who have very frequent exacerbations in 1 year and, at the same time, many more patients who may have no exacerbations. The second approach to estimating the CI and p value of the rate ratio recognizes this greater than expected heterogeneity in exacerbation rates between patients. To account for this important "extra-Poisson" variability in the rate of exacerbation between patients, the same Poisson regression approach can be extended: the number of exacerbations within a patient continues to be assumed to follow the Poisson distribution, but now the variability in the exacerbation rate between patients is estimated and accounted for using an overdispersion parameter (14). This overdispersion parameter is essentially a standard deviation–like measure of the additional differences in exacerbation rates between subjects, above and beyond that expected by the Poisson distribution alone. As a result, the p value and CI surrounding the rate ratio of exacerbation will be based on the sum of two variance components, namely the within- and between-subject variabilities, rather than only on the within-subject variability that is used with the standard default Poisson approach with no overdispersion parameter. When in doubt about the possibility of such heterogeneity in exacerbation rates, it is advisable to always use this general approach first and thus evaluate the significance of the overdispersion parameter on the results. ILLUSTRATION We did not have access to data from the randomized trials to illustrate these various approaches. We thus used a cohort of patients with COPD previously analyzed to describe patterns of rates of exacerbation (15), and structured it in a way to emulate data from a randomized trial. Briefly, this cohort was identified using the computerized databases of Saskatchewan Health and included all subjects at least 55 years of age who were newly treated for COPD during 1990–1997. All subjects were monitored until the date of death, the date of emigration or end of coverage, or December 31, 1999, whichever occurred first. The cohort was restricted to patients with at least 30 days of follow-up. Exacerbations, defined by a prescription for systemic antibiotics with or without an oral corticosteroid, occurring during follow-up were identified. Exacerbations occurring within 30 days of a previous exacerbation were considered as a relapse and not a new exacerbation. This database should be regarded purely as illustrative of the statistical issues addressed in the present article rather than for clinical purposes. To emulate the structure of a randomized trial, we divided the cohort into two groups defined by the most important baseline risk factor for exacerbation we could find—namely, the presence or absence of an exacerbation during the year before cohort entry (16). We thus defined the "reference" group as all patients with exacerbations before cohort entry (n = 1,137). This group was selected since it is expected to have an elevated rate of exacerbations during follow-up (16). The "treated" group of this pseudorandomized trial was made up of an equal number (n = 1,137) of patients randomly selected from the 4,317 patients with no prior exacerbations in the cohort. This exposure classification was selected to achieve rate ratio estimates of "treated" versus "reference" of less than 1, in line with the estimates in the majority of trials under scrutiny. We call this dataset with 1,137 patients per group the "large trial." To resemble the sizes of typical randomized trials, we formed "small trials" by randomly sampling 100 and 200 subjects from each of the two groups from the large trial and repeated this sampling three times to observe various scenarios in the sources of bias. For each of the sampled groups of 100 or 200 patients, we used both the weighted and unweighted mean (and median) approaches to estimate the mean rates of exacerbation and the rate ratio for the treated versus reference groups. For the unweighted approach, the mean rates were compared using an approach equivalent to a t test or simple linear regression, after a logarithmic transformation of the rates. This approach involved a generalized linear model with a logarithmic link function (this function is required to obtain a relative risk rather than a risk difference between groups) and assumes a normal distribution. This approach allows one to circumvent the computational problem caused by the undefined logarithm of zero rates. This model was also used to estimate the 95% confidence limits around the rate ratio. The median rates and rate ratios were also estimated, and the Wilcoxon test was used to compare these rates. The 95% confidence limits for the median rate ratio were approximated by the test-based approach applied by inverting the p value of the Wilcoxon test (17). For the weighted approach, the rate ratio and its confidence limits were estimated using a Poisson regression model with a logarithmic link function (to obtain the rate ratio or relative risk), offset by the logarithm of the follow-up time (to account for the varying follow-up times between subjects). This approach was used with and without an overdispersion parameter that accounts for between-patient variability in the frequency of exacerbations (14). RESULTS OF ILLUSTRATION The patterns of follow-up and exacerbations for the large trial of 1,137 patients per group are described in Table 1. It indicates that the mean follow-up time for the treated group is longer (3.6 years per patient) than for the reference group (3.0 years), with variations ranging widely from 1 month to 10 years. On the other hand, the number of exacerbations per patient is lower in the treated group (mean, 4.1) than for the reference group (mean, 4.7), with variations from 0 to 45 exacerbations per patient. Finally, the rates of exacerbation per patient are also seen to vary widely from 0 to 121.75, with very different means and medians. Table 2 shows similar patterns observed for the six small trials selected from this large trial.
Table 3 displays the rates and rate ratios as estimated by the weighted mean, unweighted mean, and unweighted median approaches. For the large trial, this analysis shows that the correct weighted approach produces a rate ratio of 0.75, which is highly underestimated by the unweighted mean rate ratio (0.57) and the unweighted median rate ratio (0.63). Thus, the unweighted approaches appear to tend to exaggerate the benefit of treatment. These observations are consistent across the six samples. This table also shows that the unweighted mean rates overestimate the correct rates, whereas unweighted median rates provide underestimates of the true value.
The statistical significance of the rate ratios of exacerbation for the six samples was assessed by the p value and CIs (Table 4). These suggest that, among the unweighted approaches, the analysis of the mean produces, in most instances, higher p values and wider CIs than that of the median. These p values and CIs are also quite different than those found with the weighted methods. Among the weighted approaches, the Poisson regression analysis that does not account for the between-patient variability systematically produces much lower p values and tighter CIs than the proper analysis that includes the overdispersion term accounting for this variability.
DISCUSSION The randomized trials conducted to assess the effectiveness of inhaled corticosteroids at reducing the rate of COPD exacerbation have been using different methods of data analysis, leading to widely discrepant results, where often a rate ratio can be highly significant by one approach and not significant by another. We showed that these discrepancies are the result of the use of inappropriate methods to estimate the rate of exacerbation and its p value and CI. This situation is general and applies not only to inhaled corticosteroids in COPD, but equally well to all diseases and trials that involve a recurrent or intermittent outcome during follow-up, such as exacerbations, hospitalizations, physician visits, prescriptions, and so forth. The unweighted approach, by weighing each patient equally, is sensitive to follow-up times that can vary within a study. In the Inhaled Steroids in Obstructive Lung Disease in Europe (ISOLDE) study, for example, 43% of the inhaled corticosteroid group and 53% of the placebo group did not remain for the entire 3-year follow-up, and more than 20% of the patients were followed up for less than 1 year. With such variability in the patient follow-up, the unweighted approach can produce a biased estimate of the mean rate and consequently the rate ratio. The weighted approach, on the other hand, adjusts for this asymmetry by accounting for each patient's follow-up time. Only when the follow-up times are equal for all patients, which is rare in such trials, will the weighted and unweighted approaches produce equal mean exacerbation rates. In our large pseudo–COPD trial, the weighted rate of exacerbation in the reference group was 1.55 per patient per year, compared with the unweighted rate of 4.18. This discrepancy is the result of the follow-up times varying from 1 month to 10 years and the number of exacerbations per patient varying from 0 to 45 during these follow-up times. To avoid such discrepancies, the unweighted approach should never be used. Moreover, because the unweighted approach gives rise to biased estimates, all estimates of the variance used to determine the statistical significance and confidence limits of the rate ratio are necessarily also biased. Thus, simple techniques for continuous data, such as standard deviation, t test, Wilcoxon test, or linear regression, with these unweighted rates will inherently also be subject to bias and should not be used. The weighted approach should always be the estimation technique of choice (13). Choosing the correct variance of the rate ratio from the weighted approach is, however, essential. The variance estimation must account not only for the fluctuations of the exacerbations over time within a subject but also for the additional variability in exacerbation rates that occur between subjects. Failing to do so, the resulting p values and confidence limits can be severely underestimated. In one of our illustrations, we found that the same weighted rate ratio of exacerbation of 0.79 had a p value of 0.0007 (95% CI, 0.69–0.91) when not accounting for the between-subject variation and a p value of 0.12 (95% CI, 0.59–1.06) when the between-subject variation was taken into account. This additional source of variation is particularly important in COPD since it is well known that patients are highly variable with respect to their rate of exacerbation; assuming that patients are homogeneous leads to inappropriate claims of statistical significance. One exception is when the number of exacerbations is small, such as the trial that reported six events in 79 patients (8), in which case the between-subject variability is negligible and need not be accounted for. It should also be noted that all these methods assume that patients who withdraw early or are lost to follow-up in a trial do so independently of the drug group they were assigned to and unrelated to disease severity. If this is not the case, selection bias will result. The data analysis issue we identified and discuss in the current article remains even in the absence of such nonrandom withdrawal. The meta-analysis of randomized trials found an overall 30% reduction in the rate of exacerbations with inhaled corticosteroids and this was highly statistically significant (rate ratio, 0.70; 95% CI, 0.58–0.84). For several of the trials that reported unweighted estimates, biased estimates of the rates and rate ratios were the only ones provided in the original studies. For all studies, however, the meta-analysis systematically used variance estimates of the rates and rate ratios that did not account for the between-subject variation. This explains the intriguing anomalies such as individual studies reporting "nonsignificant" findings, whereas the meta-analysis computes and uses statistically significant versions of the CI. The most dramatic example is the ISOLDE study, which reported a treatment difference in the median rate of exacerbation of –0.3 per year (95% CI, –0.4 to 0.0) (3). Using the placebo rate of 0.99 per patient per year translates this difference into an approximate rate ratio of 0.7 (95% CI, 0.6–1.0). This CI is very different from the corresponding computed values used in the meta-analysis (rate ratio, 0.67; 95% CI, 0.63–0.71) (7). Such inconsistency between individual studies and the meta-analysis of these studies, as well as the invalid estimates in several of the original trials, puts in question the findings of the meta-analysis. Of all randomized trials conducted to date to assess the effectiveness of inhaled corticosteroids on COPD exacerbations, some have used the biased unweighted approaches (3–5), whereas others used the proper weighted approach but either did not account for the between-subject variation, thus producing artificial statistical significance (10), or were unclear about the actual variance estimation technique used (6, 8, 9). It would be rather straightforward for the authors of these trials to apply the proper technique of data analysis. Only two randomized trials used the proper weighted analysis that accounted for both the within- and between-subject variation (11, 12). These studies found nonsignificant rate ratios of exacerbation with inhaled corticosteroids relative to placebo of 0.85 (95% CI, 0.66–1.10) and of 0.89 (p value of 0.308; 95% CI not reported but can be approximated to be 0.71–1.11 from p value), respectively (11, 12). With COPD being one of the major causes of morbidity and mortality worldwide, the proper statistical analysis of important clinical outcomes, such as exacerbations in randomized therapeutic trials, is critical. For inhaled corticosteroids, the estimates of effectiveness were biased by the incorrect approach to data analysis for the large majority of trials and for a meta-analysis that suggested significant benefit of these drugs in reducing exacerbation rates. Until randomized trials are conducted with exacerbations as a primary outcome and with the statistical analysis that properly accounts for all sources of variability, the effectiveness of inhaled corticosteroids at reducing exacerbations in COPD remains uncertain. Acknowledgments The author thanks Dr. Pierre Ernst for his invaluable comments. FOOTNOTES Supported by grants from the Canadian Institutes of Health Research (CIHR) and Fonds de la Recherche en Santé du Québec. S.S. is the recipient of a Distinguished Investigator award from the CIHR. The databases were acquired thanks to grants from AstraZeneca, Boehringer-Ingelheim, and GlaxoSmithKline. This study uses data provided by the Saskatchewan Department of Health. The interpretation and conclusions contained herein do not necessarily represent those of the government of Saskatchewan or the Saskatchewan Department of Health. Originally Published in Press as DOI: 10.1164/rccm.200508-1338PP on January 26, 2006 Conflict of Interest Statement: S.S. has been reimbursed for attending several conferences and also participated as a speaker in scientific meetings financed by various pharmaceutical companies (Schering-Plough, AstraZeneca, and GlaxoSmithKline) and has received funding for research grants from AstraZeneca ($89,000), from Schering-Plough ($80,000), and from GlaxoSmithKline ($159,000). Received in original form August 29, 2005; accepted in final form January 25, 2006 REFERENCES
This article has been cited by other articles:
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
