THE DISCRIMINATORY CAPACITY OF THE BRONCHODILATOR RESPONSE

To the Editor :

Nielsen and Bisgaard concluded that in children, the specific airway resistance (sR_aw) discriminates best between asthmatic and healthy subjects, because the sensitivity and specificity of that parameter was highest (1). However, that conclusion could be challenged on statistical grounds.

Table 3 in the Nielsen and Bisgaard paper lists the sensitivity and specificity of all the parameters examined, together with the positive/negative predictive value. Based on the combination of the highest sensitivity/specificity, indeed the order is as listed by Nielsen and Bisgaard. Based on the combination of the highest positive/negative predictive value, the ranking would be different. The latter serves to indicate the instability of the conclusions, which is caused by the fact that the authors apparently used the point estimates of the sensitivity/specificity to rank the tests. These estimates are random variables and are always imprecise to a certain degree: in other samples these estimates will be (slightly) different and so the ranking of the tests may change. Point estimates will vary, and one has to decide whether differences are due to chance or truly exist.

The authors seem not to test whether the discriminatory power of the parameters under question differs significantly and base their conclusions on the point estimates only. They generated receiver operating characteristics curves, but they do not list an important characteristic, i.e., the area under the curve (AUC). The latter is a widely accepted measure of the discriminatory power of a test (an AUC of 0.5 indicates no power at all, whereas an AUC of 1 is equivalent to a perfect separation). The differences between AUCs can be tested for significance: Hanley designed methods to do so (both for the parallel and cross-over situation [2]). Such an evaluation gives a much better insight into the ranking of the tests than a comparison of point estimates.

Universitair Medisch Centrum, Utrecht, The Netherlands

3. Nielsen KG, Bisgaard H. Discriminative capacity of bronchodilator response measured with three different lung function techniques in asthmatic and healthy children aged 2 to 5 years. Am J Respir Crit Care Med 2001; 164: 554-559 [Abstract/Free Full Text].

4. Hanley JA, McNeil BJ. A method of comparing the areas under receiver operating characteristics curves derived from the same case. Radiology 1983; 148: 839-843 [Abstract/Free Full Text].

From the Authors:

We appreciate the comments by Dr. Zanen regarding our interpretation of receiver operating characteristics (ROC) curves in our recent study on the bronchodilator response measured by different lung function techniques in preschool children (1). The ROC curve was constructed with a view to choose the single best cut-off level for each of the lung function tests. As noted by Altman (2), "the best cut-off is that which maximizes the sum of the sensitivity and specificity, which is the point nearest the top left-hand corner." Statistical analysis of the ranking of the tests would be desirable. However, there is no general agreement among statisticians as to the method of such an analysis. Area under curve analysis of the ROC curves, however widely used, is only a surrogate measure with no proper operational meaning in the context of diagnostic sensitivity and specificity and may even be misleading, as discussed by Hilden (3). Therefore, we chose the more conservative approach using the best cut-off level based on a single point on each of the ROC curves.

Copenhagen University Hospital, Copenhagen, Denmark

1. Nielsen KG, Bisgaard H. Discriminative capacity of bronchodilator response measured with three different lung function techniques in asthmatic and healthy children aged 2 to 5 years. Am J Respir Crit Care Med 2001; 164: 554-559 .

2. Altman DG. Practical statistics for medical research. London: Chapman & Hall; 1997.

3. Hilden J. The area under the ROC curve and its competitors. Med Decis Making 1991; 11: 95-101 .