INTRODUCTION

TOP ABSTRACT INTRODUCTION METHODS RESULTS DISCUSSION REFERENCES

Keywords: obstructive sleep apnea; risk stratification; polysomnography; nocturnal pulse oximetry; questionnaire

Identifying obstructive sleep apnea (OSA) syndrome is important because it leads to adverse outcomes if left untreated (1), and because it improves with treatment (5). Although in-laboratory polysomnography (PSG) is an accurate diagnostic tool, it requires significant expertise and expense (6) and is often associated with long waiting periods (7).

The use of PSG in a sleep clinic population would be optimized if a risk stratification strategy were developed that achieves the following two goals. First, it identifies patients whose risk for sleep apnea is so low that PSG is not warranted. Second, it identifies the subset of patients who are at risk for severe OSA (8), who should be prioritized for early testing. To fulfill these two objectives, as well as to maintain accuracy while containing cost compared with gold standard tests, we developed a model that relies on a two-stage method of identifying cases. We based our model on existing prototypes for other conditions (9). The two component tests that we chose for our algorithm were the Multivariable Apnea Prediction (MAP) questionnaire (16) and nocturnal pulse oximetry (nPO) (5, 17). The objectives of this two-stage algorithm are to avoid doing sleep studies on patients at low risk for sleep apnea, and to prioritize scheduling of patients who are likely to have severe disease. We tested the hypothesis that our strategy could meet these two objectives in a sleep center population.

	METHODS

TOP ABSTRACT INTRODUCTION METHODS RESULTS DISCUSSION REFERENCES

Subjects

We selected subjects from patients presenting to our sleep center for evaluation for possible sleep apnea. All completed the MAP questionnaire (16), plus a full-night sleep study (PSG) with concurrent oximetry. We excluded subjects who were already diagnosed with OSA or obesity-hypoventilation syndrome or used supplemental oxygen.

Polysomnography

Sleep studies were completed and scored by polysomnographic technologists at our sleep center according to the method of Rechtschaffen and Kales (21). Twelve-channel recordings were done on all patients that included monitoring of the electroencephalogram, eye movements, tibialis anterior and chin electromyography, respiratory effort, snoring, airflow, and oximetry. The American Sleep Disorders Association (ASDA) definition was used to score arousals during sleep (22). The respiratory disturbance or apnea-hypopnea index (RDI, AHI) was defined as the number of apneas plus hypopneas divided by the total sleep time in hours. An apnea was defined as complete cessation of airflow for at least 10 s and a hypopnea was defined

by a  50% reduction in airflow for at least 10 s, associated with a  4% fall in oxyhemoglobin saturation or an arousal. We judged OSA to be present if the RDI was  5 events/h, and severe OSA to be present if the RDI was  30 events/h (23).

MAP, Oximetry, and PSG Scoring

As described previously (16), we determined each subject's MAP score, which predicts apnea risk using a score between 0 and 1, with 0 representing low risk and 1 representing high risk. A single observer then scored the oximetry strips without knowledge of the PSG results. The oximetry desaturation index (ODI) was the number of desaturations divided by the total test time. We obtained desaturation indices using both 3% and 4% drops, that is, ODI3 and ODI4, respectively. On PSG, we judged OSA to be present if the respiratory disturbance index (RDI) was  5 events/h (case 1) and severe OSA to be present if the RDI was  30 events/h (case 2) (23).

Discriminatory Power of MAP and nPO

Using ROCKIT software (24), we performed "area under the curve" (AUC) analysis for receiver operating characteristic (ROC) curves (25) to determine relative discriminatory power of the MAP, ODI3, and ODI4 (26) and to determine their sensitivity and specificity (27).

Description of the Two-stage Algorithm

We applied our algorithm retrospectively on all included subjects. We used a predefined upper bound (UB) value and lower bound (LB) value for the MAP score to separate the subjects into three groups. Those who had high MAP scores (MAP > UB) were predicted to have OSA and had subsequent review of PSG to see if this were so. Those with low MAP scores (MAP < LB) were predicted to be free of OSA and would not be further studied. Those with MAP scores between UB and LB underwent scoring of nocturnal pulse oximetry, and the ODI was compared against a predefined threshold (ODI_threshold) desaturation index. Those with ODI  ODI_threshold were predicted to have OSA and hence would undergo PSG evaluation, whereas those with ODI < ODI_threshold were predicted to be free of OSA.

Sensitivity and Specificity

Using Statistical Application Software (SAS) programming (Cary, NC), we computed our algorithm's sensitivity and specificity against PSG for each of 125 combinations of UB, LB, and ODI_threshold (i.e., 125 "parameter sets") for case 1 and for case 2 separately (23).

Determination of the Optimal Parameter Sets (28, 29)

We first selected a random sample of 80% of the clinic population, the "estimation sample." We generated 200 bootstrap resamples from this estimation sample. To the 125 possible parameter sets associated with each bootstrap resample, we applied our criterion function to select the optimum parameter set (or sets). We then averaged all the optima from the 200 resamples to find what we called the unbiased estimate of the optimum parameter set. We did this for case 1 and for case 2 separately.

Model Validation

We applied our algorithm, using the two unbiased optimal parameter sets, to our reserved "validation sample" of 20% of our original clinic population. We computed sensitivity, specificity, and positive and negative predictive values of our two models separately, that is, for predicting RDI  5/h and RDI  30/h.

	RESULTS

TOP ABSTRACT INTRODUCTION METHODS RESULTS DISCUSSION REFERENCES

Descriptive Results

Of 421 patients considered for the study, 359 met the inclusion criteria. We excluded 62 subjects46 because they had been ordered to have "split-night" studies by their referring physician, 12 because they were previously diagnosed with OSA, 3 because they were already receiving supplemental oxygen, and 1 because of an incomplete MAP questionnaire. The final group consisted of 243 (67.7%) men and 116 (32.3%) women. Mean age ± SD was 47.2 ± 13.2 yr. The racial composition was 72.2% white, 24.2% black, and 3.6% other. The average body mass index (BMI) ± SD was 32.4 ± 8.8 kg/m². The MAP frequency distribution (Figure 1) shows a high prevalence of clinic patients with MAP scores exceeding 0.6. The mean ± SD of the RDI scores was 25.9 ± 29.5 events/h, with median 12 events/h, and range 0 to 110 events/h. Using RDI  5/h,  15/h, and  30/h as the diagnostic criterion, the prevalence of OSA in this sleep center population was 69.4%, 47.1%, and 33%, respectively.

View larger version (19K): [in this window] [in a new window]   Figure 1.   Interval-specific MAP frequencies of sleep center population in each of the 10 MAP deciles. The distribution is right skewed, with a heavy proportion of the sleep clinic population having MAP scores exceeding a value of 0.6.

Determination of the Discriminatory Power of MAP and nPO

We generated a receiver operating characteristic curve using MAP values against RDI  5/h (Figure 2, upper panel ) and RDI  30/h (Figure 2, lower panel ). For RDI  5/h, the respective values of AUC (± SE) for MAP, ODI3, and ODI4 were 0.834 (± 0.024), 0.947 (± 0.011), and 0.936 (± 0.012). For RDI  30/h, AUC for ODI4 was 0.98 (± 0.008), showing slightly improved discrimination of oximetry when using more stringent criteria for defining the presence of sleep apnea. For ODI4, sensitivity and specificity were 85% and 90%, respectively, using RDI  5/h, and 90% and 95% using RDI  30/h. Similar values were obtained for ODI3 (see Figure 2 and Table 1).

View larger version (13K): [in this window] [in a new window]   Figure 2.   Receiver operating characteristic curves using MAP questionnaire (thin solid line) or oximetry (heavy solid line). To score oximetry, we used ODI3, the number of episodes of fall in saturation  3%/h of test time. RDI  5/h (upper panel ) or RDI  30/h (lower panel ) defined OSA. Compared with reference (straight dashed line), MAP shows good discriminatory power, and ODI3 better. The curve for ODI4, which used  4% fall in saturation/h of test time, was superimposed on that for ODI3 and is not displayed separately.

Effect of Varying the Parameters (UB, LB, and ODI_threshold) on Sensitivity and Specificity of the Risk Stratification Algorithm

We computed the sensitivity and specificity of our algorithm as a function of the value we chose for UB, LB, and ODI_threshold. We varied UB from 0.5 to 0.9 and LB from 0.1 to 0.5 in 0.1 unit increments, and ODI_threshold from 5 to 25 events/h by increments of 5. We computed sensitivity and specificity for these 125 combinations of UB, LB, and ODI_threshold.

Effect of LB on sensitivity and specificity. As Table 2 shows, for a given ODI_threshold (15/h), if UB is kept constant at a specific value (0.7), varying LB has negligible effect on either sensitivity or specificity of the algorithm compared with PSG for identifying patients with RDI  5 events/h (left two columns). This effect is also true for RDI  30 events/h (right two columns). This effect is true for all levels of ODI_threshold and for any value of MAP UB that we studied (data not shown).

Effect of UB and ODI_threshold on sensitivity and specificity. The effect of varying UB is less straightforward, but still consistent and predictable (Figures 3 and 4). The effect of UB interacts with the effect of ODI_threshold. First, we discuss the case when RDI  5/h is used to define apnea. UB has a negligible effect on sensitivity when ODI_threshold is low. As an example, for ODI_threshold = 5/h, LB = 0.3, as UB is increased from 0.5 to 0.9, the sensitivity drops very little (Figure 3, upper panel, dashed black line). At larger values of ODI_threshold, however, the effect of raising UB on sensitivity increases considerably. For example, at ODI_threshold = 25 events/h, increasing UB from 0.5 to 0.9 while LB = 0.3 results in dramatic decline in sensitivity (Figure 3, lower panel, solid black line) and dramatic increase in specificity (Figure 3, lower panel, solid gray line). Moreover, the effect of varying UB on sensitivity is less pronounced than its effect on specificity for any value of ODI_threshold. An interaction between ODI_threshold and LB was not observed (data not shown). Table 3 (left four columns) provides a numerical synopsis of these data.

View larger version (10K): [in this window] [in a new window]   Figure 3.   Upper bound-ODIthreshold interaction in predicting RDI  5 events/h for a representative lower bound value of 0.3. Specificity (lower panel ) and particularly sensitivity (upper panel ) change more prominently with upper bound when ODIthreshold is 25 events/h (solid lines), compared with the lower value of 5 events/h (dashed lines).

View larger version (11K): [in this window] [in a new window]   Figure 4.   Upper bound-ODIthreshold interaction in predicting RDI  30 events/h for a representative lower bound value of 0.3. As upper bound is increased from 0.5 to 0.9 (x-axis), we see greater changes in sensitivity (upper panel ) and specificity (lower panel ) when ODIthreshold is high at 25/h (solid lines), compared with when ODIthreshold is low, at 5/h (dashed lines).

Prevalence of OSA by MAP Decile

As Figure 5 (white bars) shows, apnea prevalence (RDI  5/h) increases with MAP score, ranging from 0% (for MAP 0.0- 0.1) to 93% (for MAP 0.8-0.9). The overall prevalence of OSA in the population as a whole was 69%. The prevalence of severe apnea (RD  30 events/h, solid bars) increased with MAP score. For those with MAP  0.4, prevalence of severe apnea ranged from 12% to 71%, while only one patient with an MAP score below 0.4 had severe OSA.

View larger version (23K): [in this window] [in a new window]   Figure 5.   Prevalence of OSA by MAP interval in clinic population. OSA prevalence is high in nearly all intervals in the sleep center population, and particularly so in those with MAP  0.6. This was true regardless of the cutpoint for OSA: RDI  5 events/h or 30 events/h.

Optimal Model Parameters Obtained by the Bootstrapping Technique

Using our criterion functions for each of our two objectives, we obtained the biased optimal parameter set. For Objective 1, this set was UB = 0.58 (range 0.5-0.8), LB = 0.14 (range 0.1-0.2), and ODI_threshold = 5.02 desaturations/h (range 5-10). For Objective 2, that is, to detect those with RDI  30 events/h who would be suitable for split-night studies, the biased optimal parameter set was UB = 0.9 (range 0.9-0.9), LB = 0.38 (range 0.2-0.5), and ODI_threshold = 21/h (range 10-25). Interestingly, the entire bootstrap distribution of UB values was fixed at 0.9. The results are summarized in Table 4. Table 4 also shows the total number of parameter sets (which resulted from the 200 bootstrap resamples), which were averaged to obtain the biased optimal parameter set for each of our two objectives.

Model Validation in a Subset of Clinic Patients Using Parameters Obtained by Bootstrapping

Using our unbiased estimates of each of the two optimal parameter sets, we computed model accuracy in our validation sample, which consisted of 75 patients (20%) from the original

clinic population (Table 5).

For Objective I, identifying all potential cases of apnea, the model shows a sensitivity of 95%, with a negative predictive value of 87%. The specificity and positive predictive values are 68% and 86%, respectively. These values are given in Table 5 (top), along with their 2.5% lower confidence limit and 97.5% upper confidence limits. The misclassification rate, defined as the sum of the total false positive and false negative rates divided by the total sample size (n = 75), was 17%. The false negative rate was very low at 7.9%, and the false positive rate was 9.2%.

For Objective II, identifying patients with severe apnea (RDI  30/h) for split-night studies, we found that the model had a specificity of 97% and a positive predictive value of 94%. The sensitivity was 85%, and the negative predictive value was 92%. These values are also given in Table 5 (bottom). The false negative and false positive rates were 5.3% and 3.9%, respectively.

By comparison, the 20% validation sample yielded estimates of accuracy that were comparable to those obtained for the 80% estimation sample for both objectives. (Compare the first and fourth columns of Table 5.)

Comparison of Prevalence of OSA in Estimation and Validation Samples

The prevalence of OSA (RDI  5/h) in the estimation, validation, and total sample was 70%, 68%, and 69%, respectively. Using RDI  30/h, these respective values were 35%, 24%, and 33%. Thus the three groups had similar prevalences of apnea and severe apnea.

	DISCUSSION

TOP ABSTRACT INTRODUCTION METHODS RESULTS DISCUSSION REFERENCES

Our first objective was to identify subjects who do not need a sleep study, which our model accomplished with 95% sensitivity. We reviewed the five cases of apnea that were missed by our algorithm, out of a total of 51 patients with apnea, in the model validation population (n = 75). The individual RDI scores of these five patients were 5, 5, 6, 10, and 11 events/h. Thus, all missed cases had mild disease.

The number of such missed cases is a function of the parameters we chose. Our specific parameter selection criteria resulted in an optimal LB value of 0.1. However, review of Figure 5 shows that most cases of OSA are associated with MAP  0.4. If we use LB = 0.4 as the optimal parameter, we exclude 73/359 = 20% of subjects from unneeded sleep studies, and if we use LB = 0.1, we exclude only 25/359 = 8% of subjects, none of whom had apnea. Excluding this additional 12% of subjects by using LB = 0.4 instead of 0.1 comes at the cost of missing 13 additional cases of apnea, with RDI values 7-23/h for all but one, who had RDI 73/h. Thus, most patients who are missed when we use LB = 0.4 still have mild to moderate apnea (30). However, missing these additional cases might prove to be expensive in the long run, as evidence suggests that patients who have even mild OSA may be at risk for adverse effects, including hypertension (31, 32) and vehicular crashes (33), and may also benefit from therapy (34).

These results suggest that although our risk stratification algorithm works well, it does not remove the need for sleep studies in many patients. This is hardly surprising, as patients being evaluated had come with the clinical suspicion of sleep apnea, and the prevalence of the disorder in this sample is high (69%). This particular aspect of our algorithm might therefore have more utility if applied as a screening tool in a more general population. This is an area for future study. Importantly, this finding highlights the heterogeneity of clinical presentation among those with milder forms of sleep apnea, wherein subjects with similar disease severity may present with quite variable symptoms.

Identification of severe cases of OSA was our second objective, and our specificity was 97%. We correctly identified 15 of the 18 patients in the validation sample with RDI  30 events/h, and 54 of the 57 patients with RDI < 30 events/h. We reviewed the RDI scores of patients incorrectly predicted to have severe apnea by our algorithm. Only three patients were incorrectly identified as having severe apnea, and these had RDI scores of 1, 20, and 25 events/h. Thus, two of three of these subjects had at least moderate apnea, characterized by RDI between 15 and 30 events/h. In addition, three cases of severe apnea were not prioritized by the model to undergo PSG, and these had RDI values of 34/h, 42/h, and 66/h. However, they would have been selected for PSG when the first model is applied, as they had MAP scores of 0.67, 0.80, and 0.71, respectively, all in the high MAP group for the first model. Hence, the diagnosis would not have been missed in any of them.

Thus, the risk stratification algorithm is more useful for prioritizing patients to receive sleep studies than for excluding apnea. Given its value, it is reasonable to consider whether we could select the range of MAP values that is critical. In our model, we found that patients who have an MAP  0.9 should have sleep studies, independent of oximetry results. In the clinic population as a whole, 13 (4%) of 359 patients had MAP  0.9, and 8 of these (62%) had RDI  30/h. In addition, 11 of the 13 (85%) had RDI  5/h. If we instead used MAP  0.8, we would find that 82 of 359, or 23%, had MAP  0.8, of whom 47 of 82 (57%) had RDI  30/h, and 75 of 82 (91%) had RDI  5/h. This is a substantial number of subjects. Thus, we believe that it is reasonable to propose that the sleep center use the optimal model parameters outlined here, or alternatively send the larger number of patients with MAP  0.8 for full sleep studies. The latter represents a slightly adjusted algorithm, but could be used if overnight oximetry was not available, and moreover would remove the costs for this additional test.

Although these diagnostic rates are reasonably accurate, techniques other than MAP and oximetry may yield even better results, including other questionnaires (16, 35), relative proportions of intraoral measurements (41), or other simpler systems to monitor for respiratory disturbances that include readings of nasal pressure (42).

In interpreting these results, we note that oximetry was conducted concurrently with PSG, raising the concern that the two tests could not be scored independently. However, a second observer who had no knowledge of the PSG results scored the oximetry records. Additionally, random rescoring of nPO tracings by a second as well as by the original interpreter showed no significant differences in scores. Test-retest and interrater reliabilities (TRT and IRR) (43) measured using intraclass correlation coefficients were high, and were 0.997 and 0.981, respectively. Performing both nPO and PSG together had benefits, such as ensuring that the individual was sleeping in an identical position and was in an identical stage of sleep for both studies. Moreover, a technician was present to correct signal errors immediately. Future investigations should assess the predictive value of unattended oximetry conducted in the patient's home.

In our sensitivity analysis of oximetry, we observed the trends in sensitivity and specificity when we used a 3% drop in saturation instead of 4% to define events. We found negligible differences in sensitivity and specificity trends using this alternative method of interpreting oximetry data (data not shown).

Another strength of our study was that the prevalence of apnea was similar in the validation sample, in the estimation sample, and in the total sample, for either apnea definition RDI  5/h or RDI  30/h. This suggests that the validation sample was representative of both the estimation sample and of the total sample.

In our algorithm, we explored the value of questionnaire and oximetry, but other techniques may yield even better results, including other questionnaires (16, 35). Relative proportions of intraoral measurements combined with BMI (41) have yielded an AUC of 0.996, with BMI having an exceptionally strong predictive value in this study, having an AUC of 0.938, compared with the value of 0.734 found by us previously (16). Other simpler systems to monitor for respiratory disturbances include readings of nasal pressure (42), and yielded a sensitivity of 100%, specificity of 92%, positive predictive value of 92%, and negative predictive value of 100%, as compared with PSG. These values were better than those of nPO, which produced sensitivity of 75%, specificity of 85%, positive predictive value of 75%, and negative predictive value of 89%. An approach that combines both nasal pressure measurement and oximetry may therefore be the optimal one for the second stage of our model.

In conclusion, our algorithm offers simplicity, feasibility, and sufficient accuracy. The model can omit unnecessary sleep studies in subjects who are unlikely to have OSA, and prioritize high-risk patients to receive early testing. The former reduces the number of sleep studies by only 8-12%, given the high prevalence of apnea in a sleep center population. It is likely to have greater utility in more general populations. The second objective addresses a larger percentage of the population (up to 23%), and gives sleep specialists a tool to identify high-risk patients who should be studied quickly. The optimal tests to be used in this two-stage method have yet to be determined. As diagnostic technology develops, we anticipate that this algorithm can be further refined.