INTRODUCTION

TOP ABSTRACT INTRODUCTION METHODS RESULTS DISCUSSION REFERENCES

Peak expiratory flow (PEF) measurements are currently recommended for monitoring and assessing treatment of asthmatic patients (1, 2), and they have been widely employed to assess outcome in patients who are being followed in clinical and epidemiologic studies. Although the linearity and accuracy of peak flow meters (PFM) have been extensively evaluated (3), objective information about the performance of PFM in field conditions and clinical trials is not available.

In a recent clinical trial carried out by the Asthma Clinical Research Network (ACRN), A.M.-PEF and P.M.-A.M. PEF differences were chosen as important outcome variables (9). The mini-Wright peak flow meter was chosen as the PFM for this trial because the literature contained the most information about its accuracy and precision (3), it was inexpensive, and it was thought to be durable. Although the need for quality control of the PEF measurements during this trial was felt to be essential, there was no simple technique for determining the accuracy and precision of the PFM or for recognizing malfunction during such a large trial.

The purpose of this report is to describe a simple, inexpensive approach to a functional assessment of the mini-Wright PFM. The described method, which requires only a testing chamber, a spirometer, and a calibration syringe, can be adapted to any peak flow meter, and it is easily employed in clinical or multicenter settings. Using this method, we have evaluated the relative accuracy, precision, and reproducibility (interdevice variability) of the mini-Wright PFM. In addition we report an assessment of the durability of this PFM under conditions of use for 26-wk by 255 asthmatic subjects.

	METHODS

TOP ABSTRACT INTRODUCTION METHODS RESULTS DISCUSSION REFERENCES

To evaluate a given PFM, it was placed in a "calibration chamber" in series with a 3-L syringe and a recording spirometer (Figure 1).

View larger version (45K): [in this window] [in a new window]   Figure 1.   Schematic of the setup for peak flow meter (PFM) quality control. The system includes a standard 31 syringe, spirometer, and calibration chamber. The magnified view shows the PFM within the calibration chamber. The arrows indicate the direction of flow through the system during a trial.

Peak Flow Meter

The PFM chosen for this trial was a model of the Mini-Wright peak flow meter (standard range AT; Armstrong Medical, Lincolnshire, IL), which has a new scale that corrects for the known alinearity of the previous model.

Spirometer

A water-sealed Stead-Wells spirometer (Survey III; W.E. Collins, Braintree, MA), which had been modified to include a temperature thermistor in the base of the bell, was employed for this study. Volume was recorded as the displacement of the linear potentiometer, and flow was recorded as the digital differential of the volume signal by computer/software displays (S&M Instruments, Doylestown, PA). Volume calibration of the spirometer was carried out by means of a calibrated 3-L syringe (Hans Rudolph, Kansas City, MO). A volume calibration error of 1% or less was considered acceptable, and it was routinely achieved with this system. Flow was calibrated by the injection and withdrawal of the 3 L of air at different flow rates (slow, medium, and as fast as possible) to develop a reference table for flow based on successful integration to recover the known injected volume. A flow calibration error of 3% or less of the volume injected was considered acceptable.

PFM Calibrator

The PFM calibration chamber, shown in the magnified view of Figure 1, consisted of a quarter-inch Plexiglas^® tube (8 cm ID), which was 30 cm long, with a fixed fitting at one end and a removable fitting at the opposite end. The PFM was attached to the removable end with a rubber fitting (Cat. no. 22351; W.E. Collins), which in turn was sealed into the tube by means of a greased O-ring so that resetting of the indicator on the PFM was easily accomplished.

To determine whether the PFM calibrator introduced any loss of peak flow signal as gas passed through it, a Fleisch type pneumotachograph was positioned in series, either in front of, or behind, the PFM-calibrator containing the PFM at two centers. The pneumotachograph was connected to a differential pressure transducer (Validyne MP45; Validyne, Northridge, CA), an amplifier, and a storage oscilloscope and calibrated with a vacuum cleaner and rotameter calibrator arranged in series. To produce flow rates varying between 100 and 900 L/min, a 3-L syringe was compressed 30 times with different starting volumes and consistent degrees of compression. The data derived in one center is shown in Figure 2. As can be seen, placement of the pneumotachograph in front (A) or behind (B) the calibration device resulted in no significant difference in the relationship between flow determined by the pneumotachograph compared with that indicated on the PFM. The correlation coefficient was 0.99 in both positions with no evidence of bias or loss of peak flow as it passed through the calibrator. The data from the other center were nearly identical.

View larger version (20K): [in this window] [in a new window]   Figure 2.   Loss of flow through the calibrator was assessed by placing a pneumotachograph before (left panel  ) and after (right panel  ) the PFM in the calibrator.

Development of Criteria for Quality Control of Peak Flow Meter

Criteria for quality control of an individual PFM to be used in the multicenter trial were developed initially from data derived from 50 PFM (10 PFM at five centers). Assessment of each PFM was carried out with the calibration system shown in Figure 1. Variable PEF was accomplished by rapidly compressing the 3-L syringe, which was set at different starting volumes in order to achieve five different flow rates between 200 and 500 L /min. With each stroke of the syringe, PEF was determined simultaneously on the PFM and the spirometer. The calibrator tube was opened and the PFM indicator reset to zero after each compression of the syringe. On each PFM, six separate sets of five compressions (i.e., a total of 30 comparisons per PFM) were conducted.

Calculation of relative accuracy. To assess the relative accuracy of measurements of flow on a PFM when compared with the peak flow determined on the spirometer, the relative bias for each pair of readings from the PFM and the spirometer was calculated using the following equation: Relative Bias = [PF (PFM)  PF (Spirometer)]/PF (Spirometer) × 100.

The relative accuracy of a particular PFM in each trial was estimated by determining the median relative bias (MRB) of the five peak flows, i.e., the middle value of the five relative biases. For each PFM, the mean MRB of the six sets of measurements was then determined.

Calculation of precision. In each trial of a PFM, the precision was determined by calculating the interquartile range (IQR) of the five relative biases, i.e., the difference between the second and fourth ranked relative biases. The mean of the six sets of trials was then determined.

Examples of the relationship between PF measurements of the PFM compared with those of the PF from the spirometer in two meters are shown in Figure 3, one with very good relative accuracy and precision (top panel) and one with poorer relative accuracy and precision (bottom panel).

View larger version (19K): [in this window] [in a new window]   Figure 3.   Examples of the quality control tests from two meters; one (top panel  ) with very good relative accuracy (MRB = 3.8%) and precision (IQR = 1%), and another (bottom panel  ) with poorer performance (MRB = 10.5%; IQR = 2.0%).

Criteria for Acceptance of an Individual Peak Flow Meter in the Clinical Trial

The criteria for acceptable performance of a PFM to be used in the clinical trial was based on the frequency distribution of the calculated relative accuracy (Figure 4, top panel) and precision (Figure 4, bottom panel) obtained from evaluation of the 50 PFM. It was arbitrarily decided that a particular PFM would be acceptable for issuance to a patient if the MRB was ± 15% and the IQR of the median relative biases was < 10%. Using these criteria, two of the original 50 PFM (4%) were considered unacceptable because they failed to meet the IQR criteria for acceptance.

View larger version (19K): [in this window] [in a new window]   Figure 4.   Frequency distribution of the relative accuracy (MRB) (top panel  ) and precision (IQR) (bottom panel  ) for the 50 meters studied.

Assessment of Durability of a Peak Flow Meter

The durability of a particular PFM was assessed by determining its MRB and IQR at each of 12 clinic visits over a 26-wk period in the 255 subjects who were enrolled in the clinical trial. Patients were instructed to record peak flow as the highest of three efforts twice a day; thus, the meters were used about 1,092 times during the 26 wk. A PFM was considered to have failed if (1) the clinic visit calculated MRB changed more than ± 5 percentage points of its original MRB, or (2) the IQR had increased more than 5 percentage points from its original IQR.

Replacement of a Failed PFM

When a PFM was identified as a failure in a Clinical Center, the replacement meter was required to meet the original issuance criteria (MRB ± 15% and IQR < 10%) and to be about ± 5% of the original PFM.

	RESULTS

TOP ABSTRACT INTRODUCTION METHODS RESULTS DISCUSSION REFERENCES

Variability of Relative Accuracy of Peak Flow Meters

The variability within and between the five centers of the relative accuracy (MRB) of the PFM compared with the spirometer is presented in the upper panel of Figure 5, which shows the mean and the maximum and minimum values of the six sets of measurements carried out in each of the 50 peak flow meters that were tested. The range of the six sets of MRB varied from 4.4 to +13.2% across all the centers, the mean MRB being +4.1%. Note that in one center (Center 2), which is at high altitude, the mean MRB was lower than that in all the other centers, the mean MRB ranging between 2.2 and 1.3, with a median MRB of 0.87%. When the data from this center are removed from the group, the range of the MRB is 11.03 to 0, with a mean of 5.08%.

View larger version (37K): [in this window] [in a new window]   Figure 5.   The relative accuracy of the PFM compared with that of the spirometer was determined as median relative bias (MRB) (top panel  ), and precision was determined as the interquartile range (IQR) (bottom panel  ). The data for the 10 individual meters studied at each of the five centers are shown. The open circle is the median, and the horizontal bars are the minimums and maximums, respectively. Center 2 is at an altitude of 5,200 ft.

Variability of Precision of Peak Flow Meters

The variability within and between the five centers of the precision (IQR) is shown in the lower panel of Figure 5 which presents the mean, maximum, and minimum values of the six sets of measurements carried out in each of the 50 peak flow meters that were tested. The IQR varied from 0.06 to 11.5% across all centers, with a mean of +1.2%.

Initial Assessment of Peak Flow Meters

Initial assessment of the relative accuracy and precision of the lots of PFM, which were subsequently issued to the 255 subjects with the criteria described earlier, using an abbreviated test (one set of five flow rates between 200 and 500 L/min), found all of the PFM to be acceptable.

Durability of Peak Flow Meters

The durability of PFM was determined by assessing the number that had to be replaced during the 26-wk period of the clinical trial. In addition to the 255 PFM that were issued to the randomized subjects at Visit 1 in the study, a further 102 PFM were issued during the course of the study because of failure to meet the acceptance criteria at follow-up visits (n = 71) or because they were no longer usable for a variety of reasons (i.e., they were broken, lost, or for some other reason). It can be seen in Table 1 that more than one PFM was issued to 73 (28.6%) of the subjects, and that 16 of the 22 subjects who received more than two PFM during the 26 wk of follow-up in the study were located in one center. No clear reason could be identified for this predilection.

Of the 71 PFM that failed to meet the acceptance criteria at follow-up visits, 36 were identified by the Clinical Centers. Of these, 33 failed MRB criteria because of loss of relative accuracy; two failed both MRB and IQR criteria (loss of precision), and one failed IQR criteria alone. The remaining 35 meters were identified as failures by the Data Center on retrospective review of forms containing quality control measurements that were submitted by the Clinical Centers during the course of the clinical trial. Of these, 33 were identified as MRB failures, and two were IQR failures. We also assessed the directionality of the failure in these 68 meters and found that 33 showed an increase in MRB and 35 showed a decrease in MRB.

Data were available to assess the time to failure of the PFM after their issue at Visit 1 in 53 of the 71 PFM that failed. The visit number and weeks during the trial at which "failure" was first identified are shown in Table 2. There was no discernible relationship between failure and the duration of utilization of the PFM (the visit or week at which the PFM failed), that is, the failure rate (meters/visit) was relatively constant.

The additional 35 PFM that met the criteria for failure on review in the data base, but were not recognized by the clinical centers and continued to be used by the subjects, provided an opportunity to evaluate the status of individual PFM at the subsequent visits over the course of the trial. The status of the repeated measurements of these meters at each of the subsequent clinic visits once they had been identified as having failed acceptance criteria are presented in Table 3. Reading left to right are the number of visits and evaluations after the initial recognition of a failed PFM; P or F indicates that the meter either passed or failed on that visit. Four meters failed at the last visit (first row) and thus, were never rechecked. On the other hand, in two meters there were nine subsequent examinations after initial recognition (last two rows). As another example, reading the failure sequence for three meters identified as failing on the eighth visit from study termination, it can be seen that these PFM never failed again. Therefore, the PFM failure was sporadic in the 31 meters that were retested after the initial identification as a PFM failure. Thus, six meters failed the majority of time on retesting, whereas most (n = 25) of the PFM passed 50% or more of the reevaluations at a subsequent clinic visit.

	DISCUSSION

TOP ABSTRACT INTRODUCTION METHODS RESULTS DISCUSSION REFERENCES

Measurement of peak flow, as determined by a peak flow meter, is an important adjunct to asthma management, and it frequently constitutes an important end point in clinical trials. One of the virtues of PF measurements is that it represents an easy way to assess pulmonary function. Although there are numerous reports of linearity and accuracy of PFM (3) under strict laboratory conditions, there have been no reports of quality control or durability of PFM in clinical trials. In this report, we describe a simple technique for evaluating PFM before they are issued to patients and a procedure for their quality control while they are in use in a clinical trial. The calibrating system consists of a chamber to hold the PFM and the generation of a peak flow signal by depression of the 3-L syringe normally used to calibrate the spirometer, to produce flow rates between 200 and 500 L/min (Figure 1). Comparison of readings of PF taken from the meter and the spirometer easily allowed calculation of the relative accuracy and precision of an individual PFM.

In a recent publication, the American Thoracic Society (ATS) has recommended performance standards for diagnostic and monitoring devices (10, 11) with particular attention to PF and the mechanical performance of PFM. It was recommended that (1) a given PFM should be capable of generating accurate flows between 0 and 900 L/min (0 to 15 L/s) and yield readings within 10% or 10 L/min of the true value, whichever is greater; (2) the repeatability (within-instrument agreement) should be 3% or 10 L/min, whichever is greater; (3) the reproducibility (interdevice variability) should be within 5% or 20 L/min, whichever is greater.

To determine whether a given brand of PFM meets these qualifications, most investigators (3) have utilized custom-designed, computer-driven piston arrangements that produce the 24 ATS waveforms (10, 11). Another approach is the use of just one waveform, i.e., a scaled single waveform (Waveform 24 from the ATS Standard Test Waveform Set) as the standard waveform for testing (2). However, computer-driven pistons are expensive and not practical for assessment and quality control (QC) of individual peak flow meters in large multicenter clinical trials or epidemiologic studies. Alternatives to this approach such as use of explosive decompression devices (7) or having subjects blow through the devices (8, 12, 13) were also not deemed to be suitable for long-term clinical trials.

Under the testing conditions that we describe, neither the PFM nor the spirometer is truly accurate, as the PFM is calibrated to ATPD conditions and adjustments have been made to the spirometer system to extrapolate the peak flow to BTPS conditions. There is also a known bias in true accuracy of the PFM that we used (3). Furthermore, we would expect the PFM reading to be higher since the peak flow reported by the spirometer we used (Collins Eagle 2) reads low (~ 10%) (R. O. Crapo, personal communication). Accordingly, the technique that has been described in this report can only assess the relative accuracy of this particular PFM in comparison with that of a spirometer that is assumed to be reasonably accurate.

The relative accuracy (MRB) of 50 mini-Wright PFM determined with this technique ranged from 4.4 to 13.2%; (mean, +4.1%). This is in good agreement with the ± 10% of 10 L/ min criteria recommended by the ATS (11). In addition, we found the precision (IQR) of the PFM to be excellent at the five centers, ranging from 0.06 to 11.5% (mean, 1.2%). This is well below the ATS recommendation for interdevice variability of 5%. Taken together, these data suggest that this extremely simple system supplies data for the PFM that are consistent with more rigorous but costly approaches.

The data also suggest that this technique detects the expected difference in peak flow caused by altitude. The mean MRB of the PFM tested in Center 2 at Denver (5,280 ft) was 0.87%, whereas the mean MRB of PFM tested in centers at or near sea level was +5.08%, i.e., a difference of 5.95%. This effect of altitude is comparable to that expected, as reported by Jensen and coworkers (14), who found the average peak flow to decrease about 7%, or Pederson and coworkers (5) who reported a 5% decrease for each 100 mm Hg decrease in barometric pressure. Thus, in Denver, where the average barometric pressure is about 620 mm Hg, peak flow would be estimated to be about 7 to 9.8% less. Hence, with this simple system we were able to detect some of the expected effects that altitude should have on measurements of flow for this PFM.

A surprising finding during the trial was the high failure rate of the PFM. This was not due to the malfunctioning of a bad lot of PFM, as all the meters were purchased centrally for the entire trial from one manufacturing lot. In addition to the 255 original meters issued, a further 102 meters were used during the trial. Of the 357 meters used, 71 were confirmed failures at some point in the study (a 19.9% failure rate). PFM failure was, in the majority of cases, due to loss of accuracy. When a meter lost its accuracy it could be either high or low, which means that the peak flow reading on a patient could be both over and under reading true PF. Because 22 of 255 subjects (8.6%) were issued three or more meters, this also raises the issue of PFM abuse by some subjects. Of interest, the majority of the subjects who used multiple meters (52% of the two or more PFM, and 73% of the three or more) were located at one center.

Also of interest was the inconsistency of failure, as indicated by the repeated testing of the PFM that were identified as failures in the data base but not at the clinical centers. Of the 31 PFM that were retested at subsequent clinic visits, the majority (n = 25) passed 50% or more of the retests. Also, nine of 10 PFM that were removed from service at one center (Center 5) subsequently passed on retest. This inconsistency may be related to the design of this particular PFM, which has a flapper valve that could be envisioned to stick periodically.

In summary, this study indicates that it is possible (1) to evaluate the relative accuracy and precision of a given PFM; and (2) to ensure the quality of peak flow measurements during a clinical trial using a simple device. This device and procedure for assessing the relative accuracy and precision, as well as durability, can be readily implemented in clinical trials. It is important to remember that these data reflect only the performance of one particular brand of PFM, and they may not reflect the performance of other meters under similar conditions. However, by adapting the chamber size and shape, the technique can be applied easily to other brands of PFM. In any case, to the extent that it is possible to extrapolate these data to other devices, our findings indicate that the failure rate of PFM over time can be high, and that quality control of the PFM over time is absolutely essential in large multicenter clinical trials as well as in routine clinical care.