Published ahead of print on September 25, 2002, doi:10.1164/rccm.200206-624OC
© 2003 American Thoracic Society
Effectiveness of Medical Resident Education in Mechanical VentilationDepartment of Medicine, University of North Carolina at Chapel Hill School of Medicine, Chapel Hill, North Carolina; Department of Medicine, Vanderbilt University School of Medicine, Nashville, Tennessee; Department of Medicine, Duke University School of Medicine, Durham, North Carolina; Department of Medicine, Johns Hopkins University School of Medicine, Baltimore, Maryland; Department of Medicine, University of California at San Francisco School of Medicine, San Francisco, California; Department of Medicine, University of Colorado School of Medicine, Denver, Colorado; Department of Medicine, New York University School of Medicine, New York, New York; Upper Chesapeake Health System, Bel Air, Maryland; and Department of Medicine, University of Chicago School of Medicine, Chicago, Illinois Correspondence and requests for reprints should be addressed to Shannon S. Carson, M.D., UNC-Chapel Hill Division of Pulmonary and Critical Care Medicine, CB #7020, Chapel Hill, NC 27599-7020. E-mail: scarson{at}med.unc.edu
Specific methods of mechanical ventilation management reduce mortality and lower health care costs. However, in the face of a predicted deficit of intensivists, it is unclear whether residency programs are training internists to provide effective care for patients who require mechanical ventilation. To evaluate these educational outcomes, we administered a validated 19-item case-based test and survey to resident physicians at 31 diverse U.S. internal medicine residency programs nationwide. Of 347 senior residents, 259 (75%) responded. The mean test score was 74% correct (SD, 14%; range, 37 to 100%). Important items representing evidence-based standards of critical care answered incorrectly were as follows: use of appropriate tidal volume in the acute respiratory distress syndrome (48% incorrect), identifying a patient ready for a weaning trial (38% incorrect), and recognizing indication for noninvasive ventilation (27% incorrect). Most accurately identified pneumothorax (86% correct) and increased intrathoracic positive end-expiratory pressure (93% correct). Better scores were associated with "closed" versus "open" intensive care unit organization (76 versus 71% correct, p = 0.001), resident perception of greater versus lesser ventilator knowledge (79 versus 71% correct, p = 0.001), and graduation from a U.S. versus international medical school (75 versus 69% correct, p = 0.033). Although overall training satisfaction correlated strongly with program use of learning objectives (r = 0.89, p < 0.0001), only 46% reported being satisfied with their mechanical ventilation training. We conclude that senior residents may not be gaining essential evidence-based knowledge needed to provide effective care for patients who require mechanical ventilation. Residency programs should emphasize evidence-based learning objectives to guide mechanical ventilation instruction.
Key Words: clinical competence • education, medical • educational measurement • internship and residency • mechanical ventilation
Randomized, controlled clinical trials have demonstrated that specific methods of mechanical ventilator management can improve mortality (1, 2), decrease the duration of mechanical ventilation and intensive care unit (ICU) stay (2, 3), and reduce costs and ICU complications (3) for the nearly 1.5 million U.S. patients who require mechanical ventilation each year (4). Critically ill patients with acute respiratory failure should be treated by evidence-based clinical practice guidelines as the standard of care (5–7). Intensivists care for only 36.8% of critically ill patients in the United States (8), whereas general internists, surgeons, and anesthesiologists manage the rest. National surveys have reported that 59% of general internists use mechanical ventilators in practice and that 67% of patients admitted to ICUs remain on the service of the primary physician (9). Of particular interest in the care of the critically ill is the growing numbers of hospitalists, most of whom are general internists who provide care for ICU patients (10). Overall, physicians trained in internal medicine direct 63% of U.S. ICUs, and only half of ICU medical directors and intensivists are certified in critical care medicine (8, 11). In the coming decade, the United States may experience a significant unmet demand for physicians who are able to provide care to critically ill patients, primarily because of the aging of the population (8). Greater numbers of older patients are being treated in intensive care units than ever before (12), with those over age 65 years incurring about 60% of all ICU days (8). The management of patients requiring mechanical ventilation is a particular concern, as the incidence of acute respiratory failure requiring mechanical ventilation increases nearly 10-fold between the ages of 55 and 85 years (13). Because even more generalists will likely be needed to care for these patients in the future, it is incumbent on residency training programs to teach medical residents important elements of the management of persons requiring mechanical ventilation, including evidence-based standards of care. It is unknown how successful residency programs have been in educating trainees in the management of mechanically ventilated patients. Therefore, we created a specific written mechanical ventilation test and questionnaire with the goals of (1) measuring the knowledge thought necessary by a panel of experts for graduating internal medicine residents to provide effective care for ventilated patients, (2) describing the perceptions of residents and residency program directors of the adequacy of this knowledge, and (3) assessing the characteristics of residents and residency programs that are associated with greater mechanical ventilator knowledge and satisfaction with training. We also surveyed residency program directors to understand how programs provide mechanical ventilation education and evaluate the outcomes of this instruction.
Item Development The overall goal was to create a short but inclusive test that would measure essential and evidence-based knowledge necessary for a senior medical resident to provide effective care for a mechanically ventilated patient. Eight board-certified medical intensivists with extensive experience in practicing and teaching critical care medicine in an academic setting, as well as in publishing on mechanical ventilation and medical education topics, were involved in test writing. This group aimed to compile a case-based, multiple choice examination written in the style of the American Board of Internal Medicine certification test. The group intended the test to address five specific areas of knowledge relevant to mechanical ventilation: (1) cardiopulmonary physiology, (2) indications for mechanical ventilation, (3) diagnosis and management of mechanical ventilator complications, (4) use of different ventilator modes and settings, and (5) interpretation of data from mechanical ventilators. The test consisted of four case vignettes featuring patients with commonly encountered potential indications for mechanical ventilation such as chronic obstructive pulmonary disease, asthma, acute respiratory distress syndrome (ARDS), and pneumonia. Multiple choice questions with four single-response options followed each case presentation.
Item Reduction
Sample Selection Between December 2000 and January 2001, we sent e-mail messages to the 291 internal medicine training programs with functional e-mail addresses on the Accreditation Council of Graduate Medical Education's Web site (14). These messages described the study protocol and requested the participation of each program in the study. A total of 137 programs responded, of which 82 (28%) agreed to participate in the study. Two months later, we polled program directors again to assess expected compliance with the study protocol and feasibility for the program. At this point 12 failed to respond, 24 either could not guarantee full participation in the study protocol or expected a significant number of residents to be off site, and 8 declined to participate. We then chose a sample of 26 programs from the 36 remaining to assemble as diverse a group of programs as possible yet equally distributed in terms of geographic region, small versus large numbers of residents in the program, urban versus suburban or rural setting, and university versus community hospital affiliation.
Validity Assessment of the Questionnaire To evaluate more rigorous aspects of validity, the initial test was administered by residency or fellowship program directors at 5 university medical centers randomly selected from the 36 that agreed to participate. A total of 132 participants returned tests, including 103 internal medicine residents of all three levels of training, 19 pulmonary and critical care medicine fellows, and 10 attending physicians trained in pulmonary and critical care medicine. The anonymous tests took about 45 minutes to finish and were completed as a proctored group in most cases. The mean percentage of correct answers on the pilot test ranged from 67% (SD, 17%) for postgraduate first-year residents to 95% (SD, 6%) for attending physicians. We evaluated the criterion validity of the questionnaire by testing for an empirical association between test scores and duration of training. Using one-way analysis of variance testing, we found that test scores increased significantly overall with year of training (p < 0.0001) from interns to senior residents. Last, we aimed to demonstrate evidence of construct validity, the theoretical assessment of validity based on the strength of relationships between test scores and other measured variables (15). We hypothesized that test scores would have high correlation with year of training (r > 0.6) but low correlations (r < 0.15) with testing site. Because our test was designed to assess knowledge needed by a senior resident to manage mechanically ventilated patients, we also hypothesized that test scores would increase significantly between each year of training, but not between fellows and attending physicians. These relationships behaved as expected, as we found Spearman correlation coefficients of 0.57 (p < 0.0001) and 0.03 (p = 0.49) between test scores and duration of training and test site, respectively. Using two-sample t tests, we also showed that there were significant differences in scores between each year of training (all p < 0.001), although there were no significant differences between the scores of fellows and attending physicians (p = 0.35). We made no further changes to the test (see resident test in online supplement).
Final Questionnaire Administration
Variables
Statistical Analysis
Participant and Residency Program Characteristics Characteristics of the senior residents and their residency programs are shown in Table 1 . A total of 259 of a possible 347 residents (75%) completed the tests. In addition, 29 of 31 program directors (94%), including 5 from the pretesting phase, returned questionnaires. Twenty-five percent of program directors had received training in pulmonary and critical care medicine. A total of 82% programs relied on the opinion of supervising physicians for overall evaluation of resident competence, and only two programs had specific methods to evaluate mechanical ventilation training adequacy.
Test Results The mean test score for senior residents on the mechanical ventilation test was 74% correct (SD, 14%; range, 37 to 100%). Ten percent of residents answered fewer than half of the questions correctly and more than one-third answered fewer than 70% correctly. Mean combined test scores by residency program ranged from 46 to 87% correct. The performance of residents on specific test items is shown in Table 2 . A total of 48% did not choose an appropriate tidal volume setting (6 ml/kg) for a patient with ARDS, 38% failed to identify a patient appropriate for a weaning trial, and 27% did not recognize the appropriate indication for noninvasive positive pressure ventilation in the management of a patient with chronic obstructive pulmonary disease and acute respiratory failure. Among the 48% of residents who answered the ARDS question incorrectly, 85% indicated that they would have provided a tidal volume nearly double the recommended 6 ml/kg of ideal body weight. Although 93% of residents identified clinical findings suggestive of severe hypotension related to "auto-PEEP" (elevated intrathoracic positive end-expiratory pressure), 35% were unable to choose a mechanical ventilator setting that could decrease it. However, most residents (86%) correctly diagnosed a serious tension pneumothorax and could discriminate between an obstructed endotracheal tube and a tension pneumothorax, using information obtained from a ventilator (also 86% correct). A total of 79% of residents understood the physiologic relationship between hypoxemia and intrapulmonary shunt, whereas 75% correctly interpreted the relationship between lung compliance, tidal volume, and inspiratory pressures.
Factors Associated with Test Scores Associations between test scores and characteristics of both residents and their residency programs are shown in Table 3 . Characteristics of training programs associated with better test scores included ICU organization (closed versus open), higher number of senior residents in the program, and greater numbers of attending physicians on staff. Resident characteristics associated with higher test scores included perceived adequacy of ventilator knowledge, awareness of learning objectives, and U.S. medical graduate status. Neither resident characteristics intuitively associated with self-motivated learning, such as plans for future fellowship in critical care and intention to care for ICU patients after residency, nor total amount of time spent in the ICU was related to test scores.
Multiple linear regression modeling demonstrated better scores for residents in closed versus open ICUs (76 versus 71% correct, p = 0.033), for those with greater perceived knowledge level (79 versus 71% correct, p = 0.001), and for those who were U.S. versus international medical graduates (75 versus 69% correct, p = 0.001) after adjustment for program site effect and cluster sampling.
Resident Satisfaction with Training
Questionnaire Responses
The principal aim of our study was to determine whether graduating senior internal medicine residents are receiving the knowledge necessary to provide effective care for mechanically ventilated patients. We found that almost half of residents could not identify an appropriate tidal volume for a patient with ARDS, 38% could not recognize a patient ready for a weaning trial, and nearly one-third did not recognize indications for noninvasive ventilation—all concepts associated with lower mortality and reduced health care costs. Nearly half of the residents indicated they were dissatisfied with their training in mechanical ventilation and many perceived their knowledge to be only minimally adequate for providing effective patient care. Overall, the results of this nationwide study indicate that the mechanical ventilator training of many senior internal medicine residents is suboptimal.
Implications
How Can Programs Improve Educational Outcomes? An essential aspect of improving knowledge of residents should include the close monitoring of educational outcomes with one or a combination of evaluation tools (23, 24). Our suggestions echo those of the Accreditation Council of Graduate Medical Education Outcome Project, a long-term initiative begun in 1999 to emphasize the use of learning objectives, to increase reliance on dependable methods of competency assessment, and to use outcome results to improve both resident performance and the quality of instruction (25). However, almost all the programs in our study relied only on the opinion of an attending physician for evaluation of resident skills, although faculty members often perform poorly in identifying residents with inadequate clinical competence (26). Half of the program directors in our study reported that the Outcome Project had not influenced their desire to change their educational programs in some way. The role of the teaching physician is important to consider in improving education. Trainees highly valued the instruction of attending physicians and fellows. This finding supports the recommendation by the Residency Review Committee of the American Board of Internal Medicine that critical care specialists be available for resident supervision and training in the ICU. However, the poor performance of residents on items relating to more recent landmark clinical studies may indicate that attending physicians and fellows have not widely implemented these newer, important changes in their standard practice (27). Clear learning objectives for mechanical ventilation based on good evidence that are written and disseminated by professional peer organizations will guide not only the education of trainees, but will also inform instructors who are unaware of the more recent literature. Program-level interventions that may improve knowledge could include a brief "hands on" course in mechanical ventilation management at the beginning of the academic year or a specific ICU rotation. This would provide a low-stress opportunity for trainees to become familiar with ventilators as well as dedicated time to review important learning objectives. Others have demonstrated that a similar 1-day annual course in ICU infection control practices for house staff was not only acceptable to residents and staff but improved patient outcomes and reduced costs due to adverse events (28). We recognize that the competing demands of training are significant, and that knowledge deficits persist in other areas of internal medicine training (29–31). Therefore, extra training in this one topic may not be feasible. This is another reason to emphasize clearly stated learning objectives during an ICU rotation. These objectives will improve the focus of instruction in the ICU and also guide residents in independent learning without adding additional burdens to the program, its instructors, and its trainees. Finally, it may be helpful for residency programs to further consider the characteristics of residents associated with worse educational outcomes in our study, such as training in an ICU with a closed organization, international medical graduate (IMG) status, and resident-perceived inadequacy of knowledge. Residents who train in open ICUs may receive less focused critical care instruction and have less exposure to specialists because of divided patient care responsibilities between hospital wards and the ICU. A previous study indicated that residents who trained in a closed ICU setting felt more experienced and had a higher confidence level when managing ICU patients compared with residents who trained in open ICUs (32). The slightly worse performance of IMGs was surprising because of superior scores achieved by IMGs relative to U.S. graduates on the Internal Medicine In-Training Examination (33). It is possible that program-level factors for which we were unable to control in our analysis could explain this observation more completely. For example, IMGs may have had less exposure to critical care medicine before residency than U.S. graduates. Residents accurately predicted their own knowledge deficits, an observation that reproduces findings from a study of physician knowledge of pulmonary artery catheters (31). This may be reassuring, assuming that those with lower levels of perceived knowledge would consult others when contemplating difficult management decisions. It is important to note that the differences between these subgroups were typically the equivalent of one more incorrect test item. Therefore, we believe that these findings should not distract the reader from the main message of this study: residents from all subgroups fared poorly on evidence-based items that have immediate impact on patient mortality and costs.
Limitations
Conclusion
The authors sincerely thank the residents, program directors, fellows, and attending physicians who participated in this study.
Supported by a National Research Service Award (NRSA) to C.E.C., administered through the University of North Carolina at Chapel Hill during the study period. Present address for C.E.C.: Department of Medicine, Duke University School of Medicine, Durham, North Carolina. This article has an online supplement, which is accessible from this issue's table of contents online at www.atsjournals.org Received in original form June 27, 2002; accepted in final form September 23, 2002
This article has been cited by other articles:
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
TOP
ABSTRACT
INTRODUCTION
2 testing was used to compare dichotomous variables. Spearman correlations between variables found in bivariate analyses to have significant (p < 0.05) or borderline significant (p < 0.25) relationships with the main outcome variable, test score, were determined before modeling to examine for potential collinearity. Variables describing university versus community program setting and number of hospital beds were dropped because of strong correlations (r = -0.68 and 0.53, respectively) with number of residents, which itself was found to be most significantly related to test scores in earlier t tests (p = 0.003). Next, a multiple linear regression model was fit with the remaining important independent variables to determine which resident and program characteristics were associated with better test scores. Partial F tests were used sequentially to drop variables that did not contribute significantly to the final model. Correction for intraclass correlation related to potential residency program site effect was performed by variance estimate adjustment in the linear regression model (
