1. printed by www.postersession.com Quantitative Assessment of Interviewing Competencies David A Goldberg MD, Steven P Reidbord MD, Dongmei Yue MD California Pacific Medical Center, San Francisco CA In recent years, educators have sought to develop means to assess the six Core Competencies defined by ACGME. It is centrally important in psychiatry that we develop reliable and valid methods to assess aspects of the doctor-patient relationship, especially the competent formation of a working alliance and competent conduct of patient interviews. While assessment of resident competency is especially challenging in these areas, reliable and valid measures of working alliance and therapist interview behavior have been developed outside the educational context. This project adapts and simplifies two such measures. By applying them to a standardized “oral board” interview format, and potentially in other clinical settings, training programs can quantitatively assess these crucial competencies. Ten residents participated in the practice “oral board” interviews. There was a nonsignificant trend (p = 0.07) for patients to rate a stronger working alliance than residents or faculty did, and a nonsignificant trend (p = 0.11) for working alliance to improve as PGY level increases (see Fig. 1). While adverse interview behaviors did not vary statistically across PGY level, the rate in PGY-1s was somewhat higher than at more advanced PGY levels (see Fig. 2). Working alliance showed a robust inverse correlation with adverse interview behaviors (p =.001). Rated “difficulty” of the patient did not predict alliance or therapist behavior ratings. In this pilot study, we selected a subset of six questions from the Working Alliance Inventory (WAI) [Horvath and Greenberg, 1989] and ten from the Inventory of Interview Behavior (IIB) [Friedman and Gelso, 2000]. The WAI asks raters to assess the degree to which goals, tasks, and bonds are shared between the interviewing resident and the interviewed patient. The IIB asks raters to assess observable positive or negative behaviors in the resident that may interfere with a patient interview (and that, from a psychodynamic viewpoint, may suggest countertransference reactions). The item subsets were chosen to simplify administration of the questionnaires, and to exclude items that were deemed hard to rate in the context of an initial evaluation interview. Residents conducted a 30-minute “oral board” type interview with a previously unknown patient. Each interview was observed by a pair of faculty members out of a pool of 11 such faculty. After the interview, and a 30-minute case presentation and examination period, each faculty observer, resident, and patient independently completed shortened versions of the WAI. Faculty also completed a shortened version of the IIB, and rated, on a seven-level Likert type scale, how much difficulty a “fully competent” psychiatrist would have forming a working alliance with this patient. Likewise, the resident was asked to rate on the same scale, “How difficult was it to form the working alliance with this patient?” Horvath, A. O., & Greenberg, L. S. (1989). Development and validation of the working alliance. Journal of Counseling Psychology, 36, 223-233. Friedman S.M; Gelso C. J. (2000). The Development of the inventory of countertransference behavior. Journal of Clinical Psychology, 56 (9), 1221-1235. We adapted standardized measures from the literature that offer a face- valid metric to assess resident Core Competencies, specifically aspects of Clinical Skills, Interpersonal Communication, and Practice-based Learning & Improvement. Our simplified versions are easily learned and applied. As expected, the two measures are inversely correlated when initial patient interviews are rated. Our preliminary results suggest that meaningful variations may occur as a result of training level. As used here, they also serve as “360 degree” evaluations of the resident, and offer the resident direct patient feedback under faculty supervision. 1.Develop an easily applied metric to measure the working alliance and therapist behaviors in initial evaluations. 2.Provide a quantitative assessment of the ACGME Core Competencies of Clinical Skills, Interpersonal Communication, and Practice-based Learning & Improvement. 3.Compare multiple perspectives (resident, faculty, and patient) on these measures. 4.Encourage collaborative research on these and related measures. BACKGROUND EDUCATIONAL OBJECTIVES METHOD RESULTS DISCUSSION BIBLIOGRAPHY FUTURE DIRECTIONS Larger studies, of greater statistical power, are needed to support our pilot effort, and to confirm the reliability and validity of these adapted measures. We have begun using the same protocol for faculty-supervised PGY-3 outpatient evaluations. We seek interested programs for collaboration, in order to collect more data, and to publish the method. FIGURE 1, LEFT FIGURE 2, RIGHT