360 Degree Evaluation Craig McClure, MD May 15, 2003 Educational Outcomes Service Group

Description Use of rating forms to report frequency of observed behavior Multiple people in contact with resident act as evaluators Often survey type form Ratings summarized by topic Include goal-setting

Background Human resources in business ACGME found no published reports of use in GME

Use for “Soft” Areas More accurate for formative than summative feedback Interpersonal & communication Professional behavior Limited –Patient care –Systems-based practice

Decision to Utilize Accepted and used by residents, faculty, staff? Develop or purchase? Cost? Who are the raters? How will the tool be used?

Decision to Utilize (2) To whom is the information available? What core competencies will be evaluated with this tool? How nurture trust the process remains confidential? Platform of evaluation

Acceptance Will all potential evaluators fully participate? Will raters be fair & honest? Will residents accept the feedback from non-faculty?

Develop or Purchase Development permits tailoring Development time may be considerable Purchasing gives a ready-made product Purchasing: computer based

Developing Expert in educational testing Programming expertise Pilot period

Purchase Items measured appropriate? Does it perform as claimed? Inter-rater reliability? Degree of support and ability to customize

Cost If purchasing, monetary cost If developing, personnel support Data management system Personnel time to complete forms Annual development plan

Cost (2) Addressing EEOC/grievance complaints Handling disputes over data Divisive & counterproductive for those resistant

Personnel Evaluation Time 5 to 10 nurse evaluators per resident to give reproducible results More for faculty More for patients

Identify Raters Patients (how explain process) Nursing staff Clerical staff members Physician faculty members Non-physician faculty members Residents

Identify Raters (2) Medical students Allied Health Personnel Self-assessment

Patients as Raters Literacy Language Culture (medical and otherwise) Personality

Intended Utility Intervals: monthly, quarterly, yearly Summative versus formative To support high stakes decisions?

Access to Information Resident Advisor Program Director

Confidentiality & Trust Raters require anonymity Residents require confidentiality Both need the process to be positive & constructive Prior history conditions expectations Education to process aids current participation

Platform of Evaluation PDA Paper Computer

Challenges Securing appropriate instruments for variety of evaluators Managing data successfully

Advantages Electronic database for documentation Ease of access for raters Rapid turnaround for feedback “Gap” analysis (self perception versus image of others)

Disadvantages Hardware/software costs Lack of validation in GME Potential information overload Selection bias Discoverability Potential for invalid feedback

References Assessment of Communication and Interpersonal Skills Competencies, C.C. Hobgood, et.al. Academic Emergency Medicine 2002;9: ACGME/ABMS Joint Initiative Toolbox of Assessment Methods, September 2000

References (2) 360-degree Feedback, K.G. Rodgers,et.al. Academic Emergency Medicine 2002;9: Letter from ADFM listserv, Goldsmith to Kikano