1. 360 Degree Evaluation Craig McClure, MD May 15, 2003 Educational Outcomes Service Group

2. Description Use of rating forms to report frequency of observed behavior Multiple people in contact with resident act as evaluators Often survey type form Ratings summarized by topic Include goal-setting

3. Background Human resources in business ACGME found no published reports of use in GME

4. Use for “Soft” Areas More accurate for formative than summative feedback Interpersonal & communication Professional behavior Limited –Patient care –Systems-based practice

5. Decision to Utilize Accepted and used by residents, faculty, staff? Develop or purchase? Cost? Who are the raters? How will the tool be used?

6. Decision to Utilize (2) To whom is the information available? What core competencies will be evaluated with this tool? How nurture trust the process remains confidential? Platform of evaluation

7. Acceptance Will all potential evaluators fully participate? Will raters be fair & honest? Will residents accept the feedback from non-faculty?

8. Develop or Purchase Development permits tailoring Development time may be considerable Purchasing gives a ready-made product Purchasing: computer based

9. Developing Expert in educational testing Programming expertise Pilot period

10. Purchase Items measured appropriate? Does it perform as claimed? Inter-rater reliability? Degree of support and ability to customize

11. Cost If purchasing, monetary cost If developing, personnel support Data management system Personnel time to complete forms Annual development plan

12. Cost (2) Addressing EEOC/grievance complaints Handling disputes over data Divisive & counterproductive for those resistant

13. Personnel Evaluation Time 5 to 10 nurse evaluators per resident to give reproducible results More for faculty More for patients

14. Identify Raters Patients (how explain process) Nursing staff Clerical staff members Physician faculty members Non-physician faculty members Residents

15. Identify Raters (2) Medical students Allied Health Personnel Self-assessment

16. Patients as Raters Literacy Language Culture (medical and otherwise) Personality

17. Intended Utility Intervals: monthly, quarterly, yearly Summative versus formative To support high stakes decisions?

18. Access to Information Resident Advisor Program Director

19. Confidentiality & Trust Raters require anonymity Residents require confidentiality Both need the process to be positive & constructive Prior history conditions expectations Education to process aids current participation

20. Platform of Evaluation PDA Paper Computer

21. Challenges Securing appropriate instruments for variety of evaluators Managing data successfully

22. Advantages Electronic database for documentation Ease of access for raters Rapid turnaround for feedback “Gap” analysis (self perception versus image of others)

23. Disadvantages Hardware/software costs Lack of validation in GME Potential information overload Selection bias Discoverability Potential for invalid feedback

24. References Assessment of Communication and Interpersonal Skills Competencies, C.C. Hobgood, et.al. Academic Emergency Medicine 2002;9: 1257- 69 ACGME/ABMS Joint Initiative Toolbox of Assessment Methods, September 2000

25. References (2) 360-degree Feedback, K.G. Rodgers,et.al. Academic Emergency Medicine 2002;9:1300-1304 Letter from ADFM listserv, Goldsmith to Kikano