1. 1 Ambulatory Pediatric Association Educational Guidelines for Pediatric Residency Tutorial 5 Written by Constance D. Baldwin, Ph.D., Carol Carraccio, M.D., and the Educational Guidelines Team Source: Kittredge, D., Baldwin, C. D., Bar-on, M. E., Beach, P. S., Trimm, R. F. (Eds.). (2004). APA Educational Guidelines for Pediatric Residency. Ambulatory Pediatric Association Website. Available online: www.ambpeds.org/egweb. ©2004 Ambulatory Pediatrics Association. All Rights Reserved.

2. 2 Part I Principles of Competency-based Evaluation

3. 3 Ambulatory Pediatric Association Educational Guidelines for Pediatric Residency Basic Principles: Definition of “Competencies”  ACGME has defined six broad competency domains, each composed of several competency elements.  Competency elements are more complex than educational objectives: they encompass knowledge, skill and attitude components.  The competency elements are best evaluated through behavioral and measurable educational objectives.

4. 4 Ambulatory Pediatric Association Educational Guidelines for Pediatric Residency Specification of Competencies While competency domains and elements are general, evaluation must be specific:  In most teaching settings, competencies can be referenced to specific educational objectives that relate to that setting.  Example: Communication and Interpersonal Skills might be evaluated in the nursery by observing how well a resident educates a mother on newborn care or breastfeeding.

5. 5 Ambulatory Pediatric Association Educational Guidelines for Pediatric Residency Criteria for Sound Learner Evaluation Methods Validity: Do ratings accurately depict what the learner knows, without bias or distortion? Reliability: Do ratings of the learner return the same result in multiple settings with multiple evaluators? Generalizability: Does the sampling of a learner’s performance provide a true and complete picture of his/her strengths and weaknesses?

6. 6 Ambulatory Pediatric Association Educational Guidelines for Pediatric Residency Criteria for Sound Evaluation Methods, con’d Feasibility: Can the evaluation method be implemented relatively easily and efficiently (i.e., will faculty and learners comply in using it)? Utility for learners: Is the method sufficiently complete and specific to help learners improve their performance? How can these criteria be applied to evaluations in the “real world”?

7. 7 Ambulatory Pediatric Association Educational Guidelines for Pediatric Residency Use Authentic, Relevant Assessments Competency-based evaluation requires authentic assessment.  One must evaluate performance in tasks that are relevant to the practicing physician, not just measure knowledge  Example: Evaluation of the resident’s ability to evaluate “best evidence” and apply it to a particular patient

8. 8 Ambulatory Pediatric Association Educational Guidelines for Pediatric Residency Don’t Use Global Evaluations Alone A proxy of assessment is not sufficient.  Global evaluations alone do not measure actual skills.  They are subject to “halo” effect (e.g., an articulate resident with good interpersonal skills may be assumed to be skilled in physical examination).  Direct observation is needed to measure performance accurately.

9. 9 Ambulatory Pediatric Association Educational Guidelines for Pediatric Residency Clarify Evaluation Criteria Transparency in evaluation is critical for both evaluator and learner.  Learners need to understand the criteria by which their performance will be judged.  Evaluators need to understand how to use the evaluation tool and the criteria by which they must judge a learner’s performance.

10. 10 Ambulatory Pediatric Association Educational Guidelines for Pediatric Residency Include Criterion- Referenced Tools Consider using criterion-referenced methods in addition to norm-referenced methods.  Norm-referenced tools compare learners to each other.  Criterion-referenced tools measure learners against a predetermined threshold.

11. 11 Ambulatory Pediatric Association Educational Guidelines for Pediatric Residency Use Multiple Evaluation Methods Multiple methods make an assessment more valid.  Example: To evaluate a resident’s ability to care for a patient, observe the resident performing a history & physical in addition to reviewing the resident’s written documentation on that patient.

12. 12 Ambulatory Pediatric Association Educational Guidelines for Pediatric Residency Emphasize Formative Feedback Focus on formative feedback rather than summative evaluation.  Learners deserve guidance in meeting the threshold you have set for achieving competence.  If learners are falling short, timely feedback and a remediation plan are critical.

13. 13 Ambulatory Pediatric Association Educational Guidelines for Pediatric Residency Emphasize Formative Feedback Good feedback and counseling should be:  Frequent and timely (during experience)  Based on careful observation  Expressed in specific behavioral terms  Constructive—non judgmental  Accompanied by a plan for improvement and a schedule for follow-up

14. 14 Ambulatory Pediatric Association Educational Guidelines for Pediatric Residency Encourage Residents to Self-evaluate Self-directed learners become life-long learners! Use table of goals/objectives for resident self-evaluation on first and last day of an experience. Assign self-assessment tasks to reinforce importance of self-direction. Encourage long-range planning by resident, based on learning needs at end of rotation.

15. 15 Part II Practical Tips for Creating a Competency-based Evaluation System

16. 16 Ambulatory Pediatric Association Educational Guidelines for Pediatric Residency Planning an Evaluation System: Basic Principles Evaluation methods must be based on specific, measurable or observable objectives. Evaluation tools should precisely specify level of expected performance. Summative evaluations can be brief and still valid only if they are based on iterative, detailed and specific formative evaluations. Faculty and residents need orientation for the system to work!

17. 17 Ambulatory Pediatric Association Educational Guidelines for Pediatric Residency 1.Develop tools and processes to evaluate residents in each competency domain.  Note RRC suggestions for methods in each domain. 2.Provide faculty development and resident orientation.  Residents must know what target they are aiming for and whether they hit it.  Faculty must become skilled at conducting valid and reliable evaluations. 3.Establish a regular feedback system. Planning an Evaluation System: A Primer

18. 18 Ambulatory Pediatric Association Educational Guidelines for Pediatric Residency Planning an Evaluation System: A Primer 4.Assess resident performance on each rotation and document progress over time. Monitor each rotation’s progress in competency integration. Periodically evaluate teaching and program performance, so you can tune the system.

19. 19 Ambulatory Pediatric Association Educational Guidelines for Pediatric Residency Practical Applications: Choose Targets and Tools To create a competency-based evaluation system, first:  Select the competencies and related educational objectives from the APA Guidelines that you want to evaluate.  Decide which evaluation tools are most effective for assessing these learning objectives.

20. 20 Ambulatory Pediatric Association Educational Guidelines for Pediatric Residency Choosing Tools: ACGME Toolbox Written/oral examinations Global ratings Procedure/case logs Record review Checklist evaluation Direct observation of clinical skill OSCE/standardized patients Simulations and models Portfolios 360-degree evaluations (peers, staff) Chart stimulated recall oral exam Patient surveys

21. 21 Ambulatory Pediatric Association Educational Guidelines for Pediatric Residency Choosing Evaluation Tools Most commonly used instruments (each measures a different aspect of competence)  Global rating by supervisor  Multiple choice exams  Standardized patients RRC would prefer more use of:  Direct observations  Multi-source (“360”) evaluations

22. 22 Ambulatory Pediatric Association Educational Guidelines for Pediatric Residency How to Improve Global Ratings? Backup ratings with direct observation. Annotate form with detailed coding instructions that are context-specific. Diversify raters (e.g., peers, staff, patients). Provide systems to support faculty training, form completion and collection, analysis, and feedback systems.

23. 23 Ambulatory Pediatric Association Educational Guidelines for Pediatric Residency Direct Observation is Essential OSCE’s & standardized patients expensive  Valuable teaching and feedback tools  Need multiple cases for good reliability in “high stakes” exams Clinical evaluation exercises more practical  CEX = Observed H&P, presentation and structured feedback  Mini CEX = 20 min focused on 1-2 elements Bottom line: You need to observe and give feedback!

24. 24 Ambulatory Pediatric Association Educational Guidelines for Pediatric Residency Choose Evaluators and Settings Plan your evaluation strategy:  Decide where residents most practically can be evaluated within your program.  Be sure all competency domains and elements are covered (not every competency needs to be measured in every rotation).  Decide who are the best evaluators.

25. 25 Ambulatory Pediatric Association Educational Guidelines for Pediatric Residency Educate Faculty and Residents Provide faculty development for your evaluators so that they understand the tool they are using and how to apply it. Emphasize the need to give formative feedback. Educate your learners about the process and criteria upon which they will be evaluated.

26. 26 Ambulatory Pediatric Association Educational Guidelines for Pediatric Residency Evaluate the Evaluation Process Don’t stop with evaluating the learner. You need to evaluate the program’s success in:  Teaching and evaluating the learners  Delivering an effective and efficient curriculum  Assessing and refining the program

27. 27 Ambulatory Pediatric Association Educational Guidelines for Pediatric Residency Evaluation: Summary of Literature Evaluation of clinical competence is a daunting, complex task. Evaluation of life-long learning skills, professionalism, and systems-based practice is an even newer, bigger challenge. No single evaluation tool can adequately assess everything. A multifaceted approach is required. The “right combination” depends on program goals, needs and resources.

28. 28 Part III Building Competency-based Evaluation Forms

29. 29 Ambulatory Pediatric Association Educational Guidelines for Pediatric Residency Building Evaluation Forms: Basic Principles Type of form (e.g., global, multi-source) should reflect setting, evaluators, program requirements. Items on form should reflect the core content of the learning experience and be specified clearly. Evaluation coding system should be well defined, so it is consistently applicable across evaluators. Specify performance expectations for each item.

30. 30 Ambulatory Pediatric Association Educational Guidelines for Pediatric Residency Building Evaluation Forms: A Primer A typical evaluation form is composed of 3 parts: 1.Header (top row) 2.Content rows 3.Footer

31. Evaluation Form Template Form Title Resident: Date: Evaluator: Date: Evaluation Anchors 1= 3= 5= N/0 = No opportunity to evaluate Content Row 1 Content Row 2, etc. Open-ended Comments Column Labels Header Footer Content Rows

32. 32 Ambulatory Pediatric Association Educational Guidelines for Pediatric Residency Designing Form Headers Evaluation codes should be:  As specific as possible in describing expected performance  Well matched to the behavior being evaluated  Easy to reproduce consistently across evaluators Include an escape option to minimize guesswork (e.g., NE=not evaluable).

33. 33 Ambulatory Pediatric Association Educational Guidelines for Pediatric Residency Designing Form Headers, cont’d Column labels can be used to establish:  Evaluation time points (e.g., mid- rotation, end of rotation; PGY level)  Performance benchmarks (behaviors expected at evaluation time points or for final certification of competence)  Action plans for learner improvement  Comments by learner or evaluator

34. 34 Ambulatory Pediatric Association Educational Guidelines for Pediatric Residency Selecting Row Content (Evaluation Items) Use Goals/objectives or Competencies/ elements: Make items match the actual activities observed in a resident. Make items as specific as space allows. Include enough items to allow a fair sampling of a resident’s performance. Length of a form, however, cannot exceed an evaluator’s willingness to complete it!

35. 35 Ambulatory Pediatric Association Educational Guidelines for Pediatric Residency Flexible Uses of the Footer Open-ended comments can be used: By evaluator, to clarify and specify ratings By evaluator, to help the learner improve performance By learner, to comment on the evaluation or action plan

36. 36 Ambulatory Pediatric Association Educational Guidelines for Pediatric Residency What Matters Most in the End An evaluation method is only as good as the evaluator who implements it. Work with faculty and residents to teach them about evaluation and promote cooperation.