1. Comparison: Traditional vs. Outcome Project Evaluative Processes
This presentation is targeted toward new faculty members or residents, being a quick introduction to differences between the “traditional” style of evaluation in graduate medical education and the style inherent to the ACGME Outcome Project.
It should be admitted immediately that for the sake of highlighting differences between the two styles, there have necessarily been over-simplifications. Certainly it would be unfair to say no one ever has previously evaluated residents in any of the ways attributed to the curriculum of the Outcome Project.
Sources for this presentation include conference presentations by members of the ACGME, information on the ACGME website and others.
Craig McClure, MD
Educational Outcomes Service Group
University of Arizona
December 2004

2. Current Problem Increasing public concerns with quality and safety.
Variable patterns of care that are not based on medical science.
Poor quality of interpersonal “service.”
Public encounters difficulty in assessing physician competence (initial and continuing ) and judging quality.
This slide provides a quick reminder of some of the national issues that led to the ACGME responding with the Outcome Project. One of the key reference is the 2001 Institute of Medicine report “Crossing the Quality Chasm: A New Health System for the 21st Century”

3. The ACGME Mission
To improve the quality of health care in the United States by ensuring and improving the quality of graduate medical educational experiences for physicians in training.
The ACGME mission as on this slide makes clear the mandate for this national body to respond to concerns about medical education and medical care in the US.. Fulfilling this critical mission led to the Outcome Project.

4. Problem Plus Mission
ACGME responded to the challenge by changing focus to:
How well do we learn what is being taught
How well do we practice what we learn?
Evaluations of interventions are the most common type of study submitted for publication to Academic Medicine. Evaluations of teaching tends to focus on the reaction of learners to the intervention.  The Outcome Project takes it forward to:the learning that was achieved due to the intervention, changes in learners’ behaviors attributed to the intervention, and measurable outcomes attributed to the intervention (what is the effect on quality of patient care).

5. How to change the educational and accreditation system from…
A new way of thinking
How to change the educational and accreditation system from…
Competency
Structure & process
And it’s not enough for the program director to check “yes,” on a Program Information Form. The program will have to show evidence that goals and objectives are being achieved.
How does the program use the results of its assessments to make improvements in the curriculum?

6. Program Goal
OLD: goal was for the Program to comply with the written RRC Requirements
NEW: the Program Director must determine if residents achieve the learning objectives set by the Program.
While the RRC will still want to know if the program has goals and objectives, now the new question is, “whether or not the residents achieve the learning objectives?”

7. Six Domains Medical Knowledge Patient Care Professionalism
Communication and Interpersonal
Practice Based Learning and Improvement
Systems Based Practice
These are the six domains set by the ACGME as the basis for a competency oriented curriculum. Further information is available on the ACGME Outcome Project website.

8. Purpose of Assessment
Assess residents' attainment of competency-based objectives
Facilitate continuous improvement of the educational experience
Facilitate continuous improvement of resident performance
Facilitate continuous improvement of residency program performance
While the initial focus by most residency programs has been on the assessment of the resident, the ACGME as evident on the Outcome website anticipates continuing evaluation also of the training program. In 2006 to 2011 the ACGME reviews will begin to look at the way data is used to improve the educational process and performance of the residency program itself.

9. Whatever we measure we tend to improve.
David C. Leach, M.D.
Executive Director
ACGME
September 12, 2002

10. Characteristics of good assessment
Measures actual performance
Identifies areas for improvement
Satisfies reasonable request for accountability
Is practical
Is done over time to discern growth
David Lynch, MD ACGME Executive Director September 2002
Good assessment techniques are ones that are practical in terms of needed resources and time commitment to utilize the technique, are repeated to document movement in behavior toward the criteria for competence and measures the behavior itself rather than some proxy (e.g. actual communication with a patient rather than written response to a scenario). Accomplishing these goals provides documentation for accountability for the educational program and by comparison to criteria reveals areas for improvement.

11. Types of Evaluation Formative Summative
Improve performance
Summative
Note achievement
From ACGME, Introduction to Outcome Project by Doris A. Stoll, PhD RRC Executive Director.
Formative feedback occurs longitudinally through a learning experience and by informing the learner of the gap between observed behavior and criteria permits the formation of competent behavior. Summative evaluation occurs at the conclusion of a learning experience and summarizes the current stage of performance at that time.
Both types of evaluation can be used to evaluate either an individual or a program.

12. Characteristics of good assessment
Systematic
Dependable
Comprehensive
Congruent
Practical
From ACGME, Introduction to Outcome Project by Doris A. Stoll, PhD RRC Executive Director.
A good assessment plan systematically covers all the 6 domains, providing information about all behaviors pertinent to competent practice. The good plan measures what it is required to and does so in multiple settings and a various times. There is congruence between the technique and the behavior being evaluated (e.g. a written test measures cognitive knowledge but is not appropriate for measuring communication skills).

13. Characteristics of good assessment (continued)
Makes professional practice more transparent
Deconstructs the role of physician
Clarifies levels of expertise by distinguishing functional levels
From ACGME, Introduction to Outcome Project by Doris A. Stoll, PhD RRC Executive Director.
If assessment begins by identifying the elements of practice to be measured then the evaluative process declares the parts of professional practice thought to be important. The role of physician is deconstructed to reveal the individual processes that, synthesized, describe behavior of a competent physician. In identifying the competencies expected for each step along the way to being a practicing physician, the level of expertise anticipated per level of training is highlighted.

14. Characteristics of good assessment (continued)
Measures actual performance
Identifies areas for improvement, i.e., self, others
Satisfies reasonable requests for accountability
From ACGME, Introduction to Outcome Project by Doris A. Stoll, PhD RRC Executive Director.

15. Traditional Evaluation
Global
End of rotation
Subjective
Anchored to norms seen by attending (therefore variable)
“I like/didn’t like the resident”
Focused on rotation goals (not movement toward competency)
Typically the traditional assessment reflected the impression of a single teaching physician about how the resident compared to other residents with whom the physician had experience. Sometimes the comparison was to the “best possible” behavior and generally which standard was used was not explicit.

16. Outcome Based Evaluation
Formative, focused on specific competencies required for a physician
Measure the full scope of professional characteristics from very specific procedures to skills involving a synthesis of component abilities
Specific evaluative techniques chosen to match the skill being assessed
The outcome based evaluation begins with identification of the behaviors felt to be importance to the competence of a practicing physician in that specialty. The type of evaluation technique chosen for each skill is determined to be the best measure for that skill. Assessment occurs frequently and specifically enough that the resident is able to adjust his behavior to a closer approximation to the criteria defining competence.

17. Assessment Tools (The Toolbox)
360° Evaluation Instrument
Chart Stimulated Recall Oral Exam (CSR)
Checklist Evaluation of Live or Recorded Performance
Objective Structured Clinical Exam (OSCE)
Procedure, Operative or Case Logs

18. The Toolbox (continued)
Patient Surveys
Portfolios
Record Review
Simulations and Models
Standardized Oral Exams
Standardized Patients (SP)
Written Exams (MCQ)

19. Evaluation Method OLD: global checklist format
NEW: Type of evaluation chosen specifically to measure the chosen skill drawn from the 6 domains
The majority of the time the traditional evaluation method was the global evaluation tool. No matter how many questions on the form, the information obtained tended to 1) clinically competent or not 2) the preceptor did or didn’t like the resident. The global evaluation tool is most appropriate for identifying the resident whose behavior deviates significantly from the norm. In the Outcome Model the type of evaluative instrument, the timing of assessment, the person(s) performing the evaluation all are chosen specifically to evaluate the competency of the resident in one of the 6 domains as compared to a set of criteria defining competence.

20. Frequency of Evaluation
OLD: once per rotation
NEW: multiple intervals assessing component behaviors as well as the integrated practice of medicine.
While sometimes there would be workshops or smaller scale learning activities with associated feedback, typically the traditional model provided an evaluation just at the conclusion of the rotation. In the Outcome format, there likely will still be an overall assessment of the ability to practice as a synthesis of the 6 domains, but in addition there will be periodic evaluations of components of medical practice such as specific procedural techniques, the ability to get an accurate history, communication between consultant and referring physician, ability to work in a team ,patient advocacy, etc.

21. Timing of Assessment OLD: End of rotation
NEW: Timing chosen to facilitate evaluation of a specific competency
Most often an evaluation traditionally was completed at the end of the block rotation with a single toll for the whole experience. The competence driven model will utilize a variety of tools as some measure one domain better than another. It may be that some skills will be measured over time (e.g. patient communication skills). It may be that some specific (e.g. procedural) skills will be measured using a checklist for elements defining the competent performance of the skill.

22. Anchors for Evaluation
OLD: Most frequently the preceptor evaluated the resident against the norm of previous residents in that experience
NEW: Criteria defining competence are utilized as the standard against which resident performance is measured
Varying standards among attendings led to variable evaluations that had more to do with different attending expectations than any alteration of resident performance. Comparison to a norm together with the reluctance to give a critical evaluation led to the majority of residents being judged as “above average.” When behavior is compared to set criteria, the measure is more objective and potentially all residents may be judged competent at some point.

23. Target of Evaluation
OLD: at best tended to address the resident’s success at the goals for the rotation
NEW:Criteria for evaluation describe the qualities of the competent physician, so are more wide ranging or more specific
Most often the traditional evaluation reported on how the resident accomplished the goals of the rotation, which were not always communicated to the resident. In the Outcome paradigm, the comparison should be to the behavioral criteria expected of a competent physician. Some assessments will be very focused: e.g. the ability to obtain a problem-focused history. Some assessments will be broader: e.g. the ability to synthesize information from history, physical and diagnostic data into an effective assessment and plan.

24. Number of Evaluators OLD: typically one per rotation
NEW: multiple, both physician and non-physician evaluators
Although the 360 degree assessment technique has been widely used in industry, its applicability to graduate medical education remains more theoretical than proven. Nevertheless, the intent is that not only the traditional source of evaluation, the attending, will be use, but comprehensive assessment will draw information from patients, nurses, clerical personnel and the resident himself to provide information in assessing skills in the 6 domains.

25. Other Outcome Characteristics
Authentic
More Individualized
Reflection and Self-knowledge Critical
Some of the characteristics of the criteria driven curriculum do not lend themselves readily to a one to one comparison with the former teaching schema.

26. “Authentic”
Justification for elements included in the curriculum is that competence as a practicing physician requires that skill, knowledge or attitude
Evaluation is of the actual skill, knowledge or attitude used by practicing physicians

27. More Individualized
A principle of a criteria-driven physician curriculum is that everyone can become competent with sufficient exposure
Residents obtain skills at different rates with requirements for disparate learning experiences
An optimal outcome-driven system would have an intake assessment followed by an individualized program of study

28. Reflection and Self-knowledge Critical
Criteria for competence are provided to the learner
Impetus for improvement arises from desire to narrow the gap between criteria and performance
Accurate self-assessment is essential to the resident gauging personal performance

29. In Summary Traditional method: Outcomes-based: Not systematic
Subjective & Normative based
Global rotation end
Outcomes-based:
Systemic and comprehensive
Based on criteria defining competence
Multiple measures and intervals