The Competencies Working Group Update: The Performance Framework Rosalyn Scott, VA Timothy Willet, SIM-one Valerie Smothers, MedBiquitous Hi, my name is _________. I’m _______________, and I’m going to walk through some slides created by the MedBiquitous Competencies working group. The group is working to develop technical standards for representing levels of performance with regards to competency achievement, and these slides illustrate what that data might look like for different types of performance frameworks. Our work is funded by the Association of American Medical Colleges and the working group includes membership from medical and nursing schools in the US, Canada, UK and Australia. The data presented is entirely fictional and for illustrative purposes only.

Competency-based learning and assessment is increasingly being applied within health professions education across the continuum.

Mastery Intermediate Novice An important component of competency-based learning is the concept of educational milestones, which are expected levels of performance related to a competency. It is these performance levels that the performance framework is designed to represent.

MedBiquitous Competencies Working Group Thank You! Lastly, we’d like to say thank you to the AAMC for funding the development of the specification and to members of the MedBiquitous Competencies Working Group for contributing their time and expertise to the development of the specification. Thank you! MedBiquitous Competencies Working Group

How competent? Which competency? Competency framework Performance framework Level 1 (eg Novice) Level 5 (eg Mastery) Level 2 Level 3 Level 4 While competencies describe what a professional must be able to do, and are commonly arranged in a hierarchy where more granular sub-competencies compile into broader competencies and competency domains, performance frameworks are used to describe how competent one is or should be. Like a ruler standardizes measurements of length, performance frameworks standardize measurements of competence. Representing these frameworks in a standard format allows them to be referenced from other kinds of data, including learner assessment data, curriculum data, and more.

How competent? Which competency? Competency framework Performance framework Level 1 (eg Novice) Level 5 (eg Mastery) Level 2 Level 3 Level 4 While competencies describe what a professional must be able to do, and are commonly arranged in a hierarchy where more granular sub-competencies compile into broader competencies and competency domains, performance frameworks are used to describe how competent one is or should be. Like a ruler standardizes measurements of length, performance frameworks standardize measurements of competence. Representing these frameworks in a standard format allows them to be referenced from other kinds of data, including learner assessment data, curriculum data, and more. Educational Achievement

Benefits Standardization of measurement across a discipline or specialty Implementation in assessment systems Compilation of performance data across courses/units/systems Exporting for benchmarking and educational research There are many benefits to leveraging technical standards for your performance frameworks. It enables standardization of measurement across a discipline or specialty. Assessment systems can import these frameworks and use them for assessment of learners. When used with other standards, it also enables the compilation of performance data across systems in a robust fashion, allowing one to form a holistic picture of an individual’s performance and its progression over time. Compilation could also be used for overall program evaluation and continuous program improvement. Finally, exporting data that is structured in a standard way enables benchmarking and education research across institutions.

Example 1: Internal Medicine To explain the different parts of the Performance framework specification, we will use the Internal Medicine Milestone Project as an example. Note that the Performance Framework Specification is generic and can be used to represent any performance framework or milestones; we are using the Internal Medicine Milestones as an example only. A link to the full document is provided in case you would like to view it in its entirety. http://www.acgme.org/acgmeweb/Portals/0/PDFs/Milestones/InternalMedicineMilestones.pdf

Metadata Performance Scale Component Component Component Component Title, effective date, authors, supporting information Performance Scale ie 1 to 5, 5 being most competent Component References a competency and defines levels of performance using behavioral indicators Component Components can be nested to reflect instances where complex competencies are broken down into component parts for evaluation Component Component

Here is one of the pages describing levels of performance related to a specific competency. This document calls these performance levels milestones. Keep in mind that this is one page of the document; there are tables describing performance levels for many other competencies.

Component The set of information that defines a spectrum of performance levels for a competency or topic is called a component in the MedBiquitous specification. Most components have a competency or topic and a set of performance levels for that competency or topic. For a particularly complex competency or topic, there may be nested components, breaking the measurement of that competency or topic into more granular facets.

Competency (reference to a Competency Object) At the top of the table, you see an example of one Competency: “Gathers and synthesizes essential and accurate information to define each patient’s clinical problem.” Within the specification, this is represented as a reference to a Competency Object. A Competency Object is an umbrella term used to describe any abstract statement of learning or performance expectations, and information related to the statement. A MedBiquitous specification exists for representing Competency Objects.

Performance Level Underneath the competency is a set of columns. Each column represents a level of performance in relation to that competency – which the specification calls a performance level. This table has 5 performance levels.

Least competent Most competent From left to right, the levels progress from least competent to most competent.

Performance Scale Least competent Most competent 1 5 Performance Scale In the MedBiquitous standard, we allow for the representation of numeric scales for these performance levels. Because there are 5 levels of performance, this example would probably use a performance scale of 1 to 5, with 1 being least competent and 5 being most competent.

Scores Least competent Most competent 1 5 Performance Scale Score 1 2 Then a score is assigned to each performance level so that it’s clear how the level fits on the scale. This makes it easier to convey learner performance data. For example, I could say that a learner has a score of 3 with regards to a specific competency using a specific performance framework. I could also say that the learner is somewhere between levels using decimal points. So a learner may be a 3.5 if he is between levels 3 and 4. If the learner score reflects the synthesis of several assessments, a learner may have a decimal score like 3.44. So while the score points defined in the performance framework specification must be integers, they do not constrain the scores that can be assigned to learners. The performance framework gives greater meaning to that score by defining upper and lower limits as well as detailed descriptions of each performance level in between. Score 1 2 3 4 5

Indicators Least competent Most competent 1 5 Performance Scale 1 2 3 Finally, within each performance level, there are one or more indicators or behavioral markers. These indicators define what performance at a particular level looks like, in terms of observable behaviors. These are the key parts of the Performance Framework specification. Now let’s look at some other performance framework examples that are a little different. The MedBiquitous’ Performance Framework Specification is capable of representing these, too. 1 2 3 4 5

Example 2: Transformation in Medical Education (TIME) The University of Texas system’s Transformation in Medical Education (or TIME) initiative has defined Transition milestones that learners must demonstrate before moving from the general academic or college campus to the medical campus. There is a link to more information about the TIME initiative. http://www.utsystem.edu/initiatives/time/

Thresholds For each Transition Milestone, TIME has defined what we would call a Threshold in the specification. A threshold describes a minimum acceptable score for a particular action or an evaluator decision, such as progression in training or entrustment. In this case, the threshold is the minimum acceptable score for a learner to transition to the medical campus. So for the Transition milestone “Explain how the principles of the Physician Charter apply to student behavior,” students are required to demonstrate achievement at or above performance level 2, which is defined as “demonstrate application of knowledge, skills, behavior to relevant healthcare problems.”

Example: An Advanced Practice Nursing Program Competency The framework was designed to support all health professions. Here we see an example performance framework from an advanced practice nursing program the Hahn School of Nursing and Health Science at University of San Diego. In this example, we see a consistent set of performance levels applied across many competencies. This example includes a complex competency that would be represented as nested components. Performance levels

MedBiquitous Process Approves new standards projects Meets via telco, in person Develops specifications Consensus body Votes Final approval Executive committee Working groups Standards committee ANSI

Standards Committee review (30 days) Public Review (45 days) Ballot (6 weeks)

More information Specification and Schema http://medbiq.org/std_specs/specifications/index.html#performance_framework Contact Valerie Smothers vsmothers@jhmi.edu To learn more, download the specification and schema from the MedBiquitous website or contact Valerie Smothers.