Applying the Performance Framework Data Model: A Conceptual Overview October 25, 2013 Hi, my name is _________. I’m _______________, and I’m going to walk through some slides created by the MedBiquitous Competencies working group. The group is working to develop technical standards for representing levels of performance related to a competency, and these slides illustrate what that data might look like for different types of performance frameworks. Our work is funded by the Association of American Medical Colleges and the working group includes membership from medical and nursing schools in the US, Canada, UK and Australia. The data presented is entirely fictional and for illustrative purposes only.

Competency-based learning and assessment is increasingly being applied within health professions education at all levels across the continuum.

Mastery Intermediate Novice An important component of competency-based learning is the concept of educational milestones, which are expected levels of performance related to a competency. It is these performance levels that the performance framework is intended designed to represent.

What competence? How competent? Deficient Novice Intermediate Mastery While competencies describe what a professional must be able to do, and are commonly arranged in a hierarchy where more granular sub-competencies compile into broader competencies and competency domains, performance frameworks are used to describe how competent one is or should be. Like a ruler standardizes measurements of length, performance frameworks standardize measurements of competence. Representing these frameworks in a standard format allows them to be referenced from other kinds of data, including learner assessment data, curriculum data, and more. ** I added the “What competence?” part to further distinguish between the two concepts. ** I removed the point “Competent” in case it causes confusion.

Example 1: Internal Medicine To explain the different parts of the Performance framework specification, we will use the Internal Medicine Milestone Project as an example. Note that the Performance Framework Specification is generic and can be used to represent any performance framework or milestones; we are using the Internal Medicine Milestones as an example only. A link to the full document is provided in case you would like to view it in its entirety. http://www.acgme-nas.org/assets/pdf/Milestones/InternalMedicineMilestones.pdf

Framework Metadata Version 12/2012 As we dissect the Milestone document into its functional parts, in the first few pages of the document, we see some information about the framework as a whole: the title, the date of the version, and some information about the publishers the Accreditation Council for Graduate Medical Education (ACGME) and the American Board of Internal medicine (ABIM), and the authors. In the performance framework specification, this information about the framework as a whole is called Metadata. Version 12/2012

Further in Also near the beginning of the document there is information about the intent and scope of the framework as well as a high level description of the framework overall.

Supporting Information In the MedBiquitous performance framework specification, we call this Supporting Information. The specification allows you to point to or include supporting information for the performance framework.

Here is one of the pages describing levels of performance related to a specific competency – what this document calls a “milestone”.

Framework Component The set of information that defines a spectrum of performance levels for a competency or topic is called a component in the MedBiquitous specification. Most components have a competency or topic and a set of performance levels for that competency or topic. For a particularly complex competency or topic, there may be nested components, breaking the measurement of that competency or topic into more granular facets.

Competency (reference to a Competency Object) At the top of the table, you see a Competency: Gathers and synthesizes essential and accurate information to define each patient’s clinical problem. Within the specification, this is represented as a reference to a Competency Object. Keep in mind that this is one page of the document; there are tables describing performance levels for many other competencies.

Performance Level Underneath the competency is a set of columns. Each column represents a level of performance in relation to that competency – which the specification calls a performance level. This table has 5 performance levels.

Least competent Most competent From left to right, the levels move progress from least competent to most competent.

Performance Scale Least competent Most competent 1 5 Performance Scale In the MedBiquitous standard, we create numeric scales for these performance levels for a specific competency. Because there are 5 levels of performance, this example would probably use a performance scale of 1 to 5, with 1 being least competent and 5 being most competent.

Scores Least competent Most competent 1 5 Performance Scale Score 1 2 Then a score is assigned to each performance level so that it’s clear how the level fits on the scale. This makes it easier to convey learner performance data. For example, I could say that a learner has a score of 3 with regards to a specific competency using a specific performance framework. I could also say that the learner is somewhere between levels using decimal points. So a learner may be a 3.5 if he is between levels 3 and 4. If the learner score reflects the synthesis of several assessments, a learner may have a decimal score like 3.44. So while the score points defined in the performance framework specification must be integers, they do not constrain the scores that can be assigned to learners. The performance framework gives greater meaning to that score by defining upper and lower limits as well as detailed descriptions of each performance level in between. Score 1 2 3 4 5

Indicators Least competent Most competent 1 5 Performance Scale 1 2 3 Finally, within each performance level, there are one or more indicators or behavioral markers. These indicators define what performance at a particular level looks like, in terms of observable behaviours. These are the key parts of the Performance Framework specification. Now let’s look at some other performance framework examples that are a little different. The MedBiquitous’ Performance Framework Specification is capable of representing these, too. 1 2 3 4 5

Example 2: Transformation in Medical Education (TIME) The University of Texas system’s Transformation in Medical Education (or TIME) initiative has defined Transition milestones that learners must demonstrate before moving from the general academic or college campus to the medical campus. There is a link to more information about the TIME initiative. http://www.utsystem.edu/initiatives/time/

Scale & Performance Levels 7-point scale TIME has defined 7 performance levels that define what might be expected of learners for any defined competency. They then apply this one scale and set of performance levels to all competencies within the framework.

Competency Objects TIME defines transition Milestones, behaviors that students are required to demonstrate upon transition to the medical campus. From a technical perspective, these transition Milestones would be represented as Competency Objects within a Competency framework, because they describe different aspects of competency rather than a progression from less to more competent. The Competency framework would show how the Competencies, subcompetencies, and transition Milestones relate to one another.

Thresholds For each Transition Milestone, TIME has defined what we would call a Threshold in the specification. A threshold describes a minimum acceptable score for a particular action or decision, such as progression in training or entrustment. In this case, the threshold is the minimum acceptable score for a learner to transition to the medical campus. So for the Transition milestone “Explain how the principles of the Physician Charter apply to student behavior,” students are required to demonstrate achievement to performance level 2, which is defined as “demonstrate application of knowledge, skills, behavior to relevant healthcare problems.”, level 2 in the TIME performance levels.

Example 3: Diagnostic Radiology Additional Information This is an Here is a third example from the Diagnostic Radiology Milestone project. In this description of performance levels related to Health care economics, the authors provide possible methods of assessment or examples and suggested educational strategies. In the Performance framework specification, these would be represented as Additional Information about the Health care economics component. Components, performance levels, and indicators may have additional information associated with them, to help educators apply the performance levels.

Example 4: Tufts University Score range Single score Performance level Performance level In some cases, a performance level may have not one single score associated with it, but a range of scores. In this example taken from Tufts University, the performance level labeled Below Minimum has a score range of 1 to 3. The MedBiquitous Performance Framework specification allows you to indicate an individual score or a score range for a specific performance level.

Example 5: Nursing at University of San Diego Competency The framework was designed to support all health professions. Here we see an example performance framework from the Hahn School of Nursing and Health Science at University of San Diego. In this example, we see a consistent set of performance levels applied across many competencies. This example includes a complex competency that would be represented as nested components. Performance levels

Benefits Standardization of measurement across a discipline or specialty Implementation in assessment systems Compilation of performance data across courses/units/systems Exporting for benchmarking and educational research There are many benefits to leveraging technical standards for your performance frameworks. It enables standardization of measurement across a discipline or specialty. Assessment systems can import these frameworks and use them for assessment of learners. When used with other standards, it also enables the compilation of performance data across systems in a robust fashion, allowing one to form a holistic picture of an individual’s performance and its progression over time. Finally, exporting data that is structured in a standard way enables benchmarking and education research across institutions.

More information Specification and Schema http://medbiq.org/std_specs/specifications/index.html#performance_framework Contact Valerie Smothers vsmothers@jhmi.edu To learn more, download the specification and schema from the MedBiquitous website or contact Valerie Smothers.

MedBiquitous Competencies Working Group Thank You! Lastly, we’d like to say thank you to the AAMC for funding the development of the specification and to members of the MedBiquitous Competencies Working Group for contributing their time and expertise to the development of the specification. Thank you! MedBiquitous Competencies Working Group