Inter-Rater Reliability Respiratory Ivy Tech Community College-Indianapolis

What Is Inter-Rater Reliability (IRR)? The extent to which two or more raters agree A fair measurement of student competency Addresses the uniformity of the implementation of evaluation systems being utilized

Importance of IRR Required by CoARC Accreditation Standard Requirement: Student Evaluation 3.09, 3.10, 3.11 CoARC Standard 3:09: The program must conduct and document evaluations with sufficient frequency to keep students apprised of the progress toward achieving the curriculum competencies, and to allow immediate identification of learning deficiencies and the development of a means for their remediation in a reasonable time frame. CoARC Standard 3:10: The program must administer evaluations uniformly and equitably to all students in the program for didactic, laboratory, and clinical education components CoARC Standard 3.11: The program must develop processes that facilitate the development of inter-rater reliability among those individuals who perform student clinical evaluations

Importance of IRR IRR requires some level of objectivity IRR provides validation of evaluation results Certify skills competency Assign the appropriate “grade” Increases “confidence” that people are following the guidelines in a similar method

Ensuring IRR—per CoARC Respiratory Programs should: Implement Preceptor training on a regular and repeating basis Training should include: Roles and responsibilities of preceptors Direct observation skills Effective feedback and evaluations Inter-rater reliability exercises (e.g. evaluation of video demonstration) Keep records

Goal of Presentation Goal of presentation Provide insight into the student evaluation process How to perform objective evaluations Achieve consensus amongst raters Describe the IRR process system and how it is used in the Ivy Tech Community College Respiratory Care Program-Indianapolis

Competency Domains CoARC, our accreditation agency states that students must demonstrate competency in 3 domains: Knowledge Technical performance Behavior From an educational perspective this can present a dilemma!

The Dilemma Evaluating student clinical performance and behavior is difficult and can be somewhat subjective How do we remove the subjective? How can we get an objective record of the most meaningful behaviors?

Reliable Techniques Most Educators would agree that the most valuable and widely used techniques for clinical evaluation are: Anecdotal notes Rating scales/procedural evaluations Checklist/clinical objectives

Performance Evaluations Clinical Performance Evaluations of Learning and Development provide a means to DIRECTLY evaluate: Clinical competency Technical performance Behaviors

Validity/Procedure Evaluations How can this be accomplished in a valid way? Clinical paperwork Frequent and timely student evaluations including student feedback Competency check-offs Procedure Evaluations or Competency Check-Off Forms: Tool to evaluate clinical competencies Have clear cut steps on what is being assessed Has a rating scale

Example A student who is performing unsatisfactorily is rated as satisfactory creates the following problems: Student thinks their performance is ok and will continue with poor technique or behavior Students aren’t provided objective feedback, and therefore aren’t aware of their deficiencies of the need to make changes. The ITCC faculty aren’t aware of the defiance's and necessary corrective action can’t be provided ULTIMATELY we have failed the student and profession!

Completing the IRR Exercise You will be viewing four procedures via video The videos and competency check offs may be completed in one time frame or if needed, individually While watching the video complete the competency check off form used in the Indianapolis region Make sure to indicate your name, date, and facility at the top of each competency check off form to document completion for tracking purposes Click on the submit button when competency check-off form is completed. All four competencies are due based on your clinical affiliates deadline, 2014!

Thank You Thank you once again for your support in providing quality respiratory care education Please don’t hesitate to call the Director of Clinical Education (DCE) of the Ivy Tech Community College-Indianapolis Campus at with questions!