An Introduction to Understanding the OT and OTA Fieldwork Performance Evaluations (FWPEs) Karen Atler, MS, OTR Colorado State University, Fort Collins, CO Roberta Wimmer, OTR/L Pacific University, Forest Grove, OR  2003 The American Occupational Therapy Association, Inc.

2 Task Force Members Carole Dennis, PhD, OTR Ithaca College, New York Carole Hays, MA, OTR Springfield Hospital Center, Maryland Becky Robler, MEd, OTR Pueblo Community College, Colorado Karen Atler, MS, OTR, Co-Chairperson Colorado State University Roberta Wimmer, OTR, Co-Chairperson Pacific University, Oregon

3 Objectives Describe the Entry-level practice competencies for OT and OTA students Purpose, format, content, and scoring of the companion evaluation forms New concepts and terminology used in the FWPE from the OT Practice Framework Begin to score items on the FWPE

4 Task Force’s Charge Revise/develop evaluation tools to measure assistant and professional Level II fieldwork student performance. Expectations Conduct review of literature across disciplines Synthesize feedback on current AOTA FWE/OT forms Incorporate 1997 NBCOT Practice Analysis results Address identified desired characteristics

5 Desired Characteristics Companion documents for assistant and professional level that Measure entry-level competence Focus on occupation-based practice Reflect current and future practice Can be used in a variety of settings Provide feedback to students Can be easily used in a timely manner

6 NBCOT Practice Analysis 1997 What OTs & OTAs Do Determining needs/priorities for interventions Identifying/designing interventions Implementing interventions Reporting/evaluating intervention effectiveness Providing OT services for populations Managing delivery of OT services Advancing effectiveness of the OT profession What OTs & OTAs Need To Know Human development and performance Principles/strategies in the identification/evaluation of strengths and needs Principles/strategies in intervention/treatment planning Principles/strategies in intervention Nature of occupation and occupational performance Service management Responsibilities as a professional

7 Standards of Practice for Occupational Therapy Identifies minimum standards Identifies key performance areas for the OT and OTA Professional standing and responsibility Referral Screening Evaluation Intervention plan Intervention Transition services Discontinuation

8 ACOTE: Minimum Standards and Outcomes OT Be a generalist Achieve entry-level competence Articulate, apply, and justify occupation interventions Supervise and collaborate with the OTA Keep current with best practice Uphold the ethics, values, and attitudes of the profession Be an effective consumer of research and knowledge OTA Be a generalist Achieve entry-level competence Work under the supervision of and in cooperation with the OT Articulate, apply, and justify interventions related to occupation Keep current with best practice Uphold the ethics, values, and attitudes of the profession

9 Goal of Level II Fieldwork Education for the OT and OTA Student Develop competent, entry-level generalists Include an in-depth experience in delivering occupational therapy services Be designed to promote reasoning, enable ethical practice, and develop professionalism

10 The Process Began with OTA evaluation Reviewed by experienced panel Submitted to COE Made revisions Completed pilot studies (2 OTA, 1 OT)

11 Design and Analysis of Pilot Studies: The Rasch Measurement Model less able Student Ability more able

12 Results of Pilot Studies Good representation in pilot samples Students and educators preferred new form Good scale and response validity Inaccurate use of scale Rating Scale Usage in Pilot Studies II and III 4 = Exceeds Standards 3 = Meets Standards 2 = Needs Improvement 1 = Unsatisfactory Rating Scale Descriptors % Usage OTA% Usage OT

13 The FWPEs for OT and OTA Students Companion documents Terminology Content layout Purpose Design Rating scale Scoring system

14 FWPEs OT and OTA Companion Documents Focus The occupational therapy process The clinical reasoning process Roles and responsibilities of the OT and OTA Structure Collaborative process—student and FW educator Same layout Same rating/scoring system

15 Terminology of the FWPEs Reflects Standards of Practice for Occupational Therapy and ACOTE Education Standards Occupational Therapy Practice Framework The glossary

16 Content Layout of FWPEs Summary Sheet Overview/instructions Organization of items Space for comments—midterm and final Performance Rating Summary Sheet

17 Content of OT and OTA Evaluations OTA Fundamentals of practice (3) Basic tenets (3) Evaluation/screening (5) Intervention (6) Communication (2) Professional behavior (6) OT Fundamentals of practice (3) Basic tenets (4) Evaluation/screening (10) Intervention (9) Management of OT services (5) Communication (4) Professional behavior (7)

18 Primary Purpose of the FWPEs Measures entry-level competence Designed to differentiate the competent student from the incompetent student Not designed to differentiate levels above entry-level competence

19 Purpose (continued) Provides student with accurate assessment of his or her competence for entry-level practice over time Growth occurs over time Midterm and final scores reflect this change Midterm scores: Satisfactory–unsatisfactory Final scores: Pass–no pass

20 Purpose (continued) Provides feedback to student Provides opportunity for student self- assessment

21 Design of the FWPEs The “doing” of the occupational therapy process is evaluated, not the individual tasks in isolation NOT all items are equal in level of difficulty (i.e., simple to complex) Evaluation is supplemented with development of site-specific objectives

22 RASCH Ordering of Items OT Interpersonal; 41 Respnds 2 fdbk 38 Diversity; 42 Ethics; 1 Collab c super;36 Workbeh 39, safety 2 Steps 3,Resp; 37 Timemng40; Legible 34 Client centered interv 22 Occupn based inter 23 Selects rel occu; 21 Produces work; 31 Clear documentation; 33 Org goals 30; verbl 32 Documents intrv; 26 Collaborates clnt; 7 Language reflects 35 Documents eval rslts 17 OTbeliefs 4 Obtain inf 12 Role of OT; 6 Rationale Tx 18, Administer asses 13 Artic ratinal eval; 8 Artic value occ 5 Modifies approach 24 Underst finances; 29 Collaborates OTA 28 Occ prof 10, Adjst ass 14 Est Plan 16, Assess ftr 11 Updates; 25, Evidence 19 Interprets eval reslts; 15 Selects rel asses methd; 9 Assigns resp OTA; 27 HARD

23 Rasch Ordering of Items OTA

24 Rating Scale of FWPEs 4 = Exceeds Standards 3 = Meets Standards 2 = Needs Improvement 1 = Unsatisfactory Performance is highly skilled and self-initiated. This rating is rarely given and would represent the top 5% of all the students you have supervised. Performance is consistent with entry-level practice. This rating is infrequently given at midterm and is a strong rating at final. Performance is progressing but still needs improvement for entry-level practice. This is a realistic rating of performance at midterm and some ratings of 2 may be reasonable at the final. Performance is below standards and requires development for entry-level practice. This rating is given when there is concern about performance.

25 Scoring System of FWPEs Each item must be scored Ethics and safety items must be passed Each item rating recorded on Performance Rating Summary Sheet All items summed up at midterm and final Score compared to scales provided

26 Midterm and Final Scores of the FWPEs OTA Overall Midterm Score Satisfactory: 54 & above Unsatisfactory: 53 & below Overall Final Score Pass: 70 & above No Pass: 69 & below OT Overall Midterm Score Satisfactory: 90 & above Unsatisfactory: 89 & below Overall Final Score Pass: 122 & above No Pass: 121 & below

27 Rating Performance Using the FWPEs Case Scenarios John—OT Evaluation Mary—OT Intervention Sandra—OTA Evaluation/Screening David—OTA Intervention

28 Individualizing the FWPEs Designed for additional objectives to be written to add clarification Site-specific objectives NOT supervisor-specific If an item is very clear and meets the RUMBA test, then there is no need to write another objective

29 Objectives: An Example From FWPE for the OT Student 16. Establishes accurate and appropriate plan School—Provides behavioral-based, measurable OT goals during IEP process Acute Care—Overall intervention plan is achievable within client’s length of stay

30 Summary Evaluations designed to measure entry- level competence, NOT level of performance above competency OT practice examined as a generalist Evaluations reflect the occupational therapy process Performance develops over time