Evaluation: A Challenging Component of Teaching Darshana Shah, PhD. PIES

Next 45 minutes………. Learning Objectives – Briefly review the characteristics of evaluation – Discuss types of evaluation instruments – Understand the framework for selecting and developing assessment methods

What is Evaluations?

Evaluation?????? Who Evaluates Whom?

Different Settings

Purpose of Evaluation Well integration of Teaching, Learning & Assessment

Evaluation/ Assessments  Assessments = Observations /measurements  Individual appraisal  Evaluation = interpretations / conclusions  Appraisal of all activities, curriculum, process outcomes, student learning and teacher effectiveness

Types of Evaluations  Formative evaluation- provide feedback so the learner can modify their learning approach  Summative evaluation- done to decide if a student has met the minimum course requirements (pass or fail)-

Purpose of Evaluation  To certify individual competence  To assure successful completion of goals/objectives  To provide feedback  To students  To faculty, course and clerkship directors  As a statement of values (what is most critical to learn)  For Program Evaluation- evaluation of an aggregate, not an individual (ex. average ability of students to perform a focused history and physical)

Consequences of evaluation  Steering effect- exams “drive the learning”- students study/learn for the exam  Impetus for change (feedback from students, Executive Curriculum, LCME)

Characteristics of Evaluation Goal oriented Value oriented Comprehensive Diagnostic valid and reliable

Blooms Taxonomy Cognitive: mental skills (Knowledge) Affective: growth in feelings (Attitude) Psychomotor: manual or physical skills (Skills)

FORMATIVESUMMATIVE WrittenObserved MCQ MEQ Application Essay Practical Note review SP OSCE Oral Longitudinal Clinical Observation KNOWS HOW SHOWS HOW DOES Erica Friedman’s presentation

Take 5 mins…….  Recall; the goals and objectives for your course or clerkship in the context of assessment  Identify the best methods of assessing your goals and objectives

Purpose of Evaluation Well integration of Teaching, Learning & Assessment

How to Match Assessment to Goals and Teaching Methods  Define the type of learning (lecture, small group, computer module/self study, etc)  Define the domain to be assessed (knowledge, skill, behavior) and the level of performance expected (knows, knows how, shows how or does)  Determine the type of feedback required

Classification Instructional Objectives Learning Activities/ Teaching Method Assessment Methods knowledge Recalling specific information Recognize List Describe Identify Retrieve Name Locate/Find Graphic Organizers Mnemonics Crosswords Puzzles Jeopardy or other games Note Taking Flash Cards Simple Multiple Choice True/False Fill in the blank Label Diagram KNOWS

Explaining ideas or concepts Interpret Summarize Infer Paraphrase Classify Explain Attribute Concept mapping Short Case Cooperative Task Write, Pair, Share Role Play Matching Games Short Case Presentation Short Answer Concept Mapping Short Presentation Poster Session MCQ with vignettes ClassificationInstructional Objectives Learning Activities/ Teaching Method Assessment Methods KNOWs HOW

ApplyingImplement Carry out Use Execute Operate Take Perform Simulation Lab Discuss Rounds Cases Standardized Patients/ Virtual Patients Chart Review Teaching Field Trips Demonstration Case Study SOAP 360 Assessment Patient/family Assessment OSCE MCQ with vignettes

Analyzing Breaking information into parts to explore understandings and relationships Analyse Compare Contrast Organize Deconstruct Outline Structure Integrate Collaborative Tasks PBL Spreadsheets Research Project 5 Minute Preceptor Writing a Wiki Reflection Problem Solving Open Ended Questions Complex Case Study Concept mapping Survey Report Paper Project Report Group Presentation Grand Rounds Create a Mind Map Reflective Writing Problem Identification Extended Matching MCQ

How we currently assess ourselves  Student evaluations (quantitative and qualitative)- most often summative  Performance of students on our exam and specific sections of USMLE

Things to consider!!!!!  Focus and feedback groups (formative)  Self evaluations- yearly grid completed by course directors and core faculty  Consider peer evaluation of teaching and teaching materials

How to be successful  Students should be clear about the course/clerkship goals and the specifics about the types of assessments used and the criteria for passing  Make sure the choice of assessments is consistent with the values of your course and the school  Final judgments about students’ progress should be based on multiple assessments using a variety of methods over a period of time (instead of one time point)

Critical factors for choosing an evaluation tool  Type of evaluation and feedback desired: formative/summative  Focus of evaluation: Knowledge, skills, behaviors (attitudes)  Level of evaluation: Know, Knows how, Shows how, Does  Pros/Cons: Validity, Reliability, Cost (time, $ resources)

A Brief Overview of Techniques For Classroom Setting Assessing Course related knowledge and skills: – Background Knowledge Probe – Minute Paper – Empty outline – Memory Matrix – Focus listing