Assessing Your Learner Lawrence R. Schiller, MD, FACG Digestive Health Associates of Texas Baylor University Medical Center, Dallas

Why do we assess trainees? It is an essential part of the learning process – Teacher needs to know what needs to be taught – Learner needs to know strengths to build on, weaknesses to be corrected It provides “quality control” for education – Effectiveness of teaching – Appropriateness of content

Education Science Cognitive psychology and neuroscience provide insights into learning process – Principles for learning – “Experience is the best teacher” (if guided) – Learning with understanding More than memorizing facts Based on pre-existing knowledge – Active learning Learner is invested in process Learner takes control of own education

Education Science Cognitive psychology and neuroscience provide insights into learning process – Principles for learning – “Experience is the best teacher” (if guided) – Learning with understanding More than memorizing facts Based on pre-existing knowledge – Active learning Learner is invested in process Learner takes control of own education EVIDENCE-BASED TEACHING

Skills-attainment Continuum Novice Advanced beginner Competent Proficient Expert After Dreyfus

How experts differ from novices Experts notice features and meaningful patterns of information that are not noticed by novices Experts have acquired content knowledge that is organized with deep understanding of field Experts’ knowledge reflects contexts of applicability (conditioned by circumstances) Experts can retrieve knowledge with little attentional effort

Adaptive Expertise “Metacognition”: thinking about how you are thinking about a given problem “Internal conversation” focusing on what additional information is needed for effective decision-making Self-assessment of how one is doing in dealing with a clinical issue

Skills-attainment Continuum Novice Advanced beginner Competent Proficient Expert After Dreyfus

Elements of Competence Deep foundation of factual knowledge Conceptual framework for applying knowledge Organization of knowledge to facilitate retrieval and application (Adequate technical/motor skills)

Types of Assessment Formative (feedback) – Measure of knowledge and/or skills Summative (appraisal/global assessment) – Integrated review of all aspects of work – May incorporate targets for future performance

Purposes of Assessments Evaluating pre-existing knowledge Monitoring learning Determining level of achievement Profiling strengths and weaknesses Assuring that predetermined minimal qualifications are met Deciding about promotion or repetition of course Selection of best candidates for program FORMATIVESUMMATIVE

Educational Objectives Expected results of learning, not methods of instruction – Should be observable, measurable, feasible Standards for assessment should be set ahead of time and discussed with learner – Norm-referenced (comparison to peers) – Criterion-referenced (absolute standard) – Limen-referenced (minimal threshold set)

Miller, 1990 Assessment Tools “No single assessment method can provide all the data required for judgment of anything so complex as the delivery of professional services by a successful physician.”

Assessment Tools Validity – The extent to which an assessment measures what it is supposed to measure Domains of validity – Content validity – Concurrent validity – Predictive validity – Construct validity – Face validity

Assessment Tools Written tests – Progress testing – Mastery testing Oral examinations In-training assessment Objective structured clinical exam (OSCE) Global rating scales – Behaviorally anchored scales – Likert scales

Workplace-based Assessments Checklists Milestones/Logs Mini-clinical evaluation exercise (mini-CEX) Multisource feedback (360 o evaluations) Directly observed procedural skills (DOPS) Procedure-based assessments (PBA) Case-based discussions Significant event analysis Chart audits Patient satisfaction surveys Review of recorded patient visits Portfolio reviews

ACGME General Competencies Patient Care Medical Knowledge Practice-based Learning and Improvement Interpersonal and Communication Skills Professionalism Systems-based Practice

Review with Learners Plan for the session – Collect data – Organize thoughts – Review standards against which trainee is to be judged Decide whether assessment is formative or summative Consider emotional intelligence aspects – The capacity for recognizing our own emotions and those of others for motivating ourselves and others, and for managing emotions well in ourselves and others to enhance work performance

Emotional Intelligence EI Competencies are learned capabilities Self-Awareness – Emotional self-awareness, accurate self- assessment, self-confidence Self Management – Self-control, transparency, optimism, adaptability, orientation to achievement, initiative Social Awareness – Empathy, organizational awareness, altruism

Emotional Intelligence Relationship Management – Developing others – Inspirational leadership – Influence decisions of others – Catalyze change – Manage conflict – Promote teamwork and collaboration

Emotional Intelligence Relationship Management – Developing others – Inspirational leadership – Influence decisions of others – Catalyze change – Manage conflict – Promote teamwork and collaboration YOU NEED TO KNOW YOUR LEARNER

Review with Learners Maintain a relaxed, positive atmosphere Private and confidential; timely Descriptive rather than judgmental Focus on observed behaviors, not intentions or interpretations; be specific rather than general Involve learner in dialogue, listen to what learner says (it provides the basis for action). Help the learner understand any problems and make plans to address issues in the future

Questions?

Summary Assessment is the process of measuring an individual’s performance during education Assessments can be – Formative: designed to improve performance – Summative: designed to assess competence Timely and appropriate assessment shared honestly with the learner is the best way to improve future performance