Leading (and Assessing) a Learning Intervention IMPACT Lunch and Learn Session August 6, 2014 Facilitated By Ozgur Ekmekci, EdD Interim Chair, Department of Clinical Research and Leadership School of Medicine and Health Sciences The George Washington University 1

Overview Leading a Learning Intervention In today's complex social environment, anyone who has been assigned the task of leading a learning initiative has to approach the process in a systematic way in order to succeed. The crucial success factor is often creating alignment amongst components that constitute a well-articulated learning intervention proposal. This session is offered for those who would like to better understand the four key questions that allow educational leaders to effectively envision, design, implement, and assess a meaningful learning intervention, whether this intervention takes place in the classroom or in the boardroom. Learning Objectives Discuss challenges associated with implementing a learning intervention Analyze major components of an effective learning intervention proposal Analyze four key questions that drive a learning intervention Discuss how assessment fits into the larger context of a learning intervention proposal Review six major assessment methods used in health sciences 2

Learning Community Context (overview of the learners’ location, background, roles, and interests) The Problem Statement (current state of the learners – in measurable outcomes - and why this is a problem) The Statement of Significance (why it is important that the intervention be implemented and who suffers in what way, if not implemented) The Proposed Solution (envisioned learning objectives - including the statement of tangible, measurable objectives, which tie back to the problem statement - i.e., current state) Implementation Time Line (a graphical depiction of the implementation plan, with major milestones along the way) Evaluation Method(s) (how to measure progress against learning objectives and how it will be known when the learners finally get to the envisioned future state) Conclusion (summary of key messages with which to leave the audience) ALIGNMENTALIGNMENT KEY SUCCESS FACTOR Alignment of Major Components 3

Define Measurable Learning Objectives: - Individual Level - Change in behavior - Change in perception - Change in performance - Organizational Level - Change in outcomes - Societal Level - Change in environment What do I want to change, for whom, by how much, and by when? In other words, what will success look like? Learning Intervention Time Line Plan Learning Intervention: - Identify learners - Select content - Develop materials - Design delivery method - Construct learning environment - Develop delivery schedule How will I change what, for whom, by how much, and by when? In other words, how will I achieve success? Design Learning Evaluation: -Identify key variables to be measured at the appropriate level of analysis (i.e., individual, organizational, societal) -Decide on the most appropriate method(s) for conducting formative and summative assessments that measure the key variables -Determine frequency and timing for formative assessment and summative assessments -Plan delivery format, medium, frequency, and timing of feedback to learners -Select quantitative statistical methods that will help determine the effectiveness of intervention How will I know what I am changing, for whom, by how much, and by when? In other words, how will I monitor progress against set objectives that define success? Implement Learning Intervention and Conduct Learning Evaluation: - Deliver content - Assess learning - Provide feedback to learners - Analyze data using statistical methods - Determine effectiveness of learning intervention - Declare success! Has what I wanted to change, for those whom I wanted to change it, actually changed by the amount I had wanted it to change? Conduct Formative Assessment Conduct Summative Assessment Review and Revise Content Method or Schedule Deliver Content START Provide Learner Feedback STOP Delivery Complete ? Conduct CAT Determine Effectiveness of Intervention YES NO Learning Intervention Timeline 4 PHASE I PHASE II PHASE IIIPHASE IV Ekmekci, O. (2013). Being there: Establishing instructor presence in an online learning environment, Higher Education Studies. 3(1)

Design Learning Initiative Evaluation How will I know what I am changing, for whom, by how much, and by when? In other words, how will I monitor progress against set objectives that define success? By asking this question, the leader will be able to identify key variables to be measured, decide on methods and frequency for formative and summative assessment, plan delivery format, medium, frequency, and timing of feedback to participants, and select statistical methods that will help determine the effectiveness of the initiative Reflecting on Four Key Questions 5 Define Measurable Learning Objectives What do I want to change, for whom, by how much, and by when? In other words, what will success look like? By asking this question, the leader will be able to contemplate and quantify the envisioned type of change, its scope (in terms of breadth, depth, and timeline), and the level at which change is to take place (i.e., individual, organizational or societal) – which will help ability to monitor progress against objectives PHASE I Plan Learning Initiative How will I change what, for whom, by how much, and by when? In other words, how will I achieve success? By asking this question, the leader will more easily identify participants, select content, develop materials, design delivery method, construct learning environment, and develop delivery schedule PHASE II PHASE III Implement Learning Initiative & Conduct Evaluation Has what I wanted to change, for those whom I wanted to change it, actually changed by the amount I had wanted it to change? By asking this question, the leader will be able to effectively deliver content, assess progress, provide feedback to participants, and analyze data using statistical methods to ultimately determine how successful the initiative has been PHASE IV Ekmekci, O. (2013). Being there: Establishing instructor presence in an online learning environment, Higher Education Studies. 3(1)

Assessment as a Process 6 Assessment is the ongoing process that allows one to: establish clear, measurable expected outcomes of learning; ensure learners have sufficient opportunities to achieve those outcomes; systematically gather, analyze, and interpret evidence to determine how well learning matches expected outcomes of learning; and using the resulting information to understand and improve learning. Downing, S.M. & Yudkowsky, R. eds. (2009). Assessment in health professions education. New York: Routledge. p. 4.

Threats to Validity 7 Construct Underrepresentation (CU) Not measuring the right things: under-sampling of the achievement domain – such as tests that are too short to support legitimate inferences to the domain, trivial questions written at low- levels of the cognitive domain, or mismatch of sample to domain Construct Irrelevant Variance (CIV) Not measuring (the right) things in the right manner: erroneous inflation or deflation of test scores due to certain types of uncontrolled or systematic measurement error – such as poorly crafted test questions, rater bias, or environmental conditions Downing, S.M. & Yudkowsky, R. eds. (2009). Assessment in health professions education. New York: Routledge.

Levels of Assessment and Matching Methods 8 Behavior Cognition DOES Requires methods that provide an assessment of routine “real” performance for the learner SHOWS HOW Requires methods that allow learner to demonstrate the integration of knowledge and skills that will translate into successful routine performance KNOWS HOW Requires methods that provide an assessment of how well learner uses knowledge in the acquisition, analysis, and interpretation of data and the development of an approach or plan KNOWS Requires methods that provide an assessment of how well learner has acquired the fundamental knowledge on which to build higher levels of knowledge and behavior Written Test Oral Examination Observational Assessment Performance Test Simulation Adapted from: Downing, S.M. & Yudkowsky, R. eds. (2009). Assessment in health professions education. New York: Routledge. Portfolio

Wrap Up 9 Questions?