From the graphic point of view, P. D. C. A From the graphic point of view, P.D.C.A. is represented by a moving circle called the Deming wheel. The movement stands for dynamism and continuity of the application process. P D C A CYCLE The Deming Wheel From the graphic point of view, P.D.C.A. is represented by a moving circle The movement stands for dynamism and continuity of the application process. 11 http://www.iwolm.com/en/the-pdca-method-or-deming-wheel-for-your-improvement/

Theoretical Frameworks Utilization Focused (Patton) Evaluation of effectiveness by potential users Who might use What will be used Intention to use 12

Evaluation Techniques

Logic Models “A graphic representation of a program that describes the program’s essential components and expected accomplishments and conveys the logical relationship between these components and their outcoms.” (Conrad et al. 1999) Logic models guide evaluation http://www.uwex.edu/ces/pdande/evaluation/evallogicmodel.html 14

How the Logic Model Guides Evaluation Provides the program description Aids in matching evaluation to the program Identifies what and when to measure Are you interested in process and/or outcomes? Keeps focus on key, important information What do we really need to know?? Where should limited evaluation resources be used? 15

Basic Logic Model INPUTS OUTPUTS OUTCOMES Activities Participants 16

Input Inputs Investment/Resources Time People Money Materials Products (teaching components) 17

Output Activities Participation What is done Who receives Training/teaching Counts Products (teaching components) Description Develop resources Satisfaction Form collaborations/partnerships Assessment tools and assessment 18

Outcomes Primary Secondary Tertiary Immediate Behavior Results (Short term) Behavior Results (Long-term) Knowledge Teaching Adaptation Skills Practice Standardization Abilities Improved practices Changes in attitudes Assessments (tools and assessment) 19

When to evaluate? Before the experience/event/class Mini-assessments within (quiz, exercise, etc) Post-test only Pre-test and Post-test Retrospective Pretest and Post-test Pre-test, Post-test, and follow-up Intermediate testing Can be combined with pre and post test designs 20

Evaluation Design Experimental Quasi-experimental Non-experimental Uses random assignment, control Quasi-experimental Groups from “natural” characteristics Males to females Class A to Class B Non-experimental Compares before and after 21

Types of Assessment Qualitative Quantitative Mixed Method Open-ended Constrained Choice Mixed Method Using both quantitative and qualitative techniques to collect data Post-test: advantages – quick Retrospective Pretest: both pre and post collected following the experience – simple, gives idea of “improvement” or change Pretest/posttest: same instrument at two points – shows actual change, controls for prior knowledge, better evidence of effectiveness of experience However, takes more time, doesn’t control for intervening events, Pre, post, follow-up – allows to see longer term impacts 22

Mixed Methods Evaluation Enhances both formative and summative evaluation Triangulation from combining both types of evaluation Merging Explanatory Exploratory 23

Mixed Methods Evaluation - Merging Triangulation from combining both types of evaluation Merging Quantitative Qualitative Interpretation 24

Mixed Methods Evaluation - Explanatory Triangulation from combining both types of evaluation Explanatory Quantitative, then Qualitative, followed by interpretation Quantitative Identify Issues Qualitative Interpretation 25

Mixed Methods Evaluation - Exploratory Triangulation from combining both types of evaluation Exploratory Qualitative Design Measure Quantitative Interpretation 26

Mixed Methods Evaluation Triangulation from combining both types of evaluation Merging Explanatory Quantitative, then Qualitative, followed by interpretation Exploratory Quantitative Qualitative Interpretation Quantitative Identify Issues Qualitative Interpretation Qualitative Design Measure Quantitative Interpretation 27

Selecting the Evaluation Method Participation records Self-report Achievement (knowledge) tests Interviews Focus groups Direct observations Medical record reviews Product count 28

Example 29

Example 30

Examples 1.1 The percent of adults aged 65 and older in the United States is currently approximately ________ and will increase to ________ by the middle of the century. a. 10% 25% b. 12% 20% c. 12% 15% *d. 13% 20% e. 13% 25%   1.2 Persons reaching age 65 have an average life-expectancy of an additional _____ years. a. 5 b. 9 c. 10 d. 15 *e. 19 31

Examples: 32

Example 33

Evaluation: Quality Improvement vs. Research Institution-specific Reporting to stakeholders is not research Funders Collaborators UAB Institutional Review Board defines research as any presentation in a local or national forum IRB approval is needed 34

American Evaluation Association Founded 1986 http://www.eval.org/p/cm/ld/fid=1 35

Evaluation Strategies for Educational Interventions November 1, 2013 GEC Faculty Scholars Program Patricia Sawyer, PhD