1. Measuring Change Evaluation Phase EME 6636 Change Management Instructor: Dr. Darabi Yang Xu

2. Fig.1: Integrated SAAPS with the HPT model

3. Align Structure  Staffing, development, measurement, rewards, communication, organization structure Optimize results:  Change integration  Workforce measure  Workforce measures questionnaire  Lessons learned Fig.2: Kate Nelson & Stacy Aaron, Change Guide LLC

4. TypeTimingPurpose FormativeDuring intervention analysis, design, and development phases  Improve the intervention selection/design and development process and outputs  Verify the alignment of design and development with the outcomes of the analysis phase SummativeEach time the intervention is implemented  Prove the immediate efficiency and effectiveness of the intervention  Predict the impact and value of the intervention ConfirmativeAfter several implementations  Prove the ongoing, long-term efficiency, effectiveness, impact, and value of the intervention MetaConcurrent with or after evaluation activities  Verify the reliability and validity of the evaluation process, outputs, and outcomes Joan Dessinger, Full-Scope Evaluation--Are You Doing It?

5. Keep track the process of change, and identify the effectiveness of process Identify the validity (outcome) of change Identify the long-term impact of change, and gain supports for future The purpose of measuring change

6. Kirkpatrick’s 4-level evaluation LevelMeasures 1. Reaction and Planned Action Participant’s reaction to the program and outlines specific plans for implementation 2. LearningSkills, knowledge, or attitude changes 3. Application Changes in behavior on the job and specific application of the training material 4. Business ImpactBusiness impact of the program

8. 5-level framework for evaluation LevelMeasures 1. Reaction and Planned Action Participant’s reaction to the program and outlines specific plans for implementation 2. LearningSkills, knowledge, or attitude changes 3. Application Changes in behavior on the job and specific application of the training material 4. Business ImpactBusiness impact of the program 5. ROIMonetary value of the results and costs for the program, usually expressed as a percentage

9. The 2003200420052006 30 50 70 120 The comparision between cost and benefits

10. Control groups Trend line analysis Forecasting Customer input Participants’ and supervisors’ estimates of intervention’s effects Expert estimation Evaluation purpose Evaluation instruments Evaluation timing Evaluation levels

11. 10 Steps to establish a results-based monitoring and evaluation system Jody Zall Kusek, Ray C. Rist, The world bank Plan ImplementMaintain

12. Objectives Links activities and their resources to objectives Performance indicagtors and targets Collect data Reports to managers and alert them problems Analyze results Assess specific causal contributions Examine implementing process Explores unintended results Recommendation Monitoring Evaluation

13. Readiness Assessment Need for assessment Capability for assessment

14. Agree on outcomes to monitor and evaluate 1.Identify the key stakeholders 2.Identify the stakeholders’ concerns 3.Transfer problems into measurable expected outcomes

15. Selecting key indicators to monitor outcomes Clear Monitorable Adequate Economic Relevant

16. Setting baselines and gathering data in indicators

17. Planning for improvement— selecting results targets By Jan. By Feb. Results targets The salary level will increase 5%. The salary level will increase 15%.

18. The systematic model Plan

19. Monitoring for results 1.Data sources 2.Data collection methods 3.Data collection frequency 4.Personnel’s Responsibility

20. Key criteria for collecting date Reliability ValidityTimeliness

21. Reporting findings Title in here To whom Title in here Which format Title in here For what 1.Demonstrate accountability 2.Convince and gain support 3.Document 4. Promote understanding Purpose of reporting

22. The systematic model Plan ImplementMaintain

23. Barriers in measuring change 1.Time consuming 2.Expensive 3.Complicated process 4.Negative evaluation results