Organizational Design, Diagnosis, and Development Session 15 The Evaluation of Change

Objectives To understand issues dealing with the evaluation of change To consider some of the major problems in evaluating change programs To review designs for evaluation

Where Evaluation Fits Diagnosis Design of interventions Implementation of Intervention Implementation Feedback Evaluation Feedback

EVALUATING OD INTERVENTIONS Evaluation to guide intervention -- formative evaluation –Is it reaching right audience? –Is the audience realizing some change? Summative evaluation -- Impact on the organization – Is the organization realizing the effect it desired?

Problems in Evaluating Change Vague goals Strong promises Weak effects Other problems: – Finding an appropriate criterion – Finding skilled personnel –The politics of evaluation –When to evaluate

Levels of Evaluation Reaction Learning Behavior Results

Questions to be Asked Does examination of the criteria indicate change has occurred? Can the change be attributed to the intervention? Is it likely that new participants in the program will demonstrate similar changes? Is it likely that new participants in the same program will demonstrate similar changes in a different site?

Pre-experimental Designs One group: post-test only –I (intervention) T (evaluation) One group: pretest/post-test design –T-1 (pre-test)IT-2 (post-test)

Experimental Designs Pretest/post-test control-group design –E: T-1 (pre-test)IT-2 (post-test) –C: T-1 (pre-test)T-2 (post-test) Solomon four-group design –G1:T-1 (pre-test)IT-2 (post-test) –G2:T-1 (pre-test)T-2 (post-test) –G3:IT-2 (post-test –G4: T-2 (post-test)

Quasi-Experimental designs Time series –T1 T2 T3 T4 I T5 T6 T7 T8 Non-equivalent control group –E: T-1 (pre-test)IT-2 (post-test) –C: T-1 (pre-test)T-2 (post-test)

Utility Analysis Cost benefit analysis Cost effectiveness analysis An example for absenteeism program –document the rate of absenteeism –calculate absenteeism hours lost per year –calculate absenteeism compensation costs –calculate cost of managing absenteeism –calculate other costs of absenteeism

Threats to Evaluation Validity Weak evaluation designs Select groups for introducing intervention

Backwards &Forwards Summing up: In today’s session we examined some problems that surround evaluation, looked at several designs evaluation intervention (pre-experimental, experimental, and quasi experimental designs). Threats to evaluation validity were also discussed Looking ahead: Next time we begin our examination of human process interventions with a look at interpersonal and group process approaches