1 CAUSAL ANALYSIS FOR MONITORING SYSTEMS11 Performance Monitoring and Evaluation College of Public and Community Service University of Massachusetts at Boston ©2006 William Holmes

2 CAUSAL ISSUES IN MONITORING Separating true change from false change (spurious effects) Identifying indirect effects on change Identifying competing effects on change Assessing program and staff influences Clarifying linkages influencing change Specifying interactive effects on change (contingent or contextual)

3 CORRELATION ISSUES IN CAUSAL MONITORING 1 Correlation is not Causation. Correlation is necessary for Causation. Temporal order is needed for Causation Beware Post Hoc fallacy.

4 CORRELATION ISSUES IN CAUSAL MONITORING 2 “For Example” is not proof. “For Example” may disprove. Belief is not fact. Belief is not theory. Belief is not causation.

5 REQUIREMENTS FOR CAUSAL MONITORING Supporting Evidence (correlation) Temporal Ordering Theoretical Justification Rejection of Rival Hypotheses

6 THEORETICAL ASSUMPTIONS: 1 Socialization Symbolic Interaction Cognitive Psychology Behavioral Psychology Medical

7 THEORETICAL ASSUMPTIONS: 2 Biosocial Economic Ecological Religious Ideological

8 THEORIES: SOCIALIZATION Motivator—learned behavior Intervention—changing sources of socialization Monitoring—socialization process and behavioral outcomes

9 THEORIES: SYMBOLIC INTERACTION Motivator—social definitions Intervention—changing labels and definitions Monitoring—use of labels and definitions and outcomes

10 THEORIES: COGNITIVE PSYCHOLOGY Motivator—self definitions Intervention—changing internal dialogue and fantasy Monitoring—alternative dialogues, fantasies, and outcomes

11 THEORIES: BEHAVIORAL PSYCHOLOGY Motivator—conditioned behavior Intervention—changing reinforcers Monitoring—change in reinforcers and outcomes

12 THEORIES: MEDICAL Motivator—biological and mental health factors Intervention—changing biological and mental health factors Monitoring—change in biology, mental health, and outcomes

13 THEORIES: BIOSOCIAL Motivator—bio-social interaction Intervention—changing bio- social interaction Monitoring—change in bio- social interaction and outcomes

14 THEORIES: ECONOMIC Motivator—material economic rewards and costs Intervention—changing economic rewards and costs Monitoring—economic changes and outcomes

15 THEORIES: RELIGION Motivator—religious beliefs of the moral and immoral Intervention—changing beliefs, promoting tolerance Monitoring—religious statements, behaviors, and outcomes

16 THEORIES: IDEOLOGY Motivator—ideological beliefs regarding duty to self and others Intervention—changing beliefs, promoting tolerance Monitoring—ideological statements and behaviors and target outcomes

17 METHODS FOR CAUSAL MONITORING Imposed experiments Natural experiments Partial experiments—non-randomization of groups or non-randomization of intervention Quasi-experiments Non-experiments—pre-post one group design

18 METHODOLOGICAL ISSUES OF NON- EXERIMENTAL MONITORING: 1 Sample selectivity—pre-group differences Contamination of groups History versus contextual analysis Maturation versus counter trends

19 METHODOLOGICAL ISSUES OF NON- EXPERIMENTAL MONITORING: 2 Instrumentation versus pre-testing Testing Effects versus debriefing Regression to mean versus subgroup analysis Differential sample attrition versus missing data analysis Availability of comparison group versus comparison with theoretical standard