2.  What are evaluation criteria?  What are step3 and step 4?  What are the step3 and step4 output report? S519

3.  Dealing with the causation issue, basically be able to answers following questions:  How certain does the client need us to say that the evaluand „caused“ a certain change?  What are the basic principles for inferring causation?  What types of evidence do we have available to help us identify or rule out possible causal links?  How should we decide what blend of evidence will generate the level of certainty needed most cost- effectively? S519

4.  Each decision-making context requires a different level of certainty  Quantitative or qualitative analysis  All-quantitative or all-qualitative  Sample choosing  Sample size  Mix of them S519

5.  Two basic principles:  Look for evidence for and against the suspected main cause (i.e., evaluand)  Look for evidence for and against any important alternative causes (i.e., rival explanations)  Too many evidences or causes, which are the primary causes  All based on the level of certainty you need for your evaluation  Stepwise process  Put yourself in the hardest critics, gather enough evidence to support your explanation  Repeat it until all remaining alternative explanations are ruled out.  Critical multiplism: triangle The harder people attack, the more solid your answers need to be. S519

6.  1: ask observers  Two possibilities  Ask actual or potential impactees  Ask indirect impactees (i.e., co-worker, parents...)  Design your interview questions to include causation questions  E.g., how much has your knowledge increased as a result of attending this program? – get primary cause  E.g., did anything else besides the program increase your knowledge in this area over the same period of time? – get other causes  E.g., please describe anything else that has happened to you or someone you know as a result of participating in this program? – get the causes which people know or believe were caused by the program. S519

7.  1: ask observers  Causation-rich questions tend to be leading (direct the respondent to answer in a particular way). Be careful about the wording when designing interview questions  The causation question is not just whether the program produced the effect but also what other factors enabled or inhibited the effect.  Individual might not be a reliable witness to answer the causation question, other evidence will be required to make justifiable causal inferences. S519

8.  1: ask observers  Methods  Questionnaires to identify the targeted groups (people who experienced substantial changes)  Using open-end to get more opinions  In-depth interview with the targeted groups  To find out causation. S519

9.  2. Check whether the content of the evaluand matches the outcome  Alcoholics treatment program  alcoholics avoid relapses  Check whether the strategies which alcoholics use to avoid relapses after the program, are the same as the strategies taught in the program S519

10.  3. Look for other telltale patterns that suggest one cause or another  Modus operandi method – look for evidence -- detective metaphor to describe the way in which potential causal explanations are identified and tested. S519

11.  4. Check whether the timing of outcomes makes sense  Common sense:  an outcome should appear only at the same time as or after whatever caused it – a considerable delay.  A further downstream the outcomes, the longer they should take to appear  Using timing to confirm or disconfirm causal links:  Is the outcome happened before the evaluation? Or Other downstream outcomes too early?  Is the timing of the outcomes logical to possible causes?  Do the further downstream outcomes in the logic model occur out of sequence? More on Lipsey, M. W. (1989). Design sensitivity: Statistical power for experimental research. Newbury Park, CA: Sage. S519

12.  4. Check whether the timing of outcomes makes sense  Example – a community health education on diet and exercise  Fairly immediate knowledge and skill gain: during or immediately after the intervention  A short delay (days or weeks) before the knowledge and skills are transformed into changing behavior  A moderate delay (weeks or months) before we see changes in individual health indicators (weight, cholesterol, blood pressure, etc.)  A long delay (months or years) to see changes on improvement on diabetes and heart diseases S519

13.  Take grantsmanship workshop as one example  List the timeline potential outcomes (fairly immediate, a short delay, a moderate delay, a long delay)  Using timing strategies to confirm or disconfirm the cause links, state one page for how and why:  One month after the workshop, 3 proposals got grants  One year after the workshop, 3 proposals got grants  Three months after the workshop, some people write good proposals, but some are not.  Think about your own solution  Form a group and discuss S519

14.  5. Check whether the „dose“ is related logically to the „response“.  The dose-response idea  The more dose of drug, the better response later on  If more A (dose), then better B (response)  Compare the less dose with more dose (not overdose) to confirm or disconfirm the cause links  E.g., for performance evaluation project, if we found that performance had been improved dramatically in the unit where the system has been poorly implemented,  this system is not the cause of the performance improvement. S519

15.  6. Make comparisons with a „control“ or „comparison“ group  Divide the participants into different groups  control group (receive the evaluand) vs. Comparison group (receive no evaluand)  Sampling should be done carefully to make sure no systematic differences between groups  Sample size  randomization S519

16.  7. Control statistically for extraneous variables  When using control and comparison groups, try to exclude external variables and make two groups no systematic differences:  Statistical methods  Regression analysis  Try to identify other potential systematic differences  E.g., math improvement for students, how to sample students and think about other potential existing difference. Is the random sampling enough? S519

17.  8. Identify and check the underlying causal mechanism(s)  Try to look for an underlying mechanism to make the case for causation more or less convincing  Cigarette smoking  lung cancer  Correlation studies  Carcinogenic in cigarette causes cancer  Normally coming from literature. S519

18.  Do we need all the evidences we collect from 8 strategies?  How to select them?  Put yourself in the shoes of a tough critic, identify the most potential threatening rival explanation, then chose the types of evidence that will most quickly and cost-effectively confirm or dispel that rival explanation.  Go to next less tough rival explanation,...  Continue, until you have amassed a body of evidence to provide you enough certainty to draw causal inferences S519

19.  Grantsmanship workshop  Grantsmanship workshop strengthen local communities  For (evidences)  Against (evidences)  Other alternative causes (i.e., rival explanations)  Using strategies to confirm or disconfirm these evidences or causes  Putting them together  Form a group to discuss S519