Blending the Theoretical and the Empirical in Evaluation Science: A Case Example from Cancer Control/Prevention and a General Discussion Ross Conner University of California Irvine USA and Senior Adviser and Past President – International Organisation for Cooperation in Evaluation – IOCE

Workshop Outline Evaluation Overview The Theoretical Component The Empirical Component Case Example: Chinese-Korean Cancer Prevention/Control Program Conclusions Questions & Answers; General Discussion

1. Evaluation Overview Inputs  Processes  Outputs  Outcomes  Impacts A simple example: program = a mathematics class Inputs: students, teacher, lesson texts, paper and pencils, classroom setting Processes: lessons and exercises taught by teacher Outputs: exercises completed by students Outcomes (short-term): Students learn new concepts, have new understandings (measures: tests) Impacts (long-term): Students apply new concepts and understandings, students advance to next level of learning For simplicity, this is set out as a linear track of 5 stages; in practice, it is often more complex with multiple tracks because of different components of the program.

2. The Theoretical Component Theory from Evaluation Science Evaluation Science: bringing the tools of science and discovery to the assessment and evaluation of programs or policies. Tracking and monitoring inputs and processes; identifying and documenting outputs; measuring outcomes; tracking and measuring impacts. Theory from the Program or Policy Area Program or Policy Area: bringing the past learning from the area to advance the area Building upon past experimental and theoretical work in the area, developing program components based on past research and practice. Using both types of theories, the program components and the evaluation components are developed.

3. The Empirical Component Program Design Specify planned inputs and planned processes Specify expected outputs, outcomes and impacts Specify evaluation questions Evaluation Design Specify evaluation design(s) to answer the evaluation questions. Design = the plan for data collection Specify evaluation measures to obtain necessary data

4. Case Example Area: Health – cancer control and prevention Populations: Chinese and Koreans in California, USA Program focus: women’s cancers initially, later men’s cancers Program aim: disease destigmatization

Overview: Chinese Program Components Materials in Chinese language Large free luncheons with expert lecturers Free screenings for cancers Cancer survivors involved in the programs High visibility in community

Overview: Korean Program Components Materials in Korean language Used networks of Korean Christian ministers to destigmatize; cancer survivors featured Free screenings for cancers Prominent role at public events and fairs

The Program Inputs  Processes  Outputs  Outcomes  Impacts Planned Inputs: materials in native language, large gatherings of community members, religious leaders (Korean), cancer experts (Chinese) Planned processes: dissemination of materials and information to 4,000 people, cancer screenings, discussions among community members Planned outputs: increased knowledge about cancer, increased understanding of consequences of disease stigmatization Expected outcomes: increased (by 10%) cancer screenings, decreased disease stigmatization Expected Impacts: 100 community cancer survivors serving as volunteers and outreach workers for the program

The Evaluation Inputs  Processes  Outputs  Outcomes  Impacts Examples of evaluation components: Inputs: tracked numbers of attendees at events Processes: observed community events – materials distributed, lectures given, discussions held Outputs: assessed knowledge about cancer Outcomes: tracked cancer screenings, tracked decreased disease stigmatization Impacts: tracked cancer survivors serving as volunteers for the program

Selected Evaluation Results Outputs and OutcomesChineseKoreanTotals Health Education Sessions Conducted (attendance)* 4,9128,02612,938 Community Outreach Workers Trained 12427** Cancer Screenings Conducted+ 2,2121,7393,951 * Estimated duplicate counts: 40% ** Ministers + Includes breast, uterus, liver, prostate and colon cancer screenings

5. Conclusions The program and the evaluation blended theoretical components from evaluation science and from the area of health education/disease prevention. The program set measurable objectives. These anchors provided a basis to judge achievements. The evaluation involved empirical assessments of inputs, processes, outputs, outcomes and impacts. The program planners/implementers from the Chinese and Korean communities were partners with the evaluation team, from start to finish. The programs continue today, with a smaller funding base

6. Audience Inputs and Comments Questions about the program Questions about the evaluation Questions about the results General questions about evaluation Any other general questions for discussion

Evaluation: Next Steps New book available soon: Оценка программ: методология и практика Редакторы: А.Кузьмин, Р.О’Салливан, Н.Кошелева