Evaluating Physical Activity Intervention Programs Thomas Schmid, PhD Physical Activity and Health Branch CDC Atlanta, Georgia, USA

 Evaluation Definitions and Principles  CDC Physical Activity Evaluation Handbook – 6 Steps  Evaluation issues

The systematic examination and assessment of features of a program and its effects. Evaluation is designed to produce information that can be used by those who have an interest in the improvement or effectiveness of an initiative or program.

 To assess the effects and value of a program ◦ for accountability ◦ for progress ◦ to identify best practice ◦ as a planning tool ◦ for research

 the program  target groups and needs  realistic achievable objectives  is the program ready for evaluation?

 Identify needs  State goals and measurable objectives  Assess progress towards objectives

Efficacy Intervention tested under ideal conditions with optimal study design Effectivenes Field based intervention testing whether the intervention works as intended for a defined population in the real world Efficiency Results in comparison to the efforts required to obtain them – often cost or cost effectiveness

IMPACT OUTCOME Measures activities, program quality, reach Measures long term effects (goals) PROCESS Formative Measures short term objectives Develop program & resources

 What is likely to work  Best practices in health promotion  Idea or concept development  Developing and pilot testing materials  Focus groups

 Focus groups and interviews with partners about their needs  Reviewing best practices from other programs such as Agita Sao Paulo  Pilot testing in one company

Process evaluation ◦ Program implemented as intended ? ◦ Quality of program components ◦ Attendance rates ◦ Audience awareness of message ◦ Appropriate message learned ◦ Participant satisfaction

 Companies enrolled  Workshops held  Promoters trained  Companies participating in community events

Impact & Outcome Evaluation ◦ Short and long term effects of program ◦ Hierarchy of effects : what level of effects indicates program success ◦ Assessing the impact upon those who attend or participate ◦ Community effects

 Physical activity level of employees  PA level in community  Disease rates or costs

Program Evaluation Framework Physical Activity Evaluation Handbook

Program Evaluation Framework Physical Activity Evaluation Handbook 1.Engage Stakeholders Include people with a vested interest in the program: implementers, partners, participants, possible opponents and decision makers. 2.Clearly Describe the Program The problem being addressed, descriptions of the planning, implementation and maintenance phases of the program, and finally, defining and communicating how all phases work together (eg. the "logic model").

Program Evaluation Framework Physical Activity Evaluation Handbook 3.Clearly Focus Asking a clear, straightforward question and ensuring linkage to the problem being addressed. Includes clarifying process and/or outcome evaluations of interest 4. Develop and Implement Data Collection Design of data collection efforts, selection of appropriate control groups, designation of appropriate baseline values and decisions about how much time should be built in to adequately measure "change”

Program Evaluation Framework Physical Activity Evaluation Handbook 5.Develop Conclusions Data analyses and interpretation. 6. Communicate Promotes the importance of tangible products from the evaluation, dissemination of those products, and follow-up (as appropriate).

20

 All physical activity interventions should be evaluated. Evaluation begins with program planning  There are many good programs, but very few are well evaluated  Involve stakeholders  Community – University partnerships can be very useful

 Process as well as outcome evaluation  Quantitative and qualitative  Tailor evaluation to stakeholders and situation  Evaluation is often undervalued compared with “hard science”  Adequate resources should be devoted to evaluation. 10% is recommended

 Evaluation results should be used to adjust programs and inform policy  Share results with stakeholders  Academic and practical training are needed  The CDC Physical Activity Evaluation Handbook provides a good framework

 Gracias/Thank you