Don Compton, Ph. D. Senior Evaluator DNPAO dcompton@cdc Don Compton, Ph.D. Senior Evaluator DNPAO dcompton@cdc.gov (770) 488-5258

Each Has an Advisory Group Made up of State and CDC Staff Evaluating Partnerships Patricia Rieker and Jan Jernigan Developing and Using an Evaluation Consultation Group Michael Baizerman and Don Compton Evaluation of State Nutrition, Physical Activity, and Obesity Plans Paul Mattessich and Don Compton

Jianglan White, GA Emma Kirkpatrick, TX Amanda Raftery, IN Kristian Gordon, SC Abdoulaye Diedhiou, SC Laura Hutton, MN Jan Liebhart, WI Shelly Sutherland, MT Don Compton, CDC Raegan Tuff, CDC Jennifer Kohr, CDC

Executive Director, Wilder Research Center, has worked in the fields of human services, social policy, and social research since 1973. In 1982, he became executive director of Wilder Research and has served as a member of the Wilder Foundation's senior leadership team since then. He has authored or co-authored more than 250 publications. He has also served on a variety of government and nonprofit boards of directors and special task forces. He is the author of 4 books: Collaboration: What Makes It Work; Community Building: What Makes It Work; the Manager’s Guide to Program Evaluation; and Information Gold Mine – all with wide nonprofit audiences.

Paul W. Mattessich, Ph.D. March 18, 2010

Overview of process for developing an evaluation of the State Plan Scope of State Plan evaluation State Plan evaluation / evaluation of initiatives State program initiatives / partner initiatives Involvement of stakeholders and partners

Describes a comprehensive, integrated approach for improving nutrition and physical activity, to prevent/reduce obesity in a state Typically created collaboratively by “partners” throughout the state: state and local health departments, education agencies, health organizations, nonprofit organizations, advocates, insurers, and others

Development of a partnership to create, publish, and disseminate the state plan Development of an implementation plan for the state plan Development of an evaluation plan for the state plan (and for the implementation plan) Ongoing evaluation and the reporting of evaluation results

1. Increase physical activity. 2. Increase consumption of fruits and vegetables. 3. Decrease the consumption of sugar-sweetened beverages. 4. Increase breastfeeding initiation and duration. 5. Reduce the consumption of high-energy-dense foods. 6. Decrease television viewing.

Information to enable states to develop and implement plans as effectively and efficiently as possible States developing first plan: process evaluation to strengthen their work, improve stakeholder involvement, etc. States with plan: “outcome evaluation” States revising plan: evaluation results (from their state and others) as input into development of new plan

Activities: Production/Dissemination Intermediate Outcomes Outputs   State Plan (completed and distributed to all intended audiences) Implementation Plan (completed, distributed to all intended audiences, used) Partnerships formed, continued Activities: Production/Dissemination   Stakeholder Engagement, Partnership Formation Writing of Plan and Implementation Plan Approvals Secured Raise Awareness - Plan Distribution, Presentations, Forums, Education of Intended Audiences Inputs   Data on Nutrition, Physical Health, and Obesity Conditions Health Department Infrastructure Human and Financial Inputs (for writing) Initial Outcomes   Increases in: Policies and standards… Access to and use of environments… Social and behavioral approaches… …to support healthful eating and physical activity Implementation of initiatives Implementation of evaluation Intermediate Outcomes   1. Increased physical activity. 2. Increased consumption of fruits and vegetables. 3. Decreased consumption of sugar-sweetened beverages. 4. Increased breastfeeding initiation and duration. 5. Reduced consumption of high-energy-dense foods. 6. Decreased television viewing. Ultimate Outcomes   Reduced obesity Reduced chronic diseases Decreased mortality rates

Primary focus, for annual review: Inputs Activities Outputs Initial Outcomes Secondary focus, for longer term review (and use in revising the state plan): Intermediate and Ultimate Outcomes

Information regarding inputs, activities and outputs provides insight on plan development, and whether plan design/content can be improved. Information regarding initial outcomes indicates whether the plan achieves its initial, short-term purpose to make certain that state and local health departments and their partners organize and improve policies, access, and approaches for the prevention/reduction of obesity among the state’s residents.

Information on intermediate and ultimate outcomes Is important for future revisions in the state plan Is available, e.g., through BRFSS However, it is not necessary to evaluate every specific initiative or policy in the state plan in order to do a “state plan evaluation”. (Those initiatives can be the subjects of separate evaluations.)

Plan identifies initiatives for implementation by state and local health departments, education agencies, health organizations, nonprofit organizations, advocates, insurers, and others Therefore: Evaluation of Plan determines whether these were implemented Because this implementation falls into the “Initial Outcomes” of the State Plan

Questions about Process Questions about Content Questions about Dissemination Questions about Awareness Questions about Initial Outcomes Questions about Intermediate Outcomes Questions about Ultimate Outcomes

Examples and Potential Methods/Sources Question Method/Source Were the appropriate stakeholders involved in the development of the state plan and the implementation plan? Stakeholder Survey Were the stakeholders satisfied with their level of involvement? Do the involved stakeholders feel “ownership” of the state plan?

1. I had sufficient opportunities to give input on the plan. SA A D SD NS 2. The amount of time requested of me as a Task Force member was reasonable (i.e., not excessive). 3. I believe that my ideas and suggestions were taken into account in developing the plan. 4. The plan reflects my personal values/priorities related to obesity prevention. 5. The plan supports the mission or priorities of the work I do. 6. It is clear to me how the work I do fits into the plan. 7. I believe that the plan will be useful in guiding or supporting my work. 8. Overall, I enjoyed serving on the Task Force. 9. Overall, I am satisfied with the Nutrition and Physical Activity Plan.

Examples and Potential Methods/Sources Question Method/Source Was a state plan completed? Observation Was an implementation plan completed?

Examples and Potential Methods/Sources Question Method/Source Does the state plan include sufficient data describing nutrition, physical activity, and obesity within the state’s population? Key informant ratings? “State Plan Index”? Are objectives clearly organized and logical? Does the plan reflect the current state of the art and science in public health? Is it evidence/science based? Does the plan incorporate national goals for Nutrition, Physical Activity, and Obesity? Are goals and objectives stated in SMART (or similar) format?

Examples and Potential Methods/Sources Question Method/Source Has a strategy for dissemination of the plan been developed and accepted by stakeholders? Observation, Checklist Have all intended audiences been reached and involved, in accordance with the implementation plan? Checklist? Survey?

Examples and Potential Methods/Sources Question Method/Source Do those who should know about the plan actually know about it? Survey Do those who should know about the plan (or parts of it) actually understand what it says? Do partners link to plan on their web site? Observation/Checklist

State Plan Index Nutrition and Physical Activity Program To Prevent Obesity and Other Chronic Diseases The Centers for Disease Control (CDC) has provided a State Plan Index to assist with an overall assessment of the plans developed by states. The index was set up to score each item on a scale of 1 to 5, once the plan is written. However, reviewing the criteria as we are writing the plan should assist with making sure that we have considered the following points:

Presentation of Data on Disease Burden and Existing Efforts to Prevent and Control Obesity Data are presented on disease burden of obesity and chronic diseases related to poor nutrition and physical inactivity. 1 2 3 4 5 Epidemiologic data are from reliable source(s) (e.g., BRFSS, NHANES). 1 2 3 4 5 State-level data are provided, including results of state-specific epidemiologic or evaluation studies. 1 2 3 4 5

Examples and Potential Methods/Sources Question Method/Source Has an increase occurred in policies and standards to support healthful eating and physical activity? Inventory Has an increase occurred in access to, and use of, environments which support healthful eating and physical activity? Has an increase occurred in social and behavioral approaches used to support healthful eating and physical activity? Survey Have agencies / organizations identified by the plan for specific roles or actions carried out their assignments? Review/checklist, based on implementation plan Survey? Do established public health programs conform to plan? Inventory – programs compared with plan

Evaluation of the “Initial Outcomes” of the State Plan may be the same as evaluation of the Implementation Plan

Indicator 16: Number of Women, Infants, & Children Program (WIC) clinics funded by DSHS WIC for obesity prevention activities Indicator 17: Percent of partners who implement specific items from the state plan that are related to their work Indicator 19: Number of policy and/or environmental changes Indicator 20: Number of legislative bills passed related to obesity prevention Number of worksites that have adopted worksite wellness programs that meet a qualified standard Indicator 23 (Target 19): Percent of school districts that have implemented an approved school wellness policy Indicator 24: Percent of schools with a school health committee or advisory group

Examples and Potential Methods/Sources Question Method/Source Did physical activity increase? BRFSS Did consumption of fruits and vegetables increase? BFRSS Did obesity rates decline?

Stakeholder Involvement Strategy How Evaluation Can Assist

Their primary interest Stakeholder Their primary interest How involve? Contact frequency Contact details

Enables you to communicate with stakeholders during the development of the State Plan Use feedback to make changes in the plan itself, or in the development process Relates to Inputs, Activities, and Outputs boxes in logic model for State Plan Evaluation

Design Data Collection Analysis Reporting

Select questions Select methods – and establish schedule for data collection Assign personnel, allocate resources Finalize all procedures – review with stakeholders, test methods, etc.

Examples and Potential Methods/Sources Question Method/Source Were the appropriate stakeholders involved in the development of the state plan and the implementation plan? Stakeholder Survey Were the stakeholders satisfied with their level of involvement? Do the involved stakeholders feel “ownership” of the state plan?

1. I had sufficient opportunities to give input on the plan. SA A D SD NS 2. The amount of time requested of me as a Task Force member was reasonable (i.e., not excessive). 3. I believe that my ideas and suggestions were taken into account in developing the plan. 4. The plan reflects my personal values/priorities related to obesity prevention. 5. The plan supports the mission or priorities of the work I do. 6. It is clear to me how the work I do fits into the plan. 7. I believe that the plan will be useful in guiding or supporting my work. 8. Overall, I enjoyed serving on the Task Force. 9. Overall, I am satisfied with the Nutrition and Physical Activity Plan.

Participation Records Surveys of Stakeholders Expert Informant Surveys Annual Inventories (policies, initiatives, etc.) State Plan Index

Obtain information, using methods developed in design stage Compile and store information in usable format

Process information Prepare for reporting Preliminary review of findings

Present findings to key stakeholders Issue report(s)

Information Collected Information Used Questions about Process, Content Early in process of Plan development, and throughout development phase During Plan development phase – to improve the process and content Questions about Dissemination After drafts or final versions are distributed Immediately – to identify gaps Questions about Awareness A few months after distribution; possibly at later intervals Immediately – to identify additional action steps Questions about Initial Outcomes One year after completion of Plan For evaluation of “Implementation” Questions about Intermediate and Ultimate Outcomes Baseline, and a few years after plan development Primarily at time of revision of State Plan