COMMUNITY HEALTH AND WELLNESS CONFERENCE APRIL, 2013 Documenting and Communicating Progress on Policy Change

Agenda How does policy change progress, so I know what to look for? What are some process and outcome measures that are appropriate for policy change? What impact measures are appropriate for policy change? What should I report and when?

Use data at every stage of policy work Chuck: “Some date we use to get started, some data we use to know if we are on the right track, and some data lets us know how we did.” Report out Theory of change Progress: process Progress: outcome Impact

Policy change theories This document identifies six theories of policy change. Each is summarized and a theory of change is provided for each theory with strategies and outcomes. Stachowiak, S. (2009). Pathways for change; Six theories about hoe policy change happens. Seattle, WA: Organizational Research Services. Funding from The California Endowment. Global theory: Large Leaps  Significant changes in policy and institutions can occur when the right conditions are in place. Conditions are: issue is defined differently or new dimensions of the issue get attention; new actors get involved; the issue becomes more salient and receives heightened attention. Promising strategies include issue framing, mobilizing supporters, and media advocacy. Focus on changing policy, not administrative or rule changes. Issue definition and agenda setting are key to mobilizing new people. Global theory: Coalition  Policy changes happens through coordinated activity among a range of individuals with the same core policy beliefs. Coalitions are held together by agreement over core beliefs about policies; core beliefs are unlikely to change. Strategies include influencing decision-makers, changing incumbents, affecting public opinion, changing perceptions about policies. Global theory: Policy Windows  Policy can be changed during a window of opportunity when advocates successfully connect two or more components of the policy process: the way a problem is defined, the policy solution to the problem, or the political climate surrounding the issue. Three streams in the policy process are problems (including definition and salience), policies, and politics. Two streams must converge to create a policy window. Strategies include impacting problem definition, developing policy options, and influencing the political climate (through coalition building, demonstrations, or media).

Based on work with tobacco and underage drinking prevention coalitions Consistent with research (including that done by Organizational Research Services, Michael Quinn Patton, etc.). Strategy theory: Messaging and Frameworks  Individuals’ policy preferences or willingness to accept them will vary depending on how options are framed or presented. Strategies include issue framing and message development, typically embedded in a broader campaign. Strategy theory: Power Politics  Policy change is made by working directly with those with power to make decisions or influence decision making; power is concentrated in the hands of a few. Strategies include relationship building, focusing on decision- makers, and establishing credibility. Strategy theory: Grassroots  Policy change is made through collective action by members of the community who work on changing problems affecting their lives. Power is changeable and dynamic and not held by a few elites. Strategies including training, capacity building, community mobilizing, etc.

Coalition strategies were consistent with CDC Coalitions utilized strategies consistent with Centers for Disease Control and Prevention guidance (2009). The most common strategies were:  educating community and decision makers through presentations and one-on-one meetings;  followed by monitoring, assessment, or local surveillance to collect data about the issue;  and then advocating directly to decision-makers through presentations or one-on-one meetings.

Comparing prevalence of strategies: successful and unsuccessful efforts Source: Coalition interviews

The prevalence of actions taken by Coalition members to move a policy forward. Source: Coalition interviews

Start with a theory of change What steps do you expect to see? What strategies are you using to move your policy work forward?

Documenting progress There are many types of evaluation, but thinking about evaluation in terms of process, outcome, and impact categories can be helpful.  Process evaluation is “a study of what goes on while a program (or policy) is in progress. Process evaluation relates to the phase of the program studied – in this case, program implementation “(Weiss, 1998, p 335).  Outcome evaluation is “a study of whether or not the program produced the intended program effects. Outcome evaluation relates to the phase o f the program studies - in this case, the end result of the program” (Weiss, p. 335).  Impact is “the net effects of a program (i.e., the gain in outcomes for program participants minus the gain for an equivalent group of nonparticipants). Impact may also refer to program effects for the larger community” (Weiss, 1998, p. 331). Weiss, C. (1998) Evaluation: methods for studying programs and policies (2nd ed.). New Jersey: Prentice Hall.

Exercise: process measures for policy change Specific: the objective must communicate exactly what is expected, who’s involved, and where is it going to happen. Measurable: the objective should include concrete criteria or data for measuring progress toward the attainment of the goal. Attainable: the goals are neither out of reach nor below standard performance as these may be considered meaningless. Relevant: the objective should be worthwhile, happening at the right time, and match other efforts and needs. Timely: ground objectives within a time frame, giving them a target date.

How did Coalitions move policy forward? Interim steps are an indicator of future success: 35 percent of the coalitions that achieved a policy reported at least one interim step; in contrast, when not successful, coalitions did not achieve an interim step.

What types of interim steps occurred? A champion stepped forward. Sometimes a school board member, county commissioner, city council person, or administrator became the champion for the policy and offered to guide it to adoption. Policy was developed. Prior to adoption, representatives from the coalition and the adopting entity often prepared drafts or worked on the actual policy components. Supports were developed. The coalition may have assisted with implementation supports prior to the actual policy being adopted / going into effect. This included helping to obtain signs, coordinating meetings between cessation service providers and the entity, providing other resources, or working on ancillary policies such as citations.

Interim steps (continued) Policy was placed on the policy agenda. This occurred when the issue or policy is publicly placed on the policy agenda of your target. This could be when the school board or county commission allows a presentation or when a landlord schedules a presentation at a tenant meeting. Verbal commitment from decision-maker was received. Prior to the actual adoption of a policy, the coalition may receive a verbal commitment to adopt a policy. For example, a landlord may state he/she wishes to move forward and then schedules the next meeting to review draft policies with representatives from the coalition. A non-coalition task force or committee was formed to adopt/implement policy. Often a task force or committee (particularly in schools) will be created to work on the policy and guide its adoption and implementation. Other examples would be a legislative committee working on a proclamation, resolution, or ordinance.

Other interim steps from the literature Policy development, policy proposal, demonstration of support, adoption, funding, and implementation Shift in social norms: changes in awareness, attitudes, values, or behaviors Increased alliances: increased number of partners, increased strength of collaborations Strengthened base of support: increased public involvement, increased advocacy activities, # of partners, increased media coverage, increased awareness among groups, increased visibility of issue

Exercise: Short and intermediate outcome measures Direction of change: identify if the area of change is going to be maintained, increase, decrease, improve, rise, or lower. Area of change: identify what will be measured. Target population: clarify the target population that will experience the change. Degree of change: identify the percentage or actual number. Time frame: provide the deadline by which the change will occur. As measured by: include the tool or source of data. Baseline: document the baseline.

Strategy: social host ordinance As noted, there may be additional interim and long-term outcomes associated with this strategy, such as ordinance adoption, reduction in citations, reduction in social access, and reduction in consumption. OutcomeMeasure, instrument, or question Data collection methodologyAnalysis Process: Youth to present information on the need for policy change and its positive impact to 5 municipalities between 7/1/2012 and 6/30/2013. AgendasSource: Municipality How: Collect or download agenda When: At time of presentation Who: Coalition staff Compare the number of presentations planned to the number of presentations completed. Outcome: By 6/30/2013, at least two interim steps toward a social host ordinance will have been achieved. Policy planner checklist (see Resources section). Source: Coalition How: Document the date that each interim step is achieved When: Quarterly Who: Coalition staff Compare the number of steps planned to the number of steps completed.

Long-term impact Number of policies obtained Number of residents impacted Behaviors of interest Others from prior studies Students report social access to alcohol Policy progress: progress measures Policy progress: interim steps Policy achieved: XX,XXX students Students report less social access to alcohol

Reporting plan content Who is your target audience (e.g. funder, partner, policymaker)? When do they need information (e.g. when planning an evaluation, when conducting an evaluation, or when you have results)? What message do you need to communicate (e.g. progress, results, impact)? What format will you use to communicate your message (e.g. in person, formal report, )? How often will you communicate with this target audience (e.g. monthly, quarterly, annually)?

Policy and advocacy resources California Endowment. (n.d). Advocacy progress planner [computer software]. Available at Sponsored by the California endowment, the component definitions provided as part of the planner are based on two reports prepared by Guthrie, K., Louie, J., David, T., and Foster, C. of Blueprint Research & Design, Inc. in 2005 and 2006 although this document is not attributed to any author. Interim steps towards policy include policy development, placement on the policy agenda, policy adoption, policy implementation, policy monitoring and evaluation, policy maintenance, and policy blocking. Document lists audiences, capacity building and preparation/planning inputs, contextual factors, policy and communication activities, and capacity and policy interim outcomes and benchmarks. Users of the system create their own models from these choices.

Policy and advocacy resources Reisman, J., Gienap, A., and Stachowiak, S. (2007). A guide to measuring advocacy and policy. Seattle, WA: Organizational Research Services. Funded by the Annie E. Casey Foundation. Reisman, J., Gienap, A., and Stachowiak, S. (2004). A practical guide to documenting influence and leverage in Making Connections communities. Seattle, WA: Organizational Research Services. Funded by the Annie E. Casey Foundation.

Wrap up  How does policy change progress, so I know what to look for?  Theory of change  What are some process and outcome measures that are appropriate for policy change?  Process measures  Outcome measure  What impact measures are appropriate for policy change?  Impact measure  What should I report and when?  Outline of a reporting plan