1 Chesapeake Bay Funders Network Program Evaluation Training Workshop OMG Center for Collaborative Learning January 9-10, 2008 Evaluating Advocacy

2 Building an Evaluation Plan I. Theory of Change II. Outcomes III. Indicators IV. Methods V. Putting Evaluation to Work for You Same Building Blocks for Evaluating Advocacy

3 What is advocacy? For our purposes, advocacy represents… Strategies devised Actions taken Solutions proposed …to inform or influence local, state, or federal decision making.

4 Evaluating Advocacy: How is it Different? Advocacy has always been considered hard to measure- less tangible Is evaluating advocacy different than evaluating traditional programs? Yes and No....

5 Evaluating advocacy is different from evaluating traditional program work because of the nature of advocacy itself. 1. Advocacy strategies evolve over time; activities and outcomes can shift quickly 2. The policy process has many unpredicted and uncontrollable variables (windows of opportunity) 3. So many groups focus on an issue – hard to isolate attribution (Hence, contribution – is it plausible that your activities contributed to the end result?) Evaluating Advocacy: How is it Different?

6 Evaluating Advocacy: How is it the same? The same principles and approaches that we talked about earlier can be applied to evaluation of advocacy 1. You can develop a Theory of Change for advocacy 2. You need to determine outcomes and the indicators that will tell you whether the outcome has happened 3. You can use qualitative or quantitative approaches to collecting the data

7 Evaluating Advocacy: How is it the same? 4. Purpose is the same: Accountability – reporting on your results Promotion – leveraging further interest in your work (other funders, volunteers, partners, etc..) Inform Constituents/Stakeholders – results of their actions, learning Decision-making – inform your work as you go, and reflection – what worked, what didn’t, what to do differently

8 What is the primary advocacy or policy change goal? Public Awareness: Aims to increase public knowledge or salience of a particular problem or policy solution Public Will: Aims to increase public willingness to act on behalf of a particular policy issue Political Will: Aims to increase policymaker willingness to act on behalf of a particular policy issue *Slides 8-17 draw from “Using the Advocacy and Policy Change Composite Logic Model to Articulate an Advocacy Strategy or Theory of Change”, Julia Coffman, Harvard Family Research Project Creating a Theory of Change for Advocacy

9 What contextual factors might affect your strategy’s success? Political Climate Economic Climate Social Climate Prior Experience with Advocacy Issue Competition Potential Partners/Competitors/Opponents Creating a Theory of Change for Advocacy

10 Determining outcomes for advocacy- – TIMING MATTERS! A Theory of Change for advocacy must have a realistic timeline – This type of work typically takes a long time to yield outcomes – Short term, intermediate, long term Creating a Theory of Change for Advocacy

11 Timing Matters: Short-term/Intermediate Outcomes Creating Partnerships or Alliances Collaboration and Alignment (include messaging) New Advocates New Champions (including policymakers) Media Coverage Creating a Theory of Change for Advocacy

12 Timing Matters: Short-term/Intermediate Outcomes Issue Reframing Awareness Salience Attitudes or Beliefs Public Will Constituency or Support Base Growth Creating a Theory of Change for Advocacy

13 Timing Matters: Longer Term Outcomes Policy Development Placement on the Policy Agenda Policy Adoption (passes through ordinance, ballot measure, legislation or legal agreement) Creating a Theory of Change for Advocacy

14 Timing Matters: Longer Term Outcomes Policy Implementation (funding, resources, q.a.) Policy Monitoring and Evaluation Policy Maintenance Policy Blocking (opposition to a policy proposal that would go against your goals) Creating a Theory of Change for Advocacy

15 Who is the audience? Most strategies will target multiple audiences For example: Elected officials, Voters, Media Other potential audiences: Public administrators, Business, Community leaders, Courts, Specific constituencies Think about who needs to be doing the advocating and who needs to be hearing it to achieve the goal (levels of influence) Creating a Theory of Change for Advocacy

16 What will it take to convince or move the audience? Activities needed to achieve the outcomes (Sequence) Creating a Theory of Change for Advocacy Policymaker/Candidate Education New Champions Placement on the Policy Agenda Demonstration Projects or Pilots Awareness Policy Adoption Constituency/Support Base Growth

17 I will know there is…When I see… Advocacy: Outcomes and Indicators New ChampionsHigh profile individuals speaking out on the issue Placement on the Policy Agenda Issue/idea appears on schedule for hearings AwarenessTarget audience recognizes the issue/is familiar with the terms Constituency/Support GrowthIncreased number of people participate in advocacy or action on the issue Policy AdoptionSuccessful passing of the policy proposal through an ordinance, ballot measure, legislation or legal agreement

18 1.Focus Groups/Interviews/Surveys Used to measure Awareness, Salience, Attitudes/Beliefs, Public Will, Political Will, Constituency or Support Base Growth Specific techniques: 1.“Bell weather Assessment”: Is the issue on people’s radar screens? Semi-structured interviews with people who know what is going on in the relevant political realm 2.Rating Scales: Is there political salience/will for an issue? Interview top advocates working on an issue to a) Rate the legislators on receptivity/likelihood of voting in favor/against; b) Rate influence within the legislature; and c) Rate confidence in their own rating Advocacy: Methods

19 2.Media Analysis – level of coverage, type of language used, increase in media hits following various actions (Lexis- Nexus on-line search engine for media mentions; services) 3.Growth in Partnerships/Alliances over time – see OMG Framework to assess partnership strength 4.Record Review to Assess Constituency/Support Base Growth; #s attending certain events, #s signing petitions, #s participating in list-serves, e-networks. Advocacy: Methods

EXAMPLE: Fullojunk Watershed Alliance Advocacy Strategic Focus Improve the environmental quality of the Fullojunk Watershed through local advocacy. Long-term Outcomes Passage of local ordinances to support comprehensive land use planning and reduction in storm water runoff Short-term Outcomes New champions and advocates (including Mayor) Increased public awareness Organizational visibility and recognition Alliance Advocacy Activities Conduct issue/policy analysis research Build relationships with decision makers and new administration staff Lead grassroots organizing and mobilization Assumptions Contextual Analysis There is a new mayor who is pro-environment The Alliance has very little advocacy experience Because there is a new mayor, a lot of advocacy organizations are clamoring to have their message heard, leading to issue competition

21 Exercise (Whole Group) Who has an advocacy goal that they are trying to reach? - What is the goal? - What are your activities? - Short term outcomes? Long term? - Assumptions along the way? Group Input