Skunk Works Evaluation Tools: How do we know if we are having an impact?

What are we trying to achieve? Strategies Mind Shaping Process: Activities to demonstrate application Capacity Building Accessible ES Tools Improved State of HWB and Biodiversity (Ecosystem Services) Outcomes Use of ES information in decisions

Operationalizing Outcomes Material living standards Physical health Mental health Social connections Spiritual fulfillment Cultural identity Security Life satisfaction Equity Use of ES and Policy Decisions Improved ES Improved HWB Increased policy documents with reference to ES as desired outcomes or factors in decision- making Increased discussion of ES in policy meetings Clean drinking water Reliable water flow in the dry season Reduced, in- stream sediment concentration

What other factors influence our intended outcomes?  Individual experience and personalities  Social and institutional context  Political climate  Other initiatives in place  Resources Land-use change (outside of decision) Climate change Extreme events (e.g., fire or flood) Other sources not related to natural resources Background factors (physical, mental, social health) Biophysical context Use of ES and Policy Decisions Improved ES Improved HWB

So what to we need to consider? 1) Stage of Evaluation – At what stage are we in completing our strategies? 2) Design – How confident can we be that our intervention caused the outcomes? 3) Data Collection Methods – Which methods are most valid to measure expected outcomes?

Evaluation Stages Developmental What is promoting innovation in the project? Formative How well are our strategies getting us to our goals? Summative (Impact) Did it work?

Design Control and Experiment group/site receive same measurement before and after intervention Quasi- experiment/BACI Control and experiment group/site randomized and measured post-intervention Posttest Only Pre and post measure the experimental group/site One Group Pre- Posttest Control and Experimental group/site non- randomly assigned, measured post intervention Natural Experiment Post intervention measurement only Case Study ✔

Some Example Methods…

To Test Mind Shaping: e.g. Stakeholder focus on ES Mediation of different ideas Understanding and Awareness of ES Method: Mental Models

Method 1: Mental Models Tells us: To what extent ES are being incorporated into Structure and Content of Knowledge and preferences Accomplished during an interview (1hr) All or sample of participants, individual or group Content analysis and stakeholder comparison Improved with pre-post design, control group, and/or randomization Necessary: control for the hypothesized intervention

To Test ES use in Decisions: e.g. Policy documents that refer to ES as desired outcomes or factors in decision-making Method: Content Analysis

Method 2: Content Analysis Tells us: The frequency and parameters of ES used to make a decision, and the relative influence of ES to other factors Desktop document review Can combine with clarifying interviews with document preparers Either inductive or deductive analysis (a rubric helps)

To Test Increased HWB: e.g. Physical Health Life Satisfaction and Happiness Political Voice and Governance Method: Community-level Survey

Method 3: Community-level Survey Tells us: Baseline and change values for actual and perceived HWB attributes Usually best developed based on ideas from interviews or focus groups In-person, paper or Web-based Distributed among the target population Some indicators may already exist from local health departments or other agencies

A note on HWB measurements  Qualitative and Quantitative approaches necessary to hone in on appropriate indicators

Other Methods Network analysis: measure increase in information exchange, collaboration, and social capital related to ES and/or HWB Focus groups/interviews: Just about anything Participant Observation: Participant interaction with the tools and which are most effective at what Observation Protocols: Adaptation of ES specific terminology in meetings or trainings

Water quality high low time Conservation impact site Control Difference in mean values improved water quality! To Test Improved ES e.g. improved water quality and dry-season flow Methods: BACI and modeled controls

Method 4: BACI and modeled controls Tells us: Water quality relative to before the conservation impact and to control areas Experimental backbone: Before-After-Control- Impact (BACI) (conservation impact) Modeled controls and exploration of “minimum monitoring” Post-treatment/impact only Experimental controls to validate NatCap models: model “no action” as a comparison