1 Thai Health Promotion Foundation Outcome Mapping Seminar Bangkok, Thailand March 25, 2007 Terry Smutylo

2 seminar objectives  Introduce Outcome Mapping  Help you decide if & when Outcome Mapping would be useful  Exchange experiences & ideas on applying the method

3 overview of presentation: 1. Evaluation in a complex world 2. Origins of Outcome Mapping 3. Core concepts 4. The 7 planning steps 5. OM in Monitoring & Evaluation

4 1.How do you experience planning, monitoring & evaluation? 2.How could M&E be made more useful and practical in your work environment? CHECKING IN:

5 Accountability & Learning: A Balancing Act

6

7 a shift from: “Did you do what you promised to do?” to… “Are we on the right track?”

8 Evaluation to support innovation: Traditional Evaluations:  Judge success or failure  Measure against fixed goals  External for objectivity  Linear cause/effect models  Accountability to external  Accountability for control, blame  Evaluator controls evaluation  Engender fear of failure Developmental Evaluations:  Provide feedback for improvement  New measures as goals evolve  Internal, integrated, interpretive  Seek to capture system dynamics  Accountability to values, commitments  Understand & respond strategically  Evaluator matches process to context  Feed hunger for learning Adapted from: Patton, Michael Q., 2006, “Evaluation for the Way We Work”, The Nonprofit Quarterly, Spring.

9 Connecting Research To Well-Being BeneficiariesNARO RESEARCH MANAGERS POLITICS New Knowledge/ Ideas POLICIES Motivations FUNDERS Policies Motivations Money + Inputs RESEARCHERS USERS (Farmers & Families) PRIVATE SECTOR National & International RURAL SOCIOLOGISTS Identification of opportunities & constraints MINISTRY Agricultural Facilitation Regulations NGOs Facilitate Adoption of Technology NGOs Environmental Women Groups Advocacy DISSEMINATION New Knowledge Extension Inputs GOV/ MINISTRIES Incentives to Facilitate Adoption of Technology EXTENSION GROUPS Government NGOs, Advisory services FARMER ORGANIZATIONS Identify problems Dissemination LOCAL ORGs Leadership Mobilization PRIVATE SECTOR Seed suppliers Technical Assistance POST PRODUCTION Marketing Transportation/ Shipping FARMERS/ PRODUCERS New Knowledge Sharing Motivation RESEARCH INSTITUTES Outside Community NARO Support Staff Research Support REG~L/ INTERN~L CENTRES Information Technology NGOs Farmer training Transfer of Technology FARMER ASSOCIATIONS Extension Services OTHER RESEARCHERS Universities Technologies Information FARMERS Farmer Orgs Advocacy for research ENVIRONMENT National/ Regional Money/ policy/ services DONORS Financial resources Human resources 8-15 Years

10 the problem with « impact » Impact Implies:  Cause & effect  Positive, intended results  Focus on ultimate effects  Credits a single contributor  Story ends when program obtains success Development Implies:  Open system  Unexpected positive & negative results occur  Upstream effects are important  Multiple actors create results & need credit  Change process never ends

12 ©TOM Jochen Enterprises, Möckernstr Berlin

focus of outcome mapping Behavioural Changes community capacity & ownership increases program influence decreases

14 what is outcome mapping? A methodology for planning and assessing the social effects & internal performance of projects, programs, & organizations

15 a flexible, multiple-use tool Planning Monitoring Evaluation

16 What are we trying to accomplish and how? What do we want to learn? What do we want to know?

17 key messages in

18 look at the bigger picture Seeing yourself as a part of a interconnected web of relationships and systems

19 recognizing that change is… Continuous Complex Non-linear Multidirectional Not controllable

20 embrace constant change “It is not possible to see the same river twice.”

21 keeping your eyes wide open Being attentive along the journey is as important as the destination

22 Relying on Indicators can be dangerous: 1.lulling: create a false sense of security 2.corrupting: they can become our objectives 3.biased: are not use, user, context or value neutral; subject to interpretation 4.static: do not show complex, incremental change 5.misleading: may be chosen for ease of access or need to aggregate rather than relevance

23 Key concepts in Outcome Mapping:  behavior change as results  direct partners  sphere of influence

24 why behaviour changes? development is done by and for people a program can influence the achievement of outcomes - it cannot control them ultimately responsibility rests with the people affected

25 focus on direct partners Key concept is « boundary partners » (the individuals, groups, and organizations you work with directly and try to influence)

26 spheres of influence project / program The rest of the world = boundary partners

27 contribution not attribution your influence towards better world you can influence but not control change in your partners

28 CHECKING IN: 1.How would these ideas resonate in your work environment? 2.Where could a focus on behavior change be useful?

29 uses of outcome mapping

30 PLANNING: articulate goals & define activities MONITORING: track program performance & partners’ progress EVALUATION: design & conduct a use- oriented evaluation primary uses

31 step 1: vision Intentional Design

32 I have a dream! Martin Luther King, Jr. August 28, 1963

33 vision is a guide Compass Light house Distant star

34 Vision describes the improved human, social, & environmental wellbeing to which the program is committed and contributing. You are NOT accountable for achieving your vision!

35 vision in graphic form, Nagaland (India)

36 Dream boldly, Evaluate modestly within the broad development context within the program’s sphere of influence design evaluate

37 step 2: mission Intentional Design

38 The mission is that “bite” of the vision statement on which the program is going to focus.

39 your mission is your “business” What do you do? Who do you do it with? Why do you work with them?

40 MISSIONVISION Beneficiary = Children Thai Health Child Media Program Government/ Policy maker Educational Institute Corporate/ Ad. Agencies Board of TV/Radio Station Academics/Psychologist on children Child media Network TV/Radio Producers Network Book for children Network = BP strategies outcomes

41 step 3: boundary partners Intentional Design

42 boundary partners Those individuals, groups, and organizations with whom the program: interacts directly to effect change anticipates opportunities for influence engages in mutual learning

43 boundary partners have boundary partners programprogram’s bpbp’s bp

44 CIDA IDRC BAIF State NGO State NGO State NGO State NGO State NGO State NGO SHGPolice Community Leaders Families Banks PHCs Swayamsiddha

45 moving from stakeholders...

46...to boundary partners project

47 Example: a network as a boundary Researchers Service providers NGOs Policy makers Project intends to get them working together to solve a common problem Expect: meet, listen to from each other Like: organize an event together Love: implement a joint initiative

48 step 5: progress markers Intentional Design

49 Being clear & specific: Greater awareness… Better access to… Participate actively… More effective management… Empowered women… Reduced conflict… Increased collaboration… Gender sensitivity…

50 progress markers ✓ A graduated set of statements describing a progression of changed behaviours in the boundary partner ✓ Describe changes in actions, activities and relationships leading to the ideal outcome ✓ Articulate the complexity of the change process

51 progress markers (ladder of change) High level of transformation Actively engaged, learning, commitment Early encouraging response to program, initial engagement Love to see Like to see Expect to see

52 progress markers are graduated  move from initial to more profound changes in behaviour  show transformation in a single boundary partner  more informative than a single indicator

53 ✓ Do not require consultant to write quantifiable indicators ✓ Show directionality of change ✓ Support learning function ✓ Can be monitored & observed from the start of the program ✓ Permit on-going assessment of partner’s progress (including unintended results) ✓ Provide basis for dialogue with partners Some advantages of progress markers

54 Taken together, progress markers: facilitate mid-course corrections and improvements articulate the complexity of change stimulate the program to consider how it can contribute to the most profound transformation possible suggest the logic model of change are NOT a checklist of accomplishments!

55 Example: 1. Attending self help group meetings 2. Participating when abuse is discussed Like to see women victims of domestic violence: 3. Educating self about combating domestic violence 4. Setting an agenda for taking collective action 5. Raising funds for victim support services Love to see women victims of domestic violence: 6. Taking action to protect self 7. Lobbying police to change enforcement practices Expect to see women victims of domestic violence:

56 OUTCOME CHALLENGE: The Program intends to see farmers and farmers’ committees which are fully engaged in the research process. They are participating in the design, management and monitoring of field trials; they regularly give researchers full and frank feed back on the technologies being tested; and they share their learning and experiences with extension agents and other farmers. EXPECT TO SEE FARMERS AND FARMERS’ COMMITTEES: 1 Participating in the research in accordance with researchers’ guidance 2 Initiating contact with researchers 3 Continuously monitoring and reporting on their field trials Example: AHI Outcome Journal

57 LIKE TO SEE FARMERS AND FARMERS’ COMMITTEES: 4 Frequently raising problems and questions with researchers 5 Keeping complete records on trials 6 Negotiating trial designs with researchers LOVE TO SEE FARMERS AND FARMERS’ COMMITTEES: 7 Promoting the feedback process among other farmers 8 Carrying out jointly planned trials and constantly feeding back assessments of the results to researchers and extension agents Observations – Sources of evidence Contributing factors & actors Planned use of, or response to, the above monitoring information

58 Checking In: How would the organizations you work with react to the concepts of “Boundary Partners” and “Progress Markers”?

59 step 6: strategy maps stage 1 Intentional Design

60 6 types of strategies causalpersuasivesupportive I aimed at individual boundary partner direct influence arouse new thinking; build skills, capacity on-going support E aimed at boundary partner’s environment alter the physical, regulatory or information environment broad information dissemination; access to new info create / strengthen peer networks

61 causalpersuasivesupportive I aimed at individual boundary partner E aimed at boundary partner’s environment

62 Value of strategy maps Clarify what the project is doing, with whom and why Anticipate the program’s influence on the BP Articulate the range, mix and logic of the strategies Detect strategy gaps and over extension of resources Encourage multiple interventions to increase effectiveness Suggest appropriate evaluation methods

63 step 7: organizational practices Intentional Design

64 organizational practices keep learning foster creativity & innovation seek better ways to assist your partners maintain your niche maintain higher level support build relationships What you do as an organization to:

65 8 organizational practices 1.Prospecting for new ideas, opportunities, & resources 2.Seeking feedback from key informants 3.Obtaining the support of your next highest power 4.Assessing & (re)designing products, services, systems, and procedures

66 8 organizational practices 5.Checking up on those already served to add value 6.Sharing your best wisdom with the world 7.Experimenting to remain innovative 8.Engaging in organizational reflection

67 planning and assessment possibilities in OM program partner outcomes (behaviour changes in the partners) implementation (interventions by the program) relevance & viability (of the program) contextual information situational data

68 ongoing OM applications GhanaKoreaSwitzerland KenyaEcuadorUnited Kingdom MadagascarMexicoBrussels MaliEgyptNetherlands NamibiaBhutanAustralia UgandaHondurasSri Lanka ZimbabweGuatemalaIndia

69 a few examples… Nagaland (India) Latin American Trade Network International Model Forests (Mexico, Russia, Chile) SchoolNet Namibia ADRA (Cambodia) ST2EEP (Zimbabwe)

70 The International Model Forest Network (Mexico, Russia, Chile)

71 Progress Markers for local communities 1.Participating in regular model forest meetings 2. Establishing a structure for cooperation 3. Acquiring new skills for managing model forests 4.Contributing resources to get the MF operational 5.Articulating a locally relevant vision for the MF 6.Promoting their MF nationally 7.Expanding the partnership 8.Calling upon external experts for advice 9.Requesting new opportunities for training 10.Publishing examples of benefits achieved through MF 11.Seeking out new partners for the MF 12.Obtaining funding from different national sources 13. Helping other communities establish MFs 14. Sharing lessons learned internationally 15. Influencing national policy debates on resource use

72 Comparing Results in 3 MF countries

73 Another example…

74 principles of use Flexible: modular to be adapted to use & context Complementary: use with other methodologies Participatory: seeks dialogue and collaboration with partners Evaluative thinking: culture of reflection, results oriented thinking, and promotes social & organizational learning

75 examples of OM use: “Not everything that matters can be counted, and not everything that can be counted matters” Albert Einstein