Outcome Mapping Team Meeting February 2010

2 Key evaluation challenges measuring development results establishing cause & effect in an open system timing encouraging iterative learning clarifying values

3 problem with « impact » Impact Implies: Cause & effect Positive, intended results Focus on ultimate effects Credits a single contributor Story ends when program obtains success Development Implies: Open system Unexpected positive & negative results occur Upstream effects are important Multiple actors create results & need credit Change process never ends

4 Recognizing that change is… Continuous Complex Non-linear Multidirectional Not controllable

5 OM - a flexible, multiple-use tool Planning Monitoring Evaluation

SPHERE OF CONTROL SPHERE OF INFLUENCE SPHERE OF INTEREST Thanks to Paul Crawford Boundary Partners Ultimate beneficiaries RESOURCES OUTPUTS OUTCOMES IMPACT GeSCI Strategic partners

7 focus on direct partners Key concept is « boundary partners » The individuals, groups, and organizations you work with directly and anticipate opportunities for influence

Focus of Outcome Mapping Behavioural Changes Local partners / beneficiaries ownership increases Program influence decreases InputsActivitiesOutputsOutcomesImpacts

9 Why behaviour changes? To stress that development is done by and for people To illustrate that although a program can influence the achievement of outcomes, it cannot control them because ultimate responsibility rests with the people affected

10 Contribution not attribution You can influence but not control change in your partners

11 PLANNING: articulate goals & define activities MONITORING: assess program performance & partners’ outcomes EVALUTION: design & conduct a use- oriented evaluation Primary uses

The Questions that Outcome Mapping will help to answer : In designing and articulating the program’s logic - What are our development goals? - How can our programme contribute to those development goals? - Who are our beneficiaries? - How can we help our beneficiaries contribute to the broader development goals? Recording internal and external monitoring data - How far have our beneficiaries progressed towards achieving outcomes? - What are we doing to support the achievement of those outcomes? - How well have we performed? Indicating cases of positive performance and areas of improvement - What worked well? Why? - How can we maximise our contributions? Kiely, Team Meeting, December 2007

The Questions that Outcome Mapping will help to answer (continued) : Evaluating intended or unexpected results - Who changed / what changed? How? - If they did not change as expected, do we need to do something different or reorient our expectations? Gathering data on the contribution that a program made to bringing about changes in its partners - What activities / strategies were used? - How did the activities influence the individuals, groups, or institutions to change? Establishing evaluation priorities and an evaluation plan - What strategies, relationships, or issues need to be studied in depth? - How, and from where, can we gather relevant data?

Components of Outcome Mapping A description of the macro-level changes to which program intends to contribute vision, mission, boundary partners and outcome challenges, progress markers Strategy maps Organisational Practices Journal A monitoring framework outcome journal, strategy journal, performance journal Evaluation Plan

15 boundary partners Those individuals, groups, and organizations with whom the program interacts directly to effect change anticipates opportunities for influence engages in mutual learning

16 Boundary partners have boundary partners programprogram’s bpbp’s bp

17 describe: the ideal behavioural changes (relationships, activities & actions) of a boundary partner how they will contribute ideally to the vision. Outcome challenges

18 Example: Outcome Challenge Statement (IDRC) Vibrant and dynamic regional and national knowledge building and sharing communities which will produce and share high quality evidence based research and experiences on ICT4E and KS. These communities will work with GeSCI in a pro-active and committed way to identify critical issues, develop appropriate solutions and support one another on a peer basis. They will make optimal use of the GeSCI resources, tools and expertise. They will commit themselves to participate in leadership capacity building for ICT4E and KS and acquire and apply critical identified skills in their respective fields for system transformation and innovation.

19 Progress markers ✓ A graduated set of statements describing a progression of changed behaviours in the boundary partner ✓ Describe changes in actions, activities and relationships leading to the ideal outcome; shows story of change ✓ Articulate the complexity of the change process ✓ Can be monitored & observed ✓ Permit on-going assessment of partner’s progress (including unintended results)

Example Boundary partnerOutcome challenge Government officials and policymakers Ministries of Education(s)Endorsement of the ICT initiatives, approval of the implementation details and establishing guidelines for future implementation. Progress markers Expect to see 1.Negotiating with SchoolNet and other service providers on details of implementation 2.Setting a priority list for implementation 3.Taking control of the initiative Like to see 1.Efficient coordination of partnerships for delivery of ICT services to schools. 2.Establishing guidelines for integrating ICT effectively into learning and teaching 3.Setting operational objectives for implementation based on the ICT policy for education 4.Direct budgeting for expenditure by the Ministries of Education on the programme to be specified in the Policy Implementation Plan Love to see 1.Creating the critical mass of ICT skills at school level to be able to realise the country’s vision for the future. 2.Setting specific guidelines on cost-effectiveness for ICT solutions for schools or putting implementation out to tender to see who offers the best option 3.Using the ICT initiative to facilitate a change in emphasis to student-centred learning and teaching.

21 Taken together, progress markers facilitate mid-course corrections and improvements stimulate the program to consider how it can contribute to the most profound transformation possible articulate the complexity of change

22 X

23 Strategy Maps Outlines the program`s approach in working with the boundary partner Breaks strategies down into different types so you can be as strategic and influential as possible

24 6 types of strategies Aimed at the Boundary Parnter Aimed at the Boundary Partner`s Environment StrategySupportivePersuasiveCausal I-1 Direct Output I-2 Arouse New Skills/ Thinking I-3 Supporter who guides change over time E-2 Modify the information system E-3 Create / Strengthen a Peer Network E-1 Alter physical or regulatory environment

25 Practice: Facilitation Questions

M&E Planning

27 Use(s) and User(s) – Who will take action(s) or make decisions with the help of the M&E process or findings? What actions and/or decisions will be taken? QuestionsTimingMethods & Data Sources Responsibility & Roles Dissemination & Communication Costs What do you want to know about our programmes? When are the findings needed in order to be useful to the user(s)? When will the data be collected? How will data be collected? (I.e., instruments) From what sources? Who is going to lead the process? Who needs to play what role to accomplish the work? (eg., who will manage collection? analysis? writing/reporting?) How will the findings be communicated to the intended user(s)? What should the final products look like in order to be useful? Who else will be interested in the findings and should be informed of them? What human and financial resources are needed and who contributes? What resources are available or not?

28 different tools you can use to collect data Outcome/ strategy mapping journals most significant change methodology project cycle analysis activity system analysis any social science tool or method ….

30 outcome journal To understand the change process in boundary partners. Collects information about: Story of change and reasons for change Unexpected changes The actors and factors that contributed to that change How we know the change occurred Learnings

31 outcome journal Work Dating From/To: Contributors to Monitoring Update: Low = Medium = High = Outcome Challenge: LMH Expect to see:Who? 3 1 2

32 set values for low, medium & high Can capture either quantity or quality of change(s): – Quantity (number of boundary partners) – Quality (depth of change)

34 strategy journal To understand how the programme is contributing to changes in boundary partners. Collects information on: Resources that have been invested Strategies and Activities Products Effectiveness Changes that need follow up Learnings

35 strategy journal Working Dating From/To: Contributors to Monitoring Update: Strategy to be Monitored: Effectiveness? (How did it help the boundary partner?) Outputs

36 how to use the journals alone or combined with other tools and methods focus groups, interviews, e-surveys, conference calls, revision of documents can incorporate quantitative and qualitative methods face-to-face workshops other…

37 new technologies for data collection discussion forums handhelds participatory video survey monkey blogs chat skype what else…?

38 Expanding accountabilities OM helps you: manage multiple accountabilities: up, down, horizontal by: –being able to document & communicate outcomes (traditional understanding of accountability) –engage in social learning with your boundary partners making you more accountable to yourself, your organization, and them

39 Taking up M&E findings What should we keep doing? What do we need to change in order to improve? What strategies/practices do we need to add? What strategies/practices do we need to drop? (i.e., they produced no results, they require too much efforts or resources to produce results?) Has any issue come up that we need to evaluate in greater depth? What? When? Why? How?

40 Reviewing programme design 1. Read the vision statement Does this still reflect the program's dream? 2. Read the mission statement Is this the greatest contribution our program can make? Have we been doing this? Why? Why not? Should we add anything or take anything away? 3. Review boundary partners Is this who we are working with directly? Do we need to work with anyone else? 4. Review outcomesDo these accurately describe the ideals way that our boundary partners could act to contribute to the achieving of the vision? 5. Review progress markers Was the change process we set out accurate and useful? What now needs to be added or taken out? 6. Review strategiesWhat did we plan to do? Have we implemented these activities? Why? Why not? 7. Review organizational practices Are we doing everything we can to maintain our capacity to support our partners?

41 Discussion questions What elements of OM would we like to integrate that we think would be helpful? Why? What do we need to do to make this happen? What spaces, methodologies & tools do we already use that OM could complement? (how can we integrate OM into what we are already doing?) What challenges will we face? What opportunities are there to take advantage of? How will we communicate to others? What will be our next steps?

OM 2009 Review of Pilot OM Implementation within Culture for Organizational Learning STRENGTHS Identify areas where OM was used to provide a benefit to organization, teams and partners in 2009 WEAKNESSES Identify areas of organizational practice which needed improvement in order to achieve desired outcome of OM integration in 2009 OPPORTUNITIES Identify areas of OM that could support organizational learning and development in that are not currently being utilized THREATS Identify possible obstacles or problems to the integration of OM horizontally across our programmes in 2010