1. Measuring Outreach Effectiveness Web4Dev – November 2006 Alex McKenzie - amckenzie@worldbank.org Knowledge & Evaluation Capacity Development Independent Evaluation Group – World Bank

2. Key Points: ►Share IEGWB’s approach for aligning web/online practice with mandate and business strategy ►Share our experience running several outreach campaigns, discuss metrics ►Measuring effectiveness and impact ►Would appreciate hearing about your own experience and/or feedback

3. Business Strategy ►IEG is an independent unit that assesses the effectiveness of World Bank Group’s interventions ►Well regarded for its evaluation capabilities and reports ►Combines Accountability & Learning ►“To be effective, IEG want’s to be an influential and reliable source of knowledge” ►New Outreach and Communications Strategy: –Target key audiences –Embed outreach throughout the evaluation process –Accelerate learning, encourage utilization –Measure results ►Decided on a campaign-based approach, 10 to 12 major evaluation products per year. ►Multi-pronged outreach campaigns: media, print, online, in-person, translations, etc. ►Piloted many new ideas, mainstreamed what worked

4. Key Audiences ►Systematic effort to identify audiences has led to two sets: “core” & campaign-specific (specialists) Identify decision-makers & influencers Identify key network nodes ►Evolving audience criteria: from “static” classifications, to others based on opportunity or time-sensitive demand ►Customize products to audience whenever economically feasible (e.g. conference CDROMs, translations, etc.)

5. Embed in Evaluation Process Web support for:Design PhaseResearch PhaseOutreach Phase Communications- Release approach paper - Interim data and research - Audience identification - Outreach campaign* - Campaign metrics Process support- Virtual team workspace - Research surveys - Feedback, discussion, comments Value Created>>Transparency, InclusionTransparency, EfficiencyKey messages for relevant audiences * Initial role for web

6. Encourage Utilization

8. Measuring Results ►Results-Based Management causal chain

9. Measuring Results – Causal Chain Source: “Strengthening EAP’s Results Orientation”, April 2004

10. Measuring Results - Outputs

12. Measuring Results - Outcomes ►Intermediate Outcomes: perception survey* of IEG evaluation products, covering: –Readership –Quality –Influence on Bank processes and policies (similar for external clients) –Utilization ►Annual Review of Operations Evaluation (AROE) 2006, to be released in November/December. *Source: IEG’s Annual Review of Operations Evaluation (AROE). See http://www.worldbank.org/ieg/aroe http://www.worldbank.org/ieg/aroe

13. Measuring Results - Outcomes ►Survey targeted individuals defined as intended audience: 4285 internal (WBG) and 2759 external contacts. Results are indicative for respondents (22%) but are not generalizable to population Readership InternalExternal Direct awareness:42%55% Overall awareness:56%76% heard via email64%69% visited web site12%28% received hard-copy20%15%

14. Perception of Quality – Staff/EDs (E4)

15. Perception of Quality – External Clients (E13)

16. Influence – Staff (E8)

17. Influence – External Clients (E15)

18. Staff Use of Evaluations (E9)

19. External Clients Use of Evaluations (E16)

20. Measuring Results – Survey Implications IEG provides good services to its main client, the Board. External audiences are also highly satisfied with the quality of IEG evaluations and considered them to be influential. However external audiences would like IEG to broaden its external consultation during evaluations. Bank staff see IEG products useful with deepening their understanding of a given subject. However they don’t see them as useful in day-to-day work. IEG needs to focus on influencing ongoing and future operations.

21. Measuring Results – Survey Implications ►All audiences suggest: High value placed on timeliness to provide information when decisions need to be made Deepening analysis through more consultation and attention to context Focus on operational value of findings ►A call for action for IEG’s outreach program 4-page briefs Customized “findings” brief, upstream, via email More use of non-print media Balance “lecturer” approach with conversations

22. Thoughts Going Forward ►"What information consumes is rather obvious; it consumes the attention of its recipients. Hence, a wealth of information creates a poverty of attention." Herbert Simon, Nobel laureate

23. Your Experience/Feedback ►What has worked in your organization? (audience analysis; relevance/timeliness; measuring outputs, outcomes, impacts, etc.) ►Choices related to channels? High quality content vs. open/collaborative models ►Outreach models used? Worthwhile consultants & advice? ►What challenges do you see ahead?