Assessing the evidence base for how STIP enhances development programming Annette N. Brown Director, Research and Evaluation Strategic Initiative, FHI 360 Consultant, 3ie @anbrowndc American Evaluation Association 29 October 2016

3ie scoping papers Purpose: Identify priority questions for future impact evaluation and systematic review investment in a sector or for a theme Inputs: Evidence gap map (supply of evidence) plus stakeholder consultation (demand for evidence) Photo by Antonio Cinotti

Evidence gap maps Many thanks for the invitation to present at the Foundation

David Halpern, CEO of UK’s Behavioral Insights Team, at What Works Global Summit Governments should publish what they don’t know but need to know.

Objectives of 3ie evidence gap maps (EGMs) To understand the size and scope of the evidence base for a given theme or sector. To enable policy-makers and practitioners to identify and explore existing evidence for a given theme or sector. To facilitate strategic and efficient research procurement. Impact evaluation evidence gap maps can identify promising questions for synthesis research. Impact evaluation evidence gap maps combined with information on evidence demand can identify priority questions for impact evaluation investment.

The basic EGM framework   Outcomes Output 1 Outcome 1 Outcome 2 Outcome 3 Impact 1 Impact 2 Interventions Interv 1 Interv 2 Study 1 Study 2  Study 1 Interv 3 Study 3 Study 4 Study4 Interv 4 Interv 5

What EGM do not do An EGM does not count, weight, or synthesize the evidence. That is, more evidence does not necessarily mean evidence in favor of the intervention.

Preliminary results Back in July we conducted our search – what did we get?

297 2,034 Stage 3: Full text screening 9,086 31,616 37,494 Search results from all databases and websites Stage 2: Abstract screening Stage 1: Title screening Duplicate removal (of databases) Stage 4: Second full text screening and coding 727 291 impact evaluations - 234 complete - 57 ongoing systematic reviews* - 4 complete - 2 protocol 297 Based on pre-specified search strategy, search of 12 databases and 35 websites – from that we yielded over 37k hits Very exciting, much more than expected And still expecting more once snowballing is done – results are preliminary because coding much more than expected Even with tech and innovation constraints found all this Exciting news about state of evidence

Completed impact evaluation studies Darker cells = more studies Address gap problem (briefly explain cross-hatching) Policies that affect innovation – global health Looks like there may be a number of gaps We will zoom in on some parts of the map

Completed impact evaluation studies Darker cells = more studies Address gap problem (briefly explain cross-hatching) Policies that affect innovation – global health Looks like there may be a number of gaps We will zoom in on some parts of the map

Completed impact evaluation studies Darker cells = more studies Address gap problem (briefly explain cross-hatching) Policies that affect innovation – global health Looks like there may be a number of gaps We will zoom in on some parts of the map

Evidence by intervention Technology has almost double the number of categories than the other groups

Outcome type Not surprising interventions can be randomized at ind. Level, or most studies use data from household or individual level surveys.

Sectors Global health – we know from IER that majority of impact evaluations are health related

Technology interventions To remind you, we had 191 impact evaluation studies in the technology group

Technology

Innovation ecosystems Outcomes by level of analysis LA1 LA2 LA3 Individual & household level outcomes Organizational level outcomes Community & societal outcomes Intervention categories   Innovation ecosystems Access to capital 4 Grants & subsidies 18 Policies & regulation that affect innovation 5 Networks & collaboration for innovation 3 1 Capacity building for innovation

Innovation ecosystems Outcomes by area of focus Education & Academic Global health Democracy, human rights & governance Agriculture & food security Crises & conflict Economic growth, finance & trade Environment & global climate change Water & sanitation Energy Intervention categories   Innovation ecosystems Access to capital 4 Grants & subsidies 18 Policies & regulation that affect innovation 5 Networks & collaboration for innovation 3 1 Capacity building for innovation

Evidence by country 234 completed studies Many in Asia – India, China Latin America – Brazil, Argentina, Peru, Mexico Gaps in mid east and north africa

Science interventions by country Now we can look at the heat map for each intervention grouping

Technology interventions by country Total number 192

Innovation ecosystems interventions by country Government funded programs investing in econ growth

Partnership interventions by country Haiti (1) Rwanda (4) Burundi (1) Earliest/flagship world bank RBF program in Rwanda – all studies measure impacts of the same program Indonesia (1)

Stakeholder survey

State of evidence for innovation intervention categories - Access to capital the only category with less than 25% of respondents considering the evidence base to be weak

Preliminary findings Need more evidence on impacts for community and societal outcomes Need more evidence on the impacts of technology enhancements on agricultural and food security outcomes Need more evidence on the impacts of technology enhancements on crisis & conflict and DRG outcomes Need more evidence on the impacts of policies & regulations and networks for innovation ecosystems across all sectors Need SRs for technology and education, innovation ecosystems and economic growth (esp org outcomes)