1. Integrating Monitoring and Evaluation into a Management Framework for Scaling Up January 2016 1

2. Traditional M&E’s focus is implementation and impact at the project level, not assessing progress towards, informing strategy on, or actively supporting movement towards sustainable, population- level service delivery and results

3. Two Perspectives on the Role of Monitoring, Evaluation and Learning 3 Social Science Research Strategic Management

4. M&E Considerations in Scaling – 4 Tiers 4 Tier 1: Proof of Concept – model articulation; evaluability assessment; pilot testing; impact evaluation of “prototype” Tier 2: Refinement and Streamlining -- 2 nd stage pilots; assessment of robustness, cost-efficiency and alternatives Tier 3: Scalability Assessment – analysis of “drivers”, “spaces”, and scaling challenges Tier 4: Change Management – monitoring implementation and fidelity of scale-up; validation of efficacy at scale; continuous improvement

5. Tier 1: Proof of Concept Identify audiences, their questions, and the evidence they would find persuasive Articulate the model’s essential features: technical, financial, process and values Specify the theory of change, metrics, data sources, comparables and counter-factual(s) to be tested at the pilot stage Experiment with delivery alternatives; bundling and unbundling elements Test efficacy, validate assumptions, and document results 5

6. 6 Assume Iteration and Recalibration USAID, Development Innovation Ventures

7. Tier 2: Refining and Streamlining: Second Stage Pilots 7 Articulate a vision of delivery at scale that includes goals, roles, and a strategy for inducing systemic and sustainable change; and examine the feasibility of key assumptions Use second stage pilots to test whether the factors for success are replicable? ■ Social ■ Organizational ■ Political … and whether the model can be refined or simplified without losing effectiveness.

8. 8 Tier 3: Scalability Assessment (1) – External Key Questions: How high a priority is the problem to which this is the solution (legitimation)? What are the competing models? Who are the key stakeholders relevant to this issue and what are their positions? Is the enabling environment (“drivers” and “spaces”) conducive?

9. 9 Do organizations exist that are willing and able to implement the intervention at scale? –Are they interested in adopting the model? –What institutional strengthening or organizational change would be necessary? –What changes would need to be made to the model or organization to make them congruent? Are there institutions in place able and willing to perform the needed intermediation functions? –Fundraising –Advocacy and Marketing –Convening –Change Management –Systems Strengthening

10. Tier 3: Scalability Assessment (2): Internal 10

11. Tier 4: Change Management (1): Assessing Systemic Change and Progress Towards It Systems Policy Projects “A bad system will trump a good program – every time, all the time.” Patrick McCarthy, CEO of the Annie E. Casey Foundation Capacity - Government -Markets -Philanthropy

12. Change Management (2): Assessing Fidelity and Impact at Scale 12 Key Elements: Agreed performance metrics Institutionalized quality control and monitoring systems Client satisfaction surveys Third party evaluation within first 5 years of operations

13. Change Management (3): Keeping Change and Performance in the Public Eye Widely used approaches and techniques include: –Public oversight panels –Open-access dissemination –Third-party monitoring and evaluation –Evidence summits –Scorecards –Sustained media coverage –Informed advocacy