Supporting continuous improvement in the replication process Getting to Grips with Replication Seminar 3: Monitoring, evaluation & continuous improvement 21 st March 2013 Kerstin Junge, Tavistock Institute of Human Relations

Introduction Why is it useful to think of replication in continuous improvement terms? How can we use monitoring and evaluation as tools for continuous improvement in the replication process? How do these tools affect impact of the replication process? Conclusions

Why is it useful to think of replication in continuous improvement terms?

How can we use monitoring and evaluation as tools for continuous improvement in the replication process?

Monitoring and evaluation for continuous improvement in the replication process: the case of Realising Ambition Replication Phase 1: Identifying innovations for replication and deciding to adopt Requirement to engage with existing impact evidence as integral part of programme design

Monitoring and evaluation for continuous improvement in the replication process: the case of Realising Ambition Replication Phase 1: Identifying innovations for replication and deciding to adopt Requirement to engage with existing impact evidence as integral part of programme design “The application stage forced me to ask the… team: why are we doing it in this way? How do we know it works? It was challenging to ask the organisation. (…) These are hard questions. Realising Ambition clarified the thinking in the organisation.” “The standards of evidence exercises in the application forms (…) helped us to really get to know [the intervention). They helped us test [it] in a systematic way to see if we wanted to use it. It helped me to look forward and see what the evidence said. (…) the evidence was stronger when teachers delivered the intervention rather than external people so we are using teachers.” Application process challenged us, (…) it asked for some quite robust information and we had to organise ourselves to get that. [It] challenged us to smarten up back end, e.g. pulling the logic model together; we knew it and played around with doing that and had not expressed that in a document. (…) we actually said even if we don’t get that, [the application process] actually helped us organise ourselves (…).”

Monitoring and evaluation for continuous improvement in the replication process: the case of Realising Ambition Replication Phase 1: Identifying innovations for replication and deciding to adopt Requirement to engage with existing impact evidence as integral part of programme design “The standards of evidence exercises in the application forms (…) helped us to really get to know [the intervention). They helped us test [it] in a systematic way to see if we wanted to use it. It helped me to look forward and see what the evidence said. (…) the evidence was stronger when teachers delivered the intervention rather than external people so we are using teachers.” “The application stage forced me to ask the… team: why are we doing it in this way? How do we know it works? It was challenging to ask the organisation. (…) These are hard questions. Realising Ambition clarified the thinking in the organisation.” Application process challenged us, (…) it asked for some quite robust information and we had to organise ourselves to get that. [It] challenged us to smarten up back end, e.g. pulling the logic model together; we knew it and played around with doing that and had not expressed that in a document. (…) we actually said even if we don’t get that, [the application process] actually helped us organise ourselves (…).” Creating preconditions for impact through: better and explicit knowledge of the intervention and effective delivery, clearer understanding of causal pathways, preparing the organisation for replication Embeds evidence based and reflective practice in programme community

Monitoring and evaluation for continuous improvement in the replication process: the case of Realising Ambition Process evaluation Sense-making and capturing of lessons learnt to support future replication investments Analysing programme architecture, programme design and application process Process evaluation Sense-making and capturing of lessons learnt to support future replication investments Analysing programme architecture, programme design and application process Replication Phase 1: Identifying innovations for replication and deciding to adopt Requirement to engage with existing impact evidence as integral part of programme design “The standards of evidence exercises in the application forms (…) helped us to really get to know [the intervention). They helped us test [it] in a systematic way to see if we wanted to use it. It helped me to look forward and see what the evidence said. (…) the evidence was stronger when teachers delivered the intervention rather than external people so we are using teachers.” “The application stage forced me to ask the… team: why are we doing it in this way? How do we know it works? It was challenging to ask the organisation. (…) These are hard questions. Realising Ambition clarified the thinking in the organisation.” Application process challenged us, (…) it asked for some quite robust information and we had to organise ourselves to get that. [It] challenged us to smarten up back end, e.g. pulling the logic model together; we knew it and played around with doing that and had not expressed that in a document. (…) we actually said even if we don’t get that, [the application process] actually helped us organise ourselves (…).” Creating preconditions for impact through: better and explicit knowledge of the intervention and effective delivery, clearer understanding of causal pathways, preparing the organisation for replication Embeds evidence based and reflective practice in programme community

Monitoring and evaluation for continuous improvement in the replication process: the case of Realising Ambition Replication phase 2: first wave implementation Monitoring system Keeping programme on track, understanding challenges, improving delivery

Monitoring and evaluation for continuous improvement in the replication process: the case of Realising Ambition Replication phase 2: first wave implementation Preparation for (new) impact evaluations Monitoring system “One of the most significant things was (…) looking at how we constructed the logic model. (…) Now it's much easier to be specific.” Webinars “(…) spark thinking which can feed into plans and keep them on track.” “Some of the work that is being on theory of change; understanding replication and fidelity. What [intervention owners] do is set up the programme, train the staff and give us assistance. But they don’t engage staff in discussing in understanding programme fidelity and replication. “ Keeping programme on track, understanding challenges, improving delivery

Monitoring and evaluation for continuous improvement in the replication process: the case of Realising Ambition Replication phase 2: first wave implementation Preparation for (new) impact evaluations Monitoring system Probability of achieving desired impact increases through: greater specificity of intervention and expected outcomes as well as improved understanding the importance of fidelity in delivery. “One of the most significant things was (…) looking at how we constructed the logic model. (…) Now it's much easier to be specific.” Webinars “(…) spark thinking which can feed into plans and keep them on track.” “Some of the work that is being on theory of change; understanding replication and fidelity. What [intervention owners] do is set up the programme, train the staff and give us assistance. But they don’t engage staff in discussing in understanding programme fidelity and replication. “ Keeping programme on track, understanding challenges, improving delivery Offers data, learning and knowledge to guide and improve replication delivery

Monitoring and evaluation for continuous improvement in the replication process: the case of Realising Ambition Replication phase 2: first wave implementation Process evaluation Sense-making and capturing of lessons learnt to support future replication investments Identification and (early) definition of replication models; overview of types and benefits of impact evaluation and other support ; replication progress indicators Process evaluation Sense-making and capturing of lessons learnt to support future replication investments Identification and (early) definition of replication models; overview of types and benefits of impact evaluation and other support ; replication progress indicators Preparation for (new) impact evaluations Monitoring system Probability of achieving desired impact increases through: greater specificity of intervention and expected outcomes as well as improved understanding the importance of fidelity in delivery. “One of the most significant things was (…) looking at how we constructed the logic model. (…) Now it's much easier to be specific.” Webinars “(…) spark thinking which can feed into plans and keep them on track.” “Some of the work that is being on theory of change; understanding replication and fidelity. What [intervention owners] do is set up the programme, train the staff and give us assistance. But they don’t engage staff in discussing in understanding programme fidelity and replication. “ Keeping programme on track, understanding challenges, improving delivery Offers data, learning and knowledge to guide and improve replication delivery

Effects on impact Creating preconditions for impact: – Better and explicit knowledge of the intervention and effective delivery – Clearer understanding of causal pathways – Preparing the organisation for replication Greater probability of achieving desired impact: – Greater specificity of intervention and expected outcomes – Improved understanding the importance of fidelity in delivery – Data, knowledge and learning to improve replication delivery

Conclusions Understanding replication as a continuous improvement process offers a ‘practical’ framework to guide activities, as well as tools Monitoring and evaluation tools should be an integral part of the replication process from the start, both at project level and at programme / policy level – Creates a structure to support constructive engagement with evidence, evidence based practice, learning from experience – And through this supports learning and continuous improvement It means understanding monitoring and evaluation at least as much about learning as about ‘auditing’ Possibility of not only ‘single loop learning’ but also ‘double loop learning’ (Argyris and Schön)? – Not just continuous improvement of implementation (revisiting ‘action strategies’) – But also revisiting values [mental models, beliefs, intentions] underpinning replication actions and strategies (‘governing variables’)