Applying systemic perspectives and lean methods to take utilization-focused evaluation further Henrik wiréhn ramböll Management consulting SVUF 2017-10-19

agenda 01 Introduction: Why is this paper written? 02 Theoretical background: UFE, Systemic perspectives, Lean 03 Pilot case assignment Pilot case solution 04 Conclusions: Strengths, weaknesses, future studies 05 Discussion: Your learnings and questions 06

Why should we make evaluations? An evaluation report of high quality is finalized The evaluation is used Production of an evaluation Positive changes in society

Increased focus on utilization Evaluations should contribute to positive changes in society Focus has been moving towards use, in order to contribute to positive changes in society Utilization Focused-Evaluation (”UFE”) has stressed this How can the use of evaluations be further improved? Where can good practices be found? High quality report is finalized Positive changes in society Produce evaluation The evaluation is used

Organizational theory works with utilization The policy cycle describes how policies are planned, implemented and evaluated The utilization of evaluations can be seen as strategic planning and implementation Organizational theory works with strategic planning and implementation Could organizational theory increase the utilization of evaluation? Based on a pilot case, this paper examines if lean and systemic perspectives can be combined with UFE to further improve the utilization of evaluations Needs analysis Strategic planning of changes Implementation Monitoring Evaluation

agenda 01 Introduction: Why is this paper written? 02 Theoretical background: UFE, Systemic perspectives, Lean 03 Pilot case assignment Pilot case solution 04 Conclusions: Strengths, weaknesses, future studies 05 Discussion: Your learnings and questions 06

UFE can include a wide range of evaluation methods UFE begins with the premise that evaluations should be judged by their utility and actual use The primary intended users must be identified and personally engaged to identify their primary intended uses These intended uses should guide all other decisions about the evaluation process UFE is a guiding framework – not a methodology. It can include a wide variety of evaluation methods within an overall participatory paradigm Use concerns how real people apply evaluation findings and experience and learn from the evaluation process How do you create such a process?

Systemic perspectives support process facilitation Systemic theory was used in family theory to understand human interaction It is now used in change management for individuals, groups and organizations Connected to post-modernist tradition The relationship (and not the individual) is the smallest unit in understanding the world To understand the meaning of other’s action and be able to cooperate, be aware of the other’s context Communication is central Ramboll uses systemic perspectives to facilitate meetings to Engage workshop participants Create a common understanding Visualize the participants’ different perspectives Concrete tools, methods and as an approach

lean has methods for analyzing and improving organizations Lean is based on research showing how Toyota realized high-quality products in a resource-efficient manner The purpose is to make an organization create high quality services efficiently with customer expectations in focus Adapted to other sectors Management philosophy and practices 4 principles Teamwork – bottom up with engaged employees Communication Efficient use of resources and waste elimination Continuous improvement Customer value is the basis for prioritizing – holistic view Finding root causes – long term

agenda 01 Introduction: Why is this paper written? 02 Theoretical background: UFE, Systemic perspectives, Lean 03 Pilot case assignment Pilot case solution 04 Conclusions: Strengths, weaknesses, future studies 05 Discussion: Your learnings and questions 06

The combination was tested in a pilot case The combination of UFE, systemic perspectives and lean was tested in a pilot case 2014-2015 The client was the Swedish Government offices The goal of the project was to develop a method to evaluate and develop the work of several organizations working against domestic violence

Problem description for the case – ideal situation In the ideal situation, organizations work coordinated towards the needs of citizens Organizations Citizens Socialtj änst Vad tar ni med er från idag? Vad är nästa steg föer ?

Problem description for the case – types of problems Gap The citizen does not get support. 2. Overlap The citizen does the same thing twice. 3. Different directions …of information or actions to the citizen

agenda 01 Introduction: Why is this paper written? 02 Theoretical background: UFE, Systemic perspectives, Lean 03 Pilot case assignment Pilot case solution 04 Conclusions: Strengths, weaknesses, future studies 05 Discussion: Your learnings and questions 06

Communal analysis on walls is central To the method Employees and first line managers from the involved organizations were analyzing together in workshops 15

Implementation and continuous improvements The steps of the method Start-up Situation analysis Improvements Implementation and continuous improvements Adjust the method to local needs Formulate the purpose to evaluate against. Focus groups with victims too Understand individuals’ routes by mapping and evaluating real cases by Lean process mapping and identi- fying challenges Analyse root causes by data and the Lean method ”5 Why?” Identify solutions to the root causes and prioritize the solutions in a diagram with value and simplicity on the axes Create an action plan Detail the action plan and form teams Implement, follow up and improve continuously with PDCA-cycle

Visualization of the communal analysis towards an action plan Individual’s route Process mapping General routes Process mapping Root causes and solutions 5 Why Prioritizing Action plan Action plan Content slide, two columns with image. Image size: 8,46 cm x 10,76 cm or 320 x 407 pixels

Guiding principles Improve holistically Focus specifically on the relationships between the various parts (eg. between different entities or organizations) Have a listening approach and involve the target group, employees and first-line managers Focus on what the target group values Find the root causes instead of symptoms Create the necessary conditions for improvements that last and develop over time

agenda 01 Introduction: Why is this paper written? 02 Theoretical background: UFE, Systemic perspectives, Lean 03 Pilot case assignment Pilot case solution 04 Conclusions: Strengths, weaknesses, future studies 05 Discussion: Your learnings and questions 06

The pilot method has been appreciated by all involved The Swedish Government offices’ investigation chose the pilot method as one of their main propositions The users in the pilot case evaluated the method thoroughly with the result 4.2/5 The pilot method was used on several other locations even before it was published and ~100 practitioners have described how they can use different parts of the method The target group of victims of domestic violence appreciated that the pilot used their perspectives as the starting point What strengths in combining the three traditions made the pilot case appreciated?

Summary of strengths There is an overlap between the three, so many strengths have roots in more than one tradition UFE provides a well established framework for making evaluations Evaluation on how to systematically gather and analyse data, to measure against something and go behind the data to explain, e.g. the logic behind (TOC) Lean has established methods and tools for analysing and improving organizations Systemic perspectives support process facilitation of meetings

Strengths of lean Process mapping – focus on the target group Root causes by “5 Why?” Prioritizing based on value and simplicity Concrete action plan or Gantt chart Implementation teams with team leaders Cycle of continuous improvements: Plan-Do-Check-Adjust Power to the target group and users / employees: Target group defines the needs, what to strive for and evaluate against – A common goal Users describe the current situation and analyse Lean provides established tools for analysing organizations in workshops, crafting recommendations and methods for making recommendations long term sustainable in the organizations

Strengths of systemic perspectives The systemic process facilitator has the role of listening and encouraging the workshop participants to listen to each other, understand each other and build trust Example of co-creation: Present whole process for feedback All exercises visible Check-in and check-out Written feedback Creates group processes in workshops where you involve, engage and empower everyone, build a long term stable group and create a common understanding while gathering a large amount of evaluation data, draw shared conclusions and move the process rapidly forward

weaknesses Post-modernist clash between systemic perspectives contra lean and evaluation tradition Who has the “truth”? Focus on positive in systemic Appreciative Enquiry and negative in Lean Adjust to local needs Test a positive focus The pilot method has mainly been tested in a specific situation – difficult to draw general conclusions Several organizations Common goal (for a common target group) Several employees involved in the work and in the workshops

Future studies: can we learn about use from organizational development? The combination of the traditions seems to give good results in the pilot This indicates a potential to increase evaluation use and impact on society on a greater scale What needs to be answered to reach that? The three traditions seem to have moved closer to each other, based on needs from users. Can they form a common approach? How are they interlinked? Has the combination been tried before? Learnings? Results? What circumstances are needed? What combinations function better? Can UFE be further improved by inspiration from the others? Can the pilot method be further improved? By focusing on strengths? What can evaluation learn from organizational development? And opposite

agenda 01 Introduction: Why is this paper written? 02 Theoretical background: UFE, Systemic perspectives, Lean 03 Pilot case assignment Pilot case solution 04 Conclusions: Strengths, weaknesses, future studies 05 Discussion: Your learnings and questions 06

Discussion: learnings and questions Discussion in pairs: What did you learn? – What can you use at work? Questions? Communal discussion

Thank you! mboll.com 08-562 494 49 Henrik Wiréhn henrik.wirehn@ramboll.com 070-754 94 48