1. Applying systemic perspectives and lean methods to take utilization-focused evaluation further
Henrik wiréhn
ramböll Management consulting
SVUF

2. agenda 01 Introduction: Why is this paper written? 02
Theoretical background: UFE, Systemic perspectives, Lean 03
Pilot case assignment
Pilot case solution 04
Conclusions: Strengths, weaknesses, future studies 05
Discussion: Your learnings and questions 06

3. Why should we make evaluations?
An evaluation report of high quality is finalized
The evaluation is used
Production of an evaluation
Positive changes in society

4. Increased focus on utilization
Evaluations should contribute to positive changes in society
Focus has been moving towards use, in order to contribute to positive changes in society
Utilization Focused-Evaluation (”UFE”) has stressed this
How can the use of evaluations be further improved? Where can good practices be found?
High quality report is finalized
Positive changes in society
Produce evaluation
The evaluation is used

5. Organizational theory works with utilization
The policy cycle describes how policies are planned, implemented and evaluated
The utilization of evaluations can be seen as strategic planning and implementation
Organizational theory works with strategic planning and implementation
Could organizational theory increase the utilization of evaluation?
Based on a pilot case, this paper examines if lean and systemic perspectives can be combined with UFE to further improve the utilization of evaluations
Needs analysis
Strategic planning of changes
Implementation
Monitoring
Evaluation

6. agenda 01 Introduction: Why is this paper written? 02
Theoretical background: UFE, Systemic perspectives, Lean 03
Pilot case assignment
Pilot case solution 04
Conclusions: Strengths, weaknesses, future studies 05
Discussion: Your learnings and questions 06

7. UFE can include a wide range of evaluation methods
UFE begins with the premise that evaluations should be judged by their utility and actual use
The primary intended users must be identified and personally engaged to identify their primary intended uses
These intended uses should guide all other decisions about the evaluation process
UFE is a guiding framework – not a methodology. It can include a wide variety of evaluation methods within an overall participatory paradigm
Use concerns how real people apply evaluation findings and experience and learn from the evaluation process
How do you create such a process?

8. Systemic perspectives support process facilitation
Systemic theory was used in family theory to understand human interaction
It is now used in change management for individuals, groups and organizations
Connected to post-modernist tradition
The relationship (and not the individual) is the smallest unit in understanding the world
To understand the meaning of other’s action and be able to cooperate, be aware of the other’s context
Communication is central
Ramboll uses systemic perspectives to facilitate meetings to
Engage workshop participants
Create a common understanding
Visualize the participants’ different perspectives
Concrete tools, methods and as an approach

9. lean has methods for analyzing and improving organizations
Lean is based on research showing how Toyota realized high-quality products in a resource-efficient manner
The purpose is to make an organization create high quality services efficiently with customer expectations in focus
Adapted to other sectors
Management philosophy and practices
4 principles
Teamwork – bottom up with engaged employees
Communication
Efficient use of resources and waste elimination
Continuous improvement
Customer value is the basis for prioritizing – holistic view
Finding root causes – long term

10. agenda 01 Introduction: Why is this paper written? 02
Theoretical background: UFE, Systemic perspectives, Lean 03
Pilot case assignment
Pilot case solution 04
Conclusions: Strengths, weaknesses, future studies 05
Discussion: Your learnings and questions 06

11. The combination was tested in a pilot case
The combination of UFE, systemic perspectives and lean was tested in a pilot case
The client was the Swedish Government offices
The goal of the project was to develop a method to evaluate and develop the work of several organizations working against domestic violence

12. Problem description for the case – ideal situation
In the ideal situation, organizations work coordinated towards the needs of citizens
Organizations
Citizens
Socialtj änst
Vad tar ni med er från idag? Vad är nästa steg föer ?

13. Problem description for the case – types of problems
Gap
The citizen does not get support.
2. Overlap
The citizen does the same thing twice.
3. Different directions
…of information or actions to the citizen

14. agenda 01 Introduction: Why is this paper written? 02
Theoretical background: UFE, Systemic perspectives, Lean 03
Pilot case assignment
Pilot case solution 04
Conclusions: Strengths, weaknesses, future studies 05
Discussion: Your learnings and questions 06

15. Communal analysis on walls is central To the method
Employees and first line managers from the involved organizations were analyzing together in workshops 15

16. Implementation and continuous improvements
The steps of the method
Start-up
Situation analysis
Improvements
Implementation and continuous improvements
Adjust the method to local needs
Formulate the purpose to evaluate against. Focus groups with victims too
Understand individuals’ routes by mapping and evaluating real cases by Lean process mapping and identi- fying challenges
Analyse root causes by data and the Lean method ”5 Why?”
Identify solutions to the root causes and prioritize the solutions in a diagram with value and simplicity on the axes
Create an action plan
Detail the action plan and form teams
Implement, follow up and improve continuously with PDCA-cycle

17. Visualization of the communal analysis towards an action plan
Individual’s route Process mapping
General routes Process mapping
Root causes and solutions
5 Why
Prioritizing
Action plan
Action plan
Content slide, two columns with image. Image size: 8,46 cm x 10,76 cm or 320 x 407 pixels

18. Guiding principles Improve holistically
Focus specifically on the relationships between the various parts (eg. between different entities or organizations)
Have a listening approach and involve the target group, employees and first-line managers
Focus on what the target group values
Find the root causes instead of symptoms
Create the necessary conditions for improvements that last and develop over time

19. agenda 01 Introduction: Why is this paper written? 02
Theoretical background: UFE, Systemic perspectives, Lean 03
Pilot case assignment
Pilot case solution 04
Conclusions: Strengths, weaknesses, future studies 05
Discussion: Your learnings and questions 06

20. The pilot method has been appreciated by all involved
The Swedish Government offices’ investigation chose the pilot method as one of their main propositions
The users in the pilot case evaluated the method thoroughly with the result 4.2/5
The pilot method was used on several other locations even before it was published and ~100 practitioners have described how they can use different parts of the method
The target group of victims of domestic violence appreciated that the pilot used their perspectives as the starting point
What strengths in combining the three traditions made the pilot case appreciated?

21. Summary of strengths
There is an overlap between the three, so many strengths have roots in more than one tradition
UFE provides a well established framework for making evaluations
Evaluation on how to systematically gather and analyse data, to measure against something and go behind the data to explain, e.g. the logic behind (TOC)
Lean has established methods and tools for analysing and improving organizations
Systemic perspectives support process facilitation of meetings

22. Strengths of lean Process mapping – focus on the target group
Root causes by “5 Why?”
Prioritizing based on value and simplicity
Concrete action plan or Gantt chart
Implementation teams with team leaders
Cycle of continuous improvements: Plan-Do-Check-Adjust
Power to the target group and users / employees:
Target group defines the needs, what to strive for and evaluate against – A common goal
Users describe the current situation and analyse
Lean provides established tools for analysing organizations in workshops, crafting recommendations and methods for making recommendations long term sustainable in the organizations

23. Strengths of systemic perspectives
The systemic process facilitator has the role of listening and encouraging the workshop participants to listen to each other, understand each other and build trust
Example of co-creation:
Present whole process for feedback
All exercises visible
Check-in and check-out
Written feedback
Creates group processes in workshops where you involve, engage and empower everyone, build a long term stable group and create a common understanding while gathering a large amount of evaluation data, draw shared conclusions and move the process rapidly forward

24. weaknesses
Post-modernist clash between systemic perspectives contra lean and evaluation tradition
Who has the “truth”?
Focus on positive in systemic Appreciative Enquiry and negative in Lean
Adjust to local needs
Test a positive focus
The pilot method has mainly been tested in a specific situation – difficult to draw general conclusions
Several organizations
Common goal (for a common target group)
Several employees involved in the work and in the workshops

25. Future studies: can we learn about use from organizational development?
The combination of the traditions seems to give good results in the pilot
This indicates a potential to increase evaluation use and impact on society on a greater scale
What needs to be answered to reach that?
The three traditions seem to have moved closer to each other, based on needs from users. Can they form a common approach? How are they interlinked?
Has the combination been tried before? Learnings? Results? What circumstances are needed? What combinations function better?
Can UFE be further improved by inspiration from the others?
Can the pilot method be further improved? By focusing on strengths?
What can evaluation learn from organizational development? And opposite

26. agenda 01 Introduction: Why is this paper written? 02
Theoretical background: UFE, Systemic perspectives, Lean 03
Pilot case assignment
Pilot case solution 04
Conclusions: Strengths, weaknesses, future studies 05
Discussion: Your learnings and questions 06

27. Discussion: learnings and questions
Discussion in pairs:
What did you learn? – What can you use at work?
Questions?
Communal discussion

28. Thank you! mboll.com 08-562 494 49 Henrik Wiréhn