Petter Øgland, Department of Informatics, University of Oslo The effect of EFQM self-assessments on action research performance: Lessons on how to improve the efficiency and effectiveness of research on New Public Management Petter Øgland, Department of Informatics, University of Oslo NEON, 25th-27th November 2014

Plan for presentation Motivation (3 minutes) Problem: Action research is important but risky design for studying NPM Possible solution: Use TQM for controlling and improving action research EFQM model and double-loop learning (7 minutes) Hypothesis 1: EFQM improves EFFICIENCY of action research Hypothesis 2: EFQM improves EFFECTIVENESS of action research Hypothesis 3: EFQM improves LEARNING about improving action research Testing the hypotheses in a NTAX action research study (7 minutes) The NTAX action research is slow and costly EFQM succeeds in diagnosing and suggesting treatments Discussion: Controlling and improving AR within AR traditions based on interpretivism, post-positivism and critical theory Conclusions (3 minutes) Critical theory allows EFQM to align with action research Action researchers should use NPM logic for improving AR performance

New Public Management (NPM): Managerial logic vs. practitioner logic How can practitioner logic be understood in the context of an economic/managerial need to improve cost-efficiency?

Action research is a way of building knowledge by designing and evaluating social interventions (Clark, 1972) But it is a research design with high risk for failure (Simonsen, 2007)

What if we applied the logic of NPM on action research? 1. Define the production process: Producing change and scientific knowledge 2. Improve the production process: Reducing costs and cycle time in AR process by use of TQM methods

Using the European Foundation for Quality Management (EFQM) model

Theoretical framework (Argyris & Schön, 1978) 1. EFFICIENCY of action research (strategy) 2. EFFECTIVENESS of action research (goals) 3. LEARNING HOW TO DESIGN Single-loop and double-loop learning

Use TQM-model (EFQM) to evaluate consequence of action strategy Hypothesis 1: EFQM improves EFFICIENCY of action research (strategy)

Use TQM-model (EFQM) to reflect on governing variables Hypothesis 2: EFQM improves EFFECTIVENESS of action research (goals)

Use TQM-model (EFQM) to improve AR performance Hypothesis 3: EFQM improves LEARNING HOW TO DESIGN Single-loop and double-loop learning

Case: Trying to improve action research at NTAX by use of EFQM Diagnosis: NPM strategy of outsourcing IT services, managing through contracts, and controlling through ISO 9001 audits. No audit process. Treatment plan: Design audit process through use of action research approach. NTAX wants AR to be externally funded. Execution and evaluation: Eleven months of trying to get funded, but no success so far. Learning: NTAX research has to focus on AR method as no action is allowed. Operational hypothesis: Applying EFQM self-assessments will result in improved efficiency, effectiveness and learning on how to do Action Research.

Using the EFQM model to diagnose the action research project Diagnosis AR: AR strategy of proposing to design audit process through action research fails because NTAX will not allow AR without external funding.

Using the EFQM model to plan treatments for action research project Treatment AR: EFQM model identifies improving customer satisfaction (NTAX) and improving partnership management (UiO) as the two most important issues.

Using the EFQM model to plan treatments for action research project Treatment AR: EFQM model identifies improving customer satisfaction (NTAX) and improving partnership management (UiO) as the two most important issues. Sub-criteria 4a, 4b, 4c, 4d, 4e

Using the EFQM model to plan treatments for action research project Treatment AR: EFQM model identifies improving customer satisfaction (NTAX) and improving partnership management (UiO) as the two most important issues. Sub-criteria 1c, 2a, 2c, 3b, 3c, 4a, 5b, 5c, 5d, 5e

Using the EFQM model to evaluate the action research project Evaluation AR: Partnership with UiO improved by collaborating on grant application, but no measurable effect on customer (NTAX) because of poorly calibrated measurement methods.

Using the EFQM model to reflect on the action research project Validity and reliability: When comparing with EFQM results with earlier experiments, the observed change from 2013 to 2014 is not significant. Learning AR: Use feedback from rejection for writing new grant application.

Analysis: Did the use of the EFQM model improve AR performance? Analysis of AR project Use of EFQM model Diagnosis AR performance Plan for improving AR performance Execute plan and record what happens Evaluate new AR performance level Reflect on causes for change in AR performance EFQM self-assessment Use EFQM theory about causes and effects New EFQM assessment of enablers New EFQM assessment of results Observed change was not statistically significant

Analysis: Did the use of the EFQM model improve AR performance? Analysis of AR project Use of EFQM model Diagnosis AR performance Plan for improving AR performance Execute plan and record what happens Evaluate new AR performance level Reflect on causes for change in AR performance EFQM self-assessment Use EFQM theory about causes and effects New EFQM assessment of enablers New EFQM assessment of results Observed change was not statistically significant

Analysis: Did the use of the EFQM model improve AR performance? Analysis of AR project Use of EFQM model Diagnosis AR performance Plan for improving AR performance Execute plan and record what happens Evaluate new AR performance level Reflect on causes for change in AR performance EFQM self-assessment Use EFQM theory about causes and effects New EFQM assessment of enablers New EFQM assessment of results Observed change was not statistically significant

Analysis: Did the use of the EFQM model improve AR performance? Analysis of AR project Use of EFQM model Diagnosis AR performance Plan for improving AR performance Execute plan and record what happens Evaluate new AR performance level Reflect on causes for change in AR performance EFQM self-assessment Use EFQM theory about causes and effects New EFQM assessment of enablers New EFQM assessment of results Observed change was not statistically significant

Analysis: Did the use of the EFQM model improve AR performance? Analysis of AR project Use of EFQM model Diagnosis AR performance Plan for improving AR performance Execute plan and record what happens Evaluate new AR performance level Reflect on causes for change in AR performance EFQM self-assessment Use EFQM theory about causes and effects New EFQM assessment of enablers New EFQM assessment of results Observed change was not statistically significant

Analysis: Did the use of the EFQM model improve AR performance? Analysis of AR project Use of EFQM model Diagnosis AR performance Plan for improving AR performance Execute plan and record what happens Evaluate new AR performance level Reflect on causes for change in AR performance EFQM self-assessment Use EFQM theory about causes and effects New EFQM assessment of enablers New EFQM assessment of results Observed change was not statistically significant

Analysis: Did the use of the EFQM model improve AR performance? Analysis of AR project Use of EFQM model Diagnosis AR performance Plan for improving AR performance Execute plan and record what happens Evaluate new AR performance level Reflect on causes for change in AR performance EFQM self-assessment Use EFQM theory about causes and effects New EFQM assessment of enablers New EFQM assessment of results Observed change was not statistically significant H1, H2 & H3: Confirmed by qualitative data H1 & H2: Not confirmed by quantitative data

Discussion: Related research - interpretive AR approach Improving AR through ethics and moral standards (Boog et al, 2007)

Discussion: Related research - interpretive AR approach Improving AR through ethics and moral standards (Boog et al, 2007) EFQM is concerned with DIAGNOSING

Discussion: Related research - interpretive AR approach Improving AR through ethics and moral standards (Boog et al, 2007) EFQM is concerned with DIAGNOSING RICH DESCRIPTIONS are of limited use if they do not result in better treatments

Discussion: Related research - interpretive AR approach Improving AR through ethics and moral standards (Boog et al, 2007) EFQM is concerned with DIAGNOSING RICH DESCRIPTIONS are of limited use if they do not result in better treatments EFQM suggests that TREATMENTS should be evaluated on objective terms

Discussion: Related research - interpretive AR approach Improving AR through ethics and moral standards (Boog et al, 2007) EFQM is concerned with DIAGNOSING, but not as a goal in itself RICH DESCRIPTIONS are of limited use if they do not result in better treatments EFQM suggests that TREATMENTS should be evaluated on objective terms

Discussion: Related research - positivist AR approach Improving AR by making it more positivist (Kock et al, 1997)

Discussion: Related research - positivist AR approach Improving AR by making it more positivist (Kock et al, 1997) EFQM is concerned with testing TREATMENTS

Discussion: Related research - positivist AR approach Improving AR by making it more positivist (Kock et al, 1997) EFQM is concerned with testing TREATMENTS The truth about GENERALISED TREATMENTS are of limited use if they cannot be used in the local context

Discussion: Related research - positivist AR approach Improving AR by making it more positivist (Kock et al, 1997) EFQM is concerned with testing TREATMENTS The truth about GENERALISED TREATMENTS are of limited use if they cannot be used in the local context An interpretive approach is needed for DIAGNOSING and REFLECTING on the outcomes of the TREATMENTS within the local context

Discussion: Related research - positivist AR approach Improving AR by making it more positivist (Kock et al, 1997) EFQM is concerned with testing TREATMENTS, but not as a goal in itself The truth about GENERALISED TREATMENTS are of limited use if they cannot be used in the local context An interpretive approach is needed for DIAGNOSING and REFLECTING on the outcomes of the TREATMENTS within the local context

Discussion: Related research - critical AR approach Herbert Simon was part of the progressive “research and reform” movement in Chicago in the 1930s

Discussion: Related research - critical AR approach Improving AR by aligning it with Herbert Simon’s design science (Järvinen, 2007)

Discussion: Related research - critical AR approach Improving AR by aligning it with Herbert Simon’s design science (Järvinen, 2007) Game theory as foundation for critical theory (Elster, 1982)

Discussion: Related research - critical AR approach Improving AR by aligning it with Herbert Simon’s design science (Järvinen, 2007) Game theory as foundation for critical theory (Elster, 1982) Game theory as a foundation for social theory in general (Gintis, 2009)

Discussion: Related research - critical AR approach √ Improving AR by aligning it with Herbert Simon’s design science (Järvinen, 2007) Game theory as foundation for critical theory (Elster, 1982) Game theory as a foundation for social theory in general (Gintis, 2009)

Conclusion: Contributions to theory H1: EFQM improves AR efficiency The action strategy is improved through SPC and related TQM methods

Conclusion: Contributions to theory H1: EFQM improves AR efficiency The action strategy is improved through SPC and related TQM methods H2: EFQM improves AR effectiveness The governing variables are updated due to causal relationships in EFQM model

Conclusion: Contributions to theory H1: EFQM improves AR efficiency The action strategy is improved through SPC and related TQM methods H2: EFQM improves AR effectiveness The governing variables are updated due to causal relationships in EFQM model H3: EFQM improves AR learning By conceptualising the social situation as a game it can be analysed in a rational manner (“empirical economics”)

Conclusion: Contributions to practice Political consciousness NPM logic is compatible with practitioner logic when viewing the conflict between the two groups through the use of game theory

Conclusion: Contributions to practice Political consciousness NPM logic is compatible with practitioner logic when viewing the conflict between the two groups through the use of game theory Conflict resolution Action researchers should use NPM logic for assessing and improving their own strategic performance within the game

Summary of presentation Contributions to theory and implications for practice Critical theory allows EFQM to align with action research Action researchers should use NPM logic for improving AR performance Why was critical theory important in this context? The NTAX action research process needs to be improved EFQM is successfully used for improving the NTAX action research process Discussion: Comparison of EFQM approach for improving AR with literature on AR improvement based interpretivism, post-positivism and critical theory Why did we believe the EFQM model would work? Hypothesis 1: EFQM improves EFFICIENCY of action research Hypothesis 2: EFQM improves EFFECTIVENESS of action research Hypothesis 3: EFQM improves LEARNING about improving action research Why was this interesting? Problem: Action research is important but risky design for studying NPM Possible solution: Use TQM for controlling and improving action research