1. Program Evaluation Unit, Univerity of Melbourne Evaluating organisational context Measurement approaches: evaluation David Dunt

2. Program Evaluation Unit, Univerity of Melbourne The contribution of realistic evaluation in studying the effects of organisational context on KU & KU’s take-up at grassroots level

3. Program Evaluation Unit, Univerity of Melbourne For me, Realistic evaluation, not a new set of ideas but a name for an approach that I had progressively discovered during my career as an evaluator (‘that’s what it is’) My outline not so much an exposition on a realist theory of science But a description of this path of discovery As it encapsualtes the path to enlightenment for many other health practitioners interested in KU particularly after their discovery of Evidence-based Medicine & healthcare

4. Program Evaluation Unit, Univerity of Melbourne Outline the main ideas of Realistic evaluation particularly the key concepts of CMOs C ontext M echanism O utcomes Outline applications to KU & organisational context

5. Program Evaluation Unit, Univerity of Melbourne Outcomes movement - point of departure The randomised control trial Treatment ProgramOutcome ‘Thing’

6. Program Evaluation Unit, Univerity of Melbourne Outcomes movement - first problem The randomised control trial Treatment ProgramOutcome ‘Thing’ Treatment, program content - a black box

7. Program Evaluation Unit, Univerity of Melbourne With a black box:- Don’t know what element of the program was responsible for success failure Was program failure due to model failure or implementation failure (of a good model)

8. Program Evaluation Unit, Univerity of Melbourne Distinction is fundamental to evidence-based decision making where effective and ineffective programs must be correctly identified

9. Program Evaluation Unit, Univerity of Melbourne Points to the importance of the study of process The importance of the use of Program logic

10. Program Evaluation Unit, Univerity of Melbourne Program logic Is a concept much loved by evaluators Similar to management flowsheets Specify successive stages (chronological, causal) in program rollout Use key performance indicators (KPIs) to assess whether a stage has been successfully reached Gather relevant information collected at each stage why stage has been reached (or not) But need to move beyond their common sense models of causation & chronology

11. Program Evaluation Unit, Univerity of Melbourne Program theory - Mechanisms Instead program logic based upon a program theory in order to identify optimal program content including all its components and their relationship (stages) Application of a Mechanism or causal model based on best information (literature, experts, stakeholders)

12. Program Evaluation Unit, Univerity of Melbourne Program theory - Context Program logic also based upon knowledge of local Context (analysis of barriers, risks eg negotiating key gatekeepers, usual work patterns, usual organisational practice and barriers

13. Program Evaluation Unit, Univerity of Melbourne In summary With program development There should be an analysis of the application of the proposed Mechanism in the local Context in order to ensure a program with maximum functionality in that context and with achievement of best Outcomes With program implementation There should be monitoring of the important stages of the program’s rollout to ensure they are occurring

14. Program Evaluation Unit, Univerity of Melbourne A second problem for the Outcomes movement embedded in this analysis Context is assumed and Context of EBM studies typically is quite atypical Teaching hospitals Atypical patients chosen for their ease of study (no co-morbidities, English-speaking, no mental impairment, and with unrepresentative social & demographic characteristics)

15. Program Evaluation Unit, Univerity of Melbourne Fundamental to evidence-based decision making where effective and ineffective programs must be correctly identified

16. Program Evaluation Unit, Univerity of Melbourne Realistic evaluation in KU Focus on Context and Mechanism occurring to some extent already EPOC group Examples: academic detailing, use of opinion leaders

17. Program Evaluation Unit, Univerity of Melbourne Realistic evaluation – in KU Still little program theory (psychological, sociological [systems], or economic) Confirmed in a recent UK review by Grimshaw et al Reliance instead on atheoretical, tickbox questionnaire-based study of behavior change Program logic approaches notably absent

18. Program Evaluation Unit, Univerity of Melbourne Realistic evaluation - measuring organisational context & change 1 Challenge is first to build program theory Application of Mechanisms using existing theory and evidence to Local organisational Contexts using best local knowledge

19. Program Evaluation Unit, Univerity of Melbourne Realistic evaluation - measuring organisational context & change 2 Generally, aim is to build heuristic models to assist in thinking about program development and implementation. Not to construct a causal model by eg path analysis, with each path’s contribution being measured by a correlation coefficient

20. Program Evaluation Unit, Univerity of Melbourne A few exceptions where may be important to build causal models For very important and central dimensions of the health care system

21. Program Evaluation Unit, Univerity of Melbourne A few exceptions where may be important to build causal models Finish briefly with an example of this from my own work in the Australian health care system The development of a psychometric tool to measure the level of integration of the GP with the remainder of the health care system

22. Program Evaluation Unit, Univerity of Melbourne GP integration with the health care system 1 70 question inventory completed by GPs about their current activities relevant to integration 9 integration factors organised around 2 higher order factors patient care management and community health Plus 5 enabling factors

23. Program Evaluation Unit, Univerity of Melbourne GP integration with the health care system 2 Two key findings It is possible to map levels of GP integration across Australia – considerable variation exists Levels strongly relate to GP’s knowledge of local services and resources

24. Program Evaluation Unit, Univerity of Melbourne GP integration with the health care system 3 Further details are a story for another day