2. Critical Realist Evaluation
Sam Porter
School of Nursing and Midwifery
March 2016

3. What RCTs Can Tell Us
RCT is a powerful tool for providing valid and reliable evaluation of the efficacy of healthcare interventions.
Depends on use of probability to approximate closed system of experiment to isolate effect of putative causal agent.

4. What RCTs Cannot Tell Us
Interventions in everyday practice occur in open systems, with many factors additional to the intervention influencing its effectiveness.
Those factors include the institutional, social and cultural context into which the interventions are introduced.
It also includes the interpretations and actions of individuals involved.

5. Enter Realism
Replacement of constant conjunction with causal mechanisms:
‘In citing a law one is referring to the … activity of a mechanism … not making a claim about the actual outcome (which in general will be co-determined by the activity of other mechanisms)’
(Bhaskar 1989:9-10)

6. Critical Realist Evaluation Formula
Contextual Mechanisms + Programme Mechanisms + Agency = Outcome
(CM+PM+A=O)

7. Defining Terms
‘Contextual mechanisms’ relates to pertinent mechanisms in operation prior to the intervention.
‘Programme mechanisms’ relates to the mechanisms contained in the programme designed to the counter contextual mechanisms that are deemed as problematic or illness-generating.
‘Agency’ relates to people’s interpretations and behaviour, including their responses to social mechanisms.

8. Defining Terms
Contextual and programme mechanisms can be either natural or social.
The term ‘natural mechanism’ relates to the physical (including biological) causes of events.
The term ‘social mechanism’ relates to the enablements and constraints, inducements and discouragements faced by people as a result of their social position.

9. The Evaluative Aims of CM+PM+A=O
Identify the generative mechanisms contained in a particular context, and analyse how they act in relation to a defined health outcome.
Identify the generative mechanisms contained in an intervention, and analyse how they act in relation to a defined health outcome.
Analyse the interactions of generative mechanisms contained in the context and the programme.
Analyse how people interpret and respond to mechanisms.
Identify the outcomes that result from the interaction of mechanisms and agency.

10. RCTs + CRE
While accepting the usefulness of RCTs, this approach regards them as an initial phase used to identify the causal powers of an intervention within the artificial confines of a ‘closed system’.
RE then used to identify outcomes that result from the co-determination of all causal mechanisms occurring in the open system of a given healthcare context.
CRE may also use pragmatic trials or other, quasi-experimental methodologies to establish outcomes in more open systems.

11. Example Early warning systems to prevent adverse events.
RCTs have demonstrated the efficacy of early intervention to prevent adverse events.
They have not demonstrated effectiveness, but cannot explain why.

12. What about the ‘Critical’ in CRE?
One of the main distinguishing features between Realist Evaluation and Critical Realist Evaluation is their approach to outcomes.
The exclusive aim of RE is to enumerate and explain changes of rates of the targeted outcome.
CRE also seeks to examine the consequences of interventions for the lives of those affected by them.
Put another way, RE aspires to stick to the facts, while CRE wishes to introduce values as criteria for evaluating interventions.

13. Annotated Bibliography
Bhaskar, R. (1989) The Possibility of Naturalism: A Philosophical Critique of the Contemporary Human Sciences, 2nd Edition. Hemel Hempstead: Harvester Wheatsheaf. Fundamental text for the application of critical realism to the social sciences.
Pawson, R. and Tilley, S. (1997) Realistic Evaluation. London: Sage. Seminal realist evaluation text. For a shorter version ….
Pawson, R. And Tilley, N. (2004) Realist Evalution. Downloadable from communitymatters.com.au

14. Annotated Bibliography
Adams, A., Sedalia, S., McNab, S. and Sarker, M. (2015) Lessons learned in using realist evaluation to assess maternal and newborn health programming in rural Bangladesh. Health Policy and Planning. doi: /heapol/czv053. A recent example of the use of RE in a similar area of enquiry.

15. Annotated Bibliography
Blackwood, B., O’Halloran, P. and Porter, S. (2010) On the problems of mixing RCTs with qualitative research: the case of the MRC framework for the evaluation of complex healthcare interventions. Journal of Research in Nursing 15(6): 511–521. Points to limitations of RCTs and argues for their combination with qualitative research. Examines both MRC framework and critical realism as frameworks forthat combination.

16. Annotated Bibliography
Porter, S. and O’Halloran, P. (2012) The use and limitation of realistic evaluation as a tool for evidence-based practice: a critical realist perspective. Nursing Inquiry 19(1): Expands argument of the useful relationship between realist evaluation and RCTs but also introduces critical realist critique of RE.
Porter, S. (2015) Realist evluation: an immanent critique. Nursing Philosophy 16(4): Expands critique of RE beyond issue of critical engagement and introduces CM+PM+A=O formula.