Dr Ana School of Sociology & Social Policy University of Leeds- United Kingdom European Evaluation Society 2014-Dublin

 Background  The study  The craft of interviewing

 In the last two decades, commissioners around the world have been funding realist evaluation studies and this (Pawson & Tilley, 1997) is now a well-established approach to programme evaluation.  Realist evaluation is a mixed-method strategy which proposes- among other methods of data collection- the use of theory-driven interviews to 'inspire/validate/falsify/ modify” hypothesis about how programmes and interventions work.  “Theorizing the interview” (Pawson, 1996): the teacher-learner cycle.

Pawson, R. and Tilley, N. (1997) Realistic Evaluation. London: Sage Pawson, R. (2006) Evidence-based Policy. A Realist Perspective. London: Sage Pawson, R (2013) The Science of Evaluation. A Realist Manifesto. London: Sage

 Programmes are ‘theories’ and so evaluation becomes a process of ‘theory testing’. But to do so the theories need to be articulated in a particular manner  Ingredients are Contexts, Mechanisms and Outcomes  Realist Evaluation begins with a period of ‘theory elicitation’ using ◦ Programme documentation ◦ Interviews with key stakeholders ◦ Existing ideas about the family of programmes ◦ Existing social / psychological theory  Data construction should be theory-driven (and not data- driven)  The subject matter of the interview is the researcher’s theory, The interview will confirm, falsify, modify, refine the researcher’s theory.  What theory???? A realist theory of the Ms, Cs and Os which define how programmes work.

 Fake handbags- Spin –off the Realist Evaluation Workshop paper (Pawson & Manzano-Santaella, 2012)  The need for standards?  The purpose of advancing realist methods  The need to do evaluation on evaluations  OBJECTIVE: To explore the empirical application of the realist interview

 1) What are the research methods used in RE studies? (2) If interviews were used, how do these studies apply the teacher-learner cycle? and (3) What methodological problems - related to interviewing- do these papers raise?  Literature review of realist evaluations in applied health  Studies published between years  A total of 40 papers were found and of those, 32 included interviews as one of the methods of data collection

40 Realist evaluations 32 used interviews 5 only used interviews 27 used interviews in combination with other methods 8 did not use interviews Most common model: documentary analysis (it could include quant data), ethnographic observation 8 combined with explicit quantitative methods

 Interview is the most common method used in RE  It is normally used in combination with other methods  It is not clear how quantitative data is used in the documentary analysis (frequent method)  Sometimes RE is used as a “more prestigious” label to do qualitative research

 Etheridge et al. 2013: “we used an interview guide derived from the main concepts in our theoretical framework”  Rycroft-Malone et al. 2013: “interviews were conducted using a topic guide based on the study programme theory and the contents of the evaluation framework”

 We found that, generally, little attention has been given to fieldwork processes with authors highlighting lack of methodological guidance at the analysis stage (Cs, Ms,...)  Tendency to reflect/offer veredict on the realist approach  The process of data collection is not recognised as analysis...Still treated as lineal

 Typical queries addressed to the team: ◦ How do you do you interview like a realist? ◦ How do you do realist topic guides? ◦ How do you present topic guides to ethics committees? ◦ How do I do iterative analysis when I am only doing one lot of interviews? ◦ I have prepared a table with CMOs, should I show it to the participants?? ◦ Can you use NVIVO? ◦ Can I see your topic guides?

“I’ll show you my theory if you show me yours” (Pawson & Tilley, 1997: 169)

 Theories are placed before the interviewee for them to comment on with a view to providing refinement  The subject matter of the interview is the researcher’s theory/ The subject (stakeholder) is there to confirm, falsify and basically, refine the theory.  Teacher-learner relationship in which the medium exchange is the CMO configuration  2 way flow of information  The realist approach assumes that interviewees accounts have a direct relationship with the real experiences in the world (beyond the interview situation)...  The research should have “Hypothesis seeking behaviour” (Pawson, 1996) during the interview  In practice: ◦ Semi-structured interviews ◦ Theory-driven interviews

Subjects More likely to be sensitized about Mechanisms (M) Than to its contextual constraints (C) and outcome patterns (O) Practitioners Will have specific ideas on what is within the programme that works (M) Likely to have experienced successes and failures (O) Some awareness of people and places (C) for whom and in which the programme works Evaluators They carry theories into the encounter with programmes Embryonic or well developed theories Knowledge from similar and previous programmes Knowledge from social science theory Wisdom stronger on form than content

 Preparing for the interview: ◦ Go to the interview with some specific theories to test. Your evidence based (observations, documents, other evaluations) guesses of “contexts”, “mechanisms”, and “outcomes” ◦ Not necessarily all of them  Phrasing questions:  The programme says that... But I can see that...  I have read some studies that say...  When they implemented this programme in York, they had this problem with...Have you seen that?  How did you do it before the programme was implemented?

 Lots of work needs to be done before the interview  You have to start thinking CMOs at the first literature review  Then while you are collecting data via the other methods (observations, analysis of secondary quantitative data, etc...)  Then at the interview... And in the next interviews  Then at the analysis (Common mistake is to start looking for CMOS here!)... Then maybe interview again...  It is hard work!!!!!!!!!!!! Realist interviewing “involves a highly specific and carefully planned route march which goes between the qualitative and quantitative traditions “ (Pawson, 1996)

 Participants will not tell you: “Yes, well done! That is a mechanism! You got it!”.  The interview is only a step in the search for “nuggets of evidence”...  One of the key mechanisms in one of my projects was built up of one sentence said to me in a Staff Christmas party... This comment was not coded with NVIVO!

 Researcher: In this case I am interested on the fact that she (the patient) was self-funder and if it affected the way the discharge went and everything... The (“fine”) policy says that with self-funders, social services are only responsible for the assessment process. But what I have found out is really… the hospital doesn’t sort of remember that, because if the patient is self- funding, it is sort of…just…  CM: Umm…Yes, they can just get on with it, go and do it.  Researcher: In the team meeting (that I observed), the physio was saying “maybe in three or four months the patient will be able to recover”. And then the physio asked you ‘Is she [the patient] very self-funder?’ And you said, ‘Oh, yes, the son says she was quite over the limit’. So why do you think the physio was asking that? Why will it make a difference if the patient wasn’t self-funder?  CM: Well, they shouldn’t, should it? But they often want them out, don’t they? Out of the way…  Researcher: I still don’t understand what she meant! Do you want to translate it?  CM: I think what she [the physio] is getting at is, if she wasn’t self-funding, they’ll be saying well… maybe a rehabilitation bed, either move her up to a ward or…  Researcher: Ah, OK. Yes… But if she is a self-funder…  CM: If they go self-funding, if they get her going there [in the nursing home] brilliant. If not, well, she can stay there. Which isn’t really fair to her! Because maybe she is not getting [rehabilitation]  Researcher: In a different way, I’ve seen with some of other self-funders that I’ve done, not with you, with the other care managers, that the promotion of independence is a bit less than with social services patients. Because they [hospital staff] tend to say ‘The family is going to find a place and that place is ready. So they can go’. So in a way they, as you said, they know that they can get them out quicker.  CM: Yes, they can. Grey literature Observation Cross-case comparison Interview with frontline practitioner