1 What is “missing data” in qualitative research? 4 th ESRC Research Methods Festival St Catherine’s College, Oxford, 7 July 2010 Graham Crow, ESRC National Centre for Research Methods Alison Powell, Oxford Internet Institute

2 Outline of presentation Background to the project A selection of types of “missing data” in qualitative research “Missing” for whom, how, why, and what is at stake? Addressing “missing data” in qualitative research

3 Background to the project Origins in discussions in the National Centre for Research Methods on missing data in quantitative research (e.g. non-response in surveys) Is there an equivalent issue in qualitative research? Collaborative project between NCRM ‘hub’ (Graham Crow, Rose Wiles), WISERD (Amanda Coffey), Oxford ‘node’ of NCeSS (Bill Dutton, Alison Powell), and Qualidata/Timescapes (Libby Bishop), based on recognition of shared sense of a problem Literature review, interviews and focus groups with researchers

4 Background to the project Geoff Payne once asked why community studies are so full of ‘nice’ people: ‘the main impression generated is one of a world populated with pleasant, likeable people’ (1996: 21) ‘In the course of fieldwork in several locations in the past half a dozen years, I have encountered people whom I did not like, and situations that felt most unpleasant…. The people in community studies are too “nice”’ (1996: 22) Problems of selective sampling and of ‘selective reporting’ (1996: 23) – less likeable people and unhappier experiences are screened out

5 Some types of “missing data” in qualitative research “Off the Record” remarks - material made known to researchers but which research participants do not consent to being made public Ethically sensitive material – requires changes to achieve anonymisation Archived material for which context is difficult to gain Mediation of research findings – through translation, technology, or teamwork

6 “Off the Record” Comments Reveal power negotiations between participants and researchers Contribute to tacit understanding of research sites and contexts Reporting these comments may mean losing access to a field site Off the record comments may not be as significant as participants think they are!

7 Ethically sensitive material Ethics are both contextual and normative: we must abide by the standards of participants but anticipate changing contexts What will happen to this data in future? Consider the media, changing lives of participants Anonymization has costs and benefits

8 Archived material Research practices have changed over time – what is in archives has different, or missing contexts Digital archives of previously rare material change the nature of research: shift towards analysis from data gathering How can the research process be represented in archiving practices?

9 Mediated Research Contexts: online Internet research provides access to hidden or marginalized groups Yet online research can obscure contexts of interaction – internet texts are not the whole story Structural features of the internet such as search algorithms configure how information is gathered

10 Mediated Research Contexts: transcription and translation Transcription is rarely examined as construction of data (Vigoroux); translation is, by specialists Both are common practice, both require high levels of trust Both mediate data in some way; both are knowledge construction as well as knowledge archiving in new forms

11 ‘Missing’ - for whom? For the researcher: –Technical faults or gaps in transcription mean data is not collected – but knowledge remains –Researchers ‘edit’ or ‘censor’ data for ethical reasons –Members of research teams pay attention to different information (Morse and Richards 2002) For the audiences: –The findings must be credible and believable: this may mean targeting research findings

12 ‘Missing’ - How? Ethically –There is an important distinction between unintended ‘gaps’ and intended ‘silences Philosophically –Difference between “missing the point” and “missing” data in terms of research quality –No research account has perfect objectivity; but good researchers negotiate the scale of their truths Functionally –What are qualitative questions good for? More than “good quotes” for mixed methods studies

13 Why? What's at Stake? Researchers have power –Anonymity must be considered in terms of future audiences and impacts of research –Choices inevitably made between who is in/who is out Participants have power –There is a distinction between unintended ‘gaps’ and intended ‘silences’ –Participants may use research for their own ends, negotiating on and off record comments Audiences have power –Media and policy audiences have different agendas

14 Conclusions about “missing data” in qualitative research To a conventional quantitative researcher, ‘The only good solution to the missing data problem is not to have any’ (Paul Allison 2002: 2-3) Qualitative researchers are not concerned with perfect knowledge – but need to reflect on choices made in design, collection and analysis, archiving and re-use Thinking about what is missing can be a heuristic for verifying research quality

15 References Allison, P. (2002) Missing Data. London: Sage. Hammersley, M. (2010) ‘Can we re-use qualitative data via secondary analysis? Notes on some terminological and substantive issues’, Sociological Research Online 15(1) Morse, J. and Richards, L. (2002) Read Me First for a User’s Guide to Qualitative Methods. London: Sage. Payne, G. (1996) ‘Imagining the community: some reflections on the community study as a method’, in E Stina Lyon and J Busfield (eds) Methodological Imaginations. Basingstoke: Macmillan, pp Richards, M. (2005) ‘Loitering with intent in a Special Care Baby Unit’ in N Hallowell, J Lawton and S Gregory (eds) Reflections on Research: The Realities of Doing Research in the Social Sciences. Maidenhead: Open University Press, pp.81-3.