Quality in qualitative research ESRC research methods festival July 2008 Jane Lewis, NCB

Coverage of talk  why is quality so difficult in qualitative research?  NatCen’s work to develop a framework  some subsequent developments

Why is quality so difficult?  diversity of theoretical approaches  diversity of methods  importance of non-standardised, unstructured, flexible approaches  concerns about ‘methodolatry’  diversity of positions on key concepts

Range of positions on quality criteria Accept ‘traditional’ criteria Alternative criteria Reject criteria Parallel but modified criteria Guiding principles and ideas

Our work to develop a framework  literature review and review of existing frameworks  consultation  developed framework  workshop  refined, tested, refined

Structure and applicability  scoped applicability esp in terms of –theoretical positions –methods  4 guiding principles  18 appraisal questions  for each, a series of quality indicators

Some developments since: Dixon-Woods et al (2007)  used CO and CASP frameworks + unprompted judgement to determine inclusion in systematic review  frameworks produced less agreement  frameworks sensitised assessors to methodological issues  tipped balance to sound but less insightful  reproducability not validity

Some developments since: Walsh and Downe (2006)  reviewed 8 frameworks  synthesised and removed ‘redundant’ criteria  12 essential criteria, 53 specific prompts

Some developments since: Daly et al (2007) hierarchy of evidence Conceptual studies Descriptive studies Single case study Generalizable studies

Conclusions  a fair amount of common ground  reducing complexity is difficult  different needs of specialists vs non- specialists  different requirements for assessing quality vs eligibility  theoretical contribution needs to be based on methodological rigour

References Spencer L, Ritchie J, Lewis J and Dillon L (2003) Quality in Qualitative Evaluation: A framework for assessing research evidence London: Cabinet Office Dixons-Woods M, Sutton A, Shaw R, Miller T, Smith J, Young B, Bonas S, Booth A and Jones D (2007) ‘Appraising qualitative research for inclusion in systematic reviews: a quantitative and qualitative comparison of three methods’ in J Health Serv Res Policy vol 12 no 1 Jan 2007 Walsh D and Downe S (2006) ‘Appraising the quality of qualitative research’ in Midwifery vol 22 Daly J, Willis K, Small R, Green J, Welch N, Kealy M and Hughes E (2007) ‘A hierarch of evidence for assessing qualitative health research’ in Journal of Clinical Epidemiology vol 60

Jane Lewis :