Workshop: Thematic Synthesis and Framework Synthesis Parts 1-4 – Data Extraction, Quality Assessment, Synthesising Across Studies, Completing the Analysis

Shared Topic: Adherence to Antiretroviral therapy (ART) for HIV in Zambia BACKGROUND: Antiretroviral therapy (ART) has significantly improved morbidity and mortality of individuals infected with HIV. However, lack of adherence to highly active antiretroviral therapy (HAART) remains a key challenge to successful management of patients with HIV/AIDS. Adherence rates lower than 95% are associated with development of viral resistance to antiretroviral medications. ‘Efforts to sustain adherence in Africa and elsewhere remain important goals to optimize outcomes for individuals and global HIV treatment.’ (Mills, Nachega, Buchan, Orbinski, Attaran, Singh et al., 2006).

Shared Topic: Adherence to Antiretroviral therapy (ART) for HIV in Zambia Different Emphases – Barriers and Facilitators to ART (Framework Synthesis) – Theory Explaining Adherence to ART (Thematic Synthesis)

Reading and Data Extraction

Data extraction  What is it? – “ The process by which researchers obtain the necessary information about study characteristics and findings from the included studies” (CRD Report 4) – “An attempt to reduce a complex, messy, context- laden and quantification resistant reality to a matrix of categories and numbers” (Orwin, 1994)

Data extraction form  Location  Setting  Sample (n)  Age  Gender  Ethnicity  Socio-economic status  Intervention (if any)  Quality assessment criteria  Results?  Further citations  See Handouts 1-3  Format?????

What results do you extract?  What is your question? Keep the question in mind as you read: Are the data relevant to this question? Is the question answered by the data?

Data Extraction Framework SynthesisThematic Synthesis Extracts data against framework. Coding framework with definitions provided to increase consistency. Data not explained by framework is “parked” for subsequent inductive stage. Distinction typically made between original data extracts and author’s analysis. Key themes and concepts extracted and reviewed for inclusiveness. Distinction preserved between original (participant) extracts and (author’s analysis) findings. Findings coded in duplicate. Discrepancy between codes resolved by third person.

Some Further Reading Munn, Z., Tufanaru, C., & Aromataris, E. (2014). JBI's systematic reviews: Data extraction and synthesis. AJN The American Journal of Nursing, 114(7), Orwin RG. Evaluating coding decisions. In: Cooper H, Hedges LV (editors). The Handbook of Research Synthesis. New York (NY): Russell Sage Foundation, 1994.

Quality Assessment Andrew Booth, Reader in Evidence Based Information Practice, Co-Convenor – Cochrane Collaboration Qualitative Methods Group

Before You Begin… Consider how you will use judgements of quality (cp. 50% of published Cochrane Quantitative Reviews performed quality appraisal but did not make it clear how judgements were used!) – To exclude or to moderate? Will chosen instrument militate against certain types of research? Quality of reporting or quality of study?

Variability in Practice papers did not describe appraisal of candidate studies 6 explicitly mentioned not conducting formal appraisal of studies 5 papers did a critical appraisal, but did not use a formal checklist 7 described modifying existing instruments 1 used an existing instrument without modification Dixon-Woods, Booth & Sutton (2007)

Variability in Current Practice papers did not describe critical appraisal 5 papers explicitly pleaded against quality assessment of papers or provided valid reason for not conducting quality appraisal. Criteria used varied between detailed descriptions of relevant items in existing or modified checklists to a set of broad criteria evaluating, for example, rich description of data, credibility or relevance of the original study. Hannes and Macaitis (2012)

Variability in Current Practice - 2 One team used overall judgement (Smith et al., 2005). Five opted for self-developed assessment instrument Three used previously developed checklists to create own. Two mentioned critical appraisal, but did not specify tool. Most used existing instruments/frameworks. 24 different assessment tools identified: Critical Appraisal Skills Programme (CASP) (n = 18) Mays and Pope criteria (n = 6) Popay criteria (n = 6) Joanna Briggs Institute (n = 4). Hannes and Macaitis (2012)

Appraising research quality 1. Epistemological criteria: Judgement of ‘trustworthiness’ requires criteria tailored to different research ‘paradigms’. 2. Theoretical Criteria: Explicit theoretical framework shaping the design of the study and informing claims for generalisability 3. Prima facie ‘Technical’ criteria: Used to assess ‘quality’ common to all research traditions e.g.: Sufficient explanation of background; Method appropriate to question; Succinct statement of objectives/research questions; Full description of methods include approach to analysis; Clear presentation of findings including justification for interpretation of data etc. Noyes J (2005)

Two dimensional approach to appraising qualitative research Technical markers – CASP Epistemological and theoretical markers – Popay et al Technical Quality HighDescription – thicker Privileges Subjective experience and meanings Use of theory to build explanations Technical Quality LowDescription - thinner Imposed pre-determined framework on respondents narratives. Limited/no/inappropriate use of theory, little explanatory insight (Noyes, 2005)

Available Tools - 1 CASP – 10 questions to help you make sense of qualitative research content/uploads/2011/11/CASP_Qualitative_Appraisal_Check list_14oct10.pdfhttp:// content/uploads/2011/11/CASP_Qualitative_Appraisal_Check list_14oct10.pdf Joanna Briggs Institute - Critical Appraisal Checklist for Interpretive & Critical Research pprais.pdf pprais.pdf National Centre for Social Research. Quality in Qualitative Evaluation: A Framework for Assessing Research Evidence. London: National Centre for Social Research/UK Cabinet Office, content/uploads/2011/09/a_quality_framework_tcm pdfhttp:// content/uploads/2011/09/a_quality_framework_tcm pdf

Available Tools - 2 Dixon-Woods M, Shaw RL, Agarwal S & Smith JA (2004) The problem of appraising qualitative research. Quality & Safety in Health Care, 13, Hannes K, Lockwood C, Pearson A (2010). A comparative analysis of three online appraisal instruments' ability to assess validity in qualitative research. Qualitative Health Research. 20(12): Popay J, Rogers A & Williams G (1998) Rationale & standards for the systematic review of qualitative literature in health services research. Qualitative Health Research, 8, Seale C & Silverman D (1997) Ensuring rigour in qualitative research. European Journal of Public Health, 7,

CASP – Technical/Procedural Tool

JBI – Theoretical Tool

Key Issue How are you going to use the quality assessment? – From quantitative assessment we know authors frequently say they do it – but they don’t incorporate it into results – Is it technical proceduralism gone mad? – Or can we use the assessments to improve our synthesis and subsequent interpretation?

Quality Assessment Framework SynthesisThematic Synthesis (e.g. as first stage of Meta-Ethnography) Pragmatic so tends to include all studies. Focuses explicitly on quality of reporting. Qualitative sensitivity analysis used to test robustness of synthesis. Quality Assessment as Hurdle (often used when plenty of studies to draw upon): Studies using qualitative design and analysis method included. Studies assessed for relevance first to continue to full-text review. Studies passing quality appraisal (are retained.

Some Further Reading Carroll C, Booth A. Quality assessment of qualitative evidence for systematic review and synthesis: Is it meaningful, and if so, how should it be performed? Research Synthesis Methods 2015; 6(2): Carroll C, Booth A, Lloyd-Jones M. Should We Exclude Inadequately Reported Studies From Qualitative Systematic Reviews? An Evaluation of Sensitivity Analyses in Two Case Study Reviews, Qualitative Health Research, 2012; 22: Garside, R. (2014). Should we appraise the quality of qualitative research reports for systematic reviews, and if so, how?. Innovation: The European Journal of Social Science Research, 27(1),

Data Synthesis

What is Data Synthesis? Process of moving from focus on single studies (cp. Data Extraction and Quality Assessment) to focus on cross-study analysis Requires identification of patterns across data, including contradictory findings and data that does not fit Iterative and requires ongoing refinement Acts as prelude to Analysis which seeks to explain patterns, contradictions and differences

 Thematic synthesis; Critical Interpretive Synthesis; Meta-ethnography 1.Only include “good” qualitative studies (?) 2.Constant comparison; iterative; interpretations generated from the data by reviewers 3.Create a theory – Inductive (theory-generating)  Examples:  Thomas J, Harden A. Methods for the thematic synthesis of qualitative research in systematic reviews. BMC Medical Research Methodology 2008; 8.  Campbell R et al. Evaluating meta-ethnography: a synthesis of qualitative research on lay experiences of diabetes and diabetes care. Social Science & Medicine 2003; 65: Methods of qualitative evidence synthesis

 Framework synthesis: 1.Only include “good” qualitative studies (?) 2.Map data from included studies onto an existing framework to test the framework/theory (a role for theory) 3.Build a conceptual model or framework – Deductive (theory-testing)  Examples:  Oliver S et al: A multidimensional conceptual framework for analysing public involvement in health services research. Health Expectations 2008, 11:  Brunton G, Oliver S, Oliver K, Lorenc T. A Synthesis of Research Addressing Children’s, Young People’s and Parents’ Views of Walking and Cycling for Transport London. London, EPPI-Centre, Social Science Research Unit, Institute of Education, University of London; Methods of qualitative evidence synthesis

 “Best-fit” framework synthesis 1.Identify relevant pre-existing conceptual models or frameworks 2.Identify and extract all relevant qualitative studies satisfying review’s inclusion criteria 3.Code data from included studies against framework 4.Use secondary thematic analysis/synthesis to generate completely new themes to supplement the framework’s themes 5.Create new framework and conceptual model or theory  Deductive and Inductive  Framework and Thematic synthesis  Carroll C, Booth A, Cooper K. A worked example of “best-fit” framework synthesis: A systematic review of views concerning the taking of potential chemopreventive agents, BMC Medical Research Methodology 2011; 11: 29 Methods of qualitative evidence synthesis

Data Synthesis Framework SynthesisThematic synthesis (may be taken forward as Meta-Ethnography) Original best fit framework is expanded to include new themes. Relationship between themes is examined and the data is used to reconstitute a new model. Particular attention is directed at discrepant cases. Second-order constructs pertinent to adherence identified and cross- compared and presented in results section. Key themes consolidated into line of argument (third-order analysis), presented in the synthesis ⁄ discussion section.

Data Analysis To move from a list of themes/framework to a conceptual model/theory Examine relationships Synergies, Antagonisms, Pre-requisites, Causal Paths, Disconfirming Cases etcetera Testing the Robustness of the synthesis

A word about “constructs” Conventionally we distinguish (using the terminology of the most common method of qualitative synthesis, meta-ethnography): 1 st Order Constructs – First hand accounts of the people being studied 2 nd Order Constructs – Interpretations by the Authors of the studies 3 rd Order Constructs – “Interpretations of the interpretations” by the Systematic Review Team

Example 1 st Order 3 rd Order

The Contribution of Meta-ethnography “Using the meta-ethnographic approach, we were able to produce a model of adherence to TB treatment by re-interpreting meaning across many individual qualitative studies. We also derived plausible hypotheses that can be used by policy makers and programme managers to re-organise treatment and care systems to improve adherence. Adapting the method for use in synthesising qualitative health research raises a number of methodological challenges that require further exploration.” (atkins et al, 2008)

From Kate’s research

From Constructs to Arguments

Booth et al, 2011

Some Practicalities Tabulation of data – looking for and explaining differences (e.g. majority…, split…, exception…) Post-Its – arranging according to patterns or clusters Mapping e.g. Mind Map, Process Maps (e.g. Pathways of Care), Logic Models Integration (with quantitative) – congruence, contradictions, gaps with explanation

Further Reading Antin, T. M., Constantine, N. A., & Hunt, G. (2014). Conflicting Discourses in Qualitative Research The Search for Divergent Data within Cases. Field Methods, X Atkins S, Lewin S, Smith H, Engel M, Fretheim A, Volmink J. Conducting a meta-ethnography of qualitative literature: lessons learnt. BMC Med Res Methodol Apr 16;8:21. doi: / Booth, A., Carroll, C., Ilott, I., Low, L. L., & Cooper, K. (2013). Desperately Seeking Dissonance Identifying the Disconfirming Case in Qualitative Evidence Synthesis. Qualitative health research, 23(1), Carroll, C., Booth, A., & Lloyd-Jones, M. (2012). Should we exclude inadequately reported studies from qualitative systematic reviews? An evaluation of sensitivity analyses in two case study reviews. Qualitative Health Research, 22(10), Finfgeld-Connett, D. (2014). Metasynthesis Findings Potential Versus Reality.Qualitative health research,