1. Data Extraction Workshop Lindsay Shepard 30 March 2011 PhD 7113 Systematic Reviews College of Social Work, University of Utah

2. Overview O The Purpose O The Process O The Contents O Data Extraction Forms O Example Data Extraction Forms O Example Data Extraction O Class Workshop

3. The Purpose of Data Extraction O “to describe the study in general, to extract the findings from each study in a consistent manner to enable later synthesis, and to extract information to enable quality appraisal so that the findings can be interpreted. Ideally this should be undertaken in such a way as to require minimal reference to the original papers at data synthesis stage.” (Social Care Institute for Excellence, 2006)

4. The Rationale for Data Extraction Processes O “The findings of a systematic review depend critically on decisions relating to which studies are included, and on decisions relating to which data from these studies are presented and analysed. Methods used for these decisions must be transparent, and they should be chosen to minimize biases and human error” (Cochrane Handbook, Section 7.1, emphasis added)

5. The Process of Data Extraction O Plan data collection in advance O Data – “any information about (or deriving from) a study, including details of methods, participants, setting, context, interventions, outcomes, results, publications and investigators” (Cochrane Handbook, Section 7.3.1) O Develop an extraction strategy O Replicable O Transparent O Use resources O Checklists and Review Group guidelines O Topical systematic reviews and primary studies O Data extraction forms O Pilot test method O Duplicate data extraction

6. Source Study ID (created by review author). Report ID (created by review author). Review author ID (created by review author). Citation and contact details. Eligibility Confirm eligibility for review. Reason for exclusion. Methods Study design. Total study duration. Sequence generation*. Allocation sequence concealment*. Blinding*. Other concerns about bias*. Participants Total number. Setting. Diagnostic criteria. Age. Sex. Country. [Co-morbidity]. [Socio-demographics]. [Ethnicity]. [Date of study]. Interventions Total number of intervention groups. For each intervention and comparison group of interest: Specific intervention. Intervention details (sufficient for replication, if feasible). [Integrity of intervention]. Outcomes Outcomes and time points (i) collected; (ii) reported*. For each outcome of interest: Outcome definition (with diagnostic criteria if relevant). Unit of measurement (if relevant). For scales: upper and lower limits, and whether high or low score is good. Results Number of participants allocated to each intervention group. For each outcome of interest: Sample size. Missing participants*. Summary data for each intervention group (e.g. 2×2 table for dichotomous data; means and SDs for continuous data).  Estimate of effect with confidence interval; P value].  Subgroup analyses]. Miscellaneous Funding source. Key conclusions of the study authors. Miscellaneous comments from the study authors. References to other relevant studies. Correspondence required. Miscellaneous comments by the review authors. *Full description required for standard items in the ‘Risk of bias’ tool (see Chapter 8, Section 8.5).8.5 Data Collection Checklist, Cochrane Handbook, Table 7.3.a Items without parentheses should normally be collected in all reviews Items in square brackets may be relevant to some reviews and not others

7. Method Collection O “Different research methods can influence study outcomes by introducing different biases into results. Basic study design characteristics should be collected” (Cochrane Handbook, Section 7.3.2) O Randomized? O Cluster or cross-over design? O Duration of the study? O Non-randomized studies included? (see Cochrane Handbook, Section 13.4) O “Information should also be collected to facilitate assessments of the risk of bias in each included study using the tool described in Chapter 8 (Section 8.5)” (Cochrane Handbook, Section 7.3.2)8.5 O Sequence generation? O Allocation sequence concealment? O Blinding? O Incomplete outcome data? O Selective outcome reporting?, etc.

8. Participant and Setting Collection O “Typically, aspects that should be collected are those that could (or are believed to) affect presence or magnitude of an intervention effect and those that could help users assess applicability” (Cochrane Handbook, Section 7.3.3) O Participant characteristics? O Total number? O Sub-group analyses? (Age, Sex, Ethnicity, Country, Co-morbidity, etc.) O Study setting? O Diagnostic criteria?

9. Intervention Collection O “Again, details are required for aspects that could affect presence or magnitude of effect, or that could help users assess applicability. Where feasible, information should be sought (and presented in the review) that is sufficient for replication of the interventions under study” (Cochrane Handbook, Section 7.3.4) O Number of intervention groups? O Routes of delivery? (Oral or intravenous, Surgical technique, etc.) O Dosage? (Amount, Intensity, Frequency, etc.) O Timing? (Between diagnosis and treatment, etc.) O Length of treatment? O Contents of the intervention? O Who delivered or administered? O Format and timing of delivery? O Intervention integrity/fidelity/compliance?

10. Outcome Measure Collection O “Review authors should decide in advance whether they will collect information about all outcomes measured in a study, or about only those outcomes of (pre-specified) interest in the review. Because we recommend in Section 7.3.6 that results should only be collected for pre-specified outcomes, we also suggest that only the outcomes listed in the protocol be described in detail” (Cochrane Handbook, Section 7.3.5) 7.3.6 O Definition? (Diagnostic method, scale, threshold, or behavior) O Timing? O Unit of measurement? O Scale interpretation? (Upper/Lower limits; Favorable outcomes, etc.) O Economic outcomes? (See Cochrane Handbook, Section 15.4.2) O Patient-reported outcomes? (See Cochrane Handbook, Chapter 17) O Adverse outcomes? (See Cochrane Handbook, Chapter 14) O Method of collection? O Number of outcomes reported?

11. Results Collection O “Other outcomes should not be extracted unless the protocol is modified to add them, and this modification should be reported in the review…Review authors should be alert to the possibility of important, unexpected findings, particularly serious adverse effects” (Cochrane Handbook, Section 7.3.6) O Number of participants allocated to each intervention? O Variation in outcome measure/time/summary statistic reporting? O Content for meta-analysis? (see Cochrane Handbook, Section 7.7) O Unit of analysis? O Type of outcome? (Dichotomous, Continuous, etc.) O Summary data for each intervention group? (Mean, Standard Deviation, Standard Error, Confidence Interval, Test Statistic, etc.) O Missing participants? (CONSORT Flow diagram) O Estimate of effect? (Effect size, confidence interval, and p- value) O Sub-group analyses? O Number of outcomes reported?

12. Additional Information for Collection O “Other information will be required from each report of a study, including the citation, contact details for the authors of the study and any other details of sources of additional information about it” (Cochrane Handbook, Section 7.3.7) O Eligibility for inclusion? O Page numbers of content/extraction? O Funding source or potential conflicts of interest? O Key conclusions? (Direction of effect) O References to other relevant studies? O Study quality notes? (Ethical approval, sample size calculation, etc.) O Correspondence required? O Comments/Notes (Date of extraction, dates of revisions, name of extractor, etc.)

13. Rationale for Data Extraction Forms O “The data collection form serves several important functions (Meade 1997)… 1. the form is linked directly to the review question and criteria for assessing eligibility of studies, and provides a clear summary of these that can be applied to identified study reports… 2. the data collection form is the historical record of the multitude of decisions (and changes to decisions) that occur throughout the review process… 3. the form is the source of data for inclusion in an analysis” (Cochrane Handbook, Section 7.5.1) O “Given the important functions of data collection forms, ample time and thought should be invested in their design” (Cochrane Handbook, Section 7.5.1, emphasis added)

14. Data Extraction Form Types Paper Advantages Electronic Advantages O convenience or preference; O data extraction can be undertaken almost anywhere; O easier to create and implement (no need for computer programming or specialist software); O provides a permanent record of all manipulations and modifications (providing these manipulations and modifications are not erased); and O simple comparison of forms completed by different review authors O convenience or preference; O combines data extraction and data entry into one step; O forms may be programmed (e.g. using Microsoft Access) to ‘lead’ the author through the data collection process, for example, by posing questions that depend on answers to previous questions; O data from reviews involving large numbers of studies are more easily stored, sorted and retrieved; O allows simple conversions at the time of data extraction; O rapid comparison of forms completed by different review authors; and O environmental considerations Cochrane Handbook, Section 7.5.2

15. Data Extraction Form Examples O Sample Data Extraction Forms: O Berkeley Systematic Reviews Group: http://www.medepi.net/meta/forms.html http://www.medepi.net/meta/forms.html O Cochrane Cystic Fibrosis and Genetic Disorders Group: http://cfgd.cochrane.org/Files/Study%20selectio n%20and%20%20extraction%20form.doc http://cfgd.cochrane.org/Files/Study%20selectio n%20and%20%20extraction%20form.doc O Petticrew and Roberts Systematic Reviews in the Social Sciences: A Practical Guide: http://onlinelibrary.wiley.com/doi/10.1002/9780 470754887.app4/pdf http://onlinelibrary.wiley.com/doi/10.1002/9780 470754887.app4/pdf O Cochrane Schizophrenia Group: http://www.cochrane- net.org/openlearning/Other/Example_from_a_rev iew_in_schizophrenia.pdf http://www.cochrane- net.org/openlearning/Other/Example_from_a_rev iew_in_schizophrenia.pdf

16. Data Extraction Form Development O “If review authors plan to develop their own electronic forms using spreadsheet or database programs, we recommend that O (i) a paper form is designed first, and piloted using more than one author and several study reports; O (ii) the data entry is structured in a logical manner with coding of responses as consistent and straightforward as possible; O (iii) compatibility of output with RevMan is checked; and O (iv) mechanisms are considered for recording, assessing and correcting data entry errors” (Cochrane Handbook, Section 7.5.2)

17. Example Data Extraction O Systematic Review Research Question: Amongst women in polygamous marriages, as compared to women in monogamous marriages in the same population, what is the prevalence of mental health issues? O See Paper Data Extraction Exercise Example O See Electronic Data Extraction Form Example

18. StudySettingSampleSubgroups Measurement Tools AnalysesResults Al-Krenawi & Slonim- Nevo, 2008 The Negev Desert Region, Israel Quasi- Random Sample; Bedouin Arab Women Recruited by Phone in Conjuncti on with a Randomi zed Trial Involving Their Children (N=315) 159 Monogam ous and 156 Senior Wives of Two Wife Families A Socio- demographic Questionnaire, the Self-Esteem scale (SE), the Brief Symptom Inventory (BSI), the McMaster Family Assessment Device (FAD), the ENRICH questionnaire, and the Index of Parental Attitudes Independ ent Samples t-test; Multiple Linear Regressio ns: Full Model and Best Predictive Model “The findings show that the wives in polygamous marriages suffered from more psychological difficulties than those in monogamous marriages” with higher levels of somatization, depression, anxiety, hostility, and paranoid ideation (p<0.05); greater general severity (GSI) and total number of symptoms (p<0.05); lower self- esteem, less marital satisfaction, and more problematic family functioning (p<0.001). Family functioning was the strongest and most consistent predictor of mental distress; when family functioning was held as a dependent variable, polygamy was only significantly predictive of self-esteem (p<0.001)

19. Workshop Task O For your Systematic Review Research Question: O Begin to develop a data extraction form O For one paper in your review, begin to pilot your data extraction O Consider how to revise your form or process to make the most of your time O How to balance detail with usefulness? (i.e., too little versus too much information)

20. Writing the Protocol O “The methods section of both the protocol and the review should detail: O the data categories that are to be collected; O how verification of extracted data from each report will be verified (e.g. extraction by two review authors, independently); O whether data extraction is undertaken by content area experts, methodologists, or both; O piloting, training and existence of coding instructions for the data collection form; O how data are extracted from multiple reports of the same study; and O how disagreements are handled if more than one author extracts data from each report” (Cochrane Handbook, Section 7.6.6)

21. References O Higgins JPT, Green S (editors). Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. The Cochrane Collaboration, 2011. Available from www.cochrane-handbook.orgwww.cochrane-handbook.org O Social Care Institute for Excellence (2006). The conduct of systematic research reviews for SCIE knowledge reviews. Available from http://www.scie.org.uk/publications/resear chresources/rr01.pdf http://www.scie.org.uk/publications/resear chresources/rr01.pdf