0. Systematic Review Module 6: Data Abstraction
Joseph Lau, MD Thomas Trikalinos, MD, PhD Tufts EPC Melissa McPheeters, PhD, MPH Jeff Seroogy, BS Vanderbilt University EPC

1. CER Process Overview
You may be attending multiple presentations on various parts of the CER process. This slide is to anchor us to which part of the process we are discussing.

2. Learning Objectives What is data abstraction? Why do it?
What kind of data to collect?
How much data to collect?
How to collect data accurately and efficiently?
How many extractors? With what background?
How do abstraction forms look like?
What are some challenges in data abstraction?
Is it feasible to query original authors?
[no comment]

3. Aims of Data Abstraction
Summarize studies to facilitate synthesis
Identify numerical data for meta-analyses
Obtain information to assess the quality of studies more objectively
Identify future research needs
A structured and organized data abstraction is necessary for a systematic review or meta-analysis.
[Third bullet:] In practice, data extraction, in-depth review, and quality assessment happen at the same time.
[Fourth bullet:] Often, during the data extraction phase, the reviewer realizes that some information is systematically missing or not correctly assessed; some outcomes are never studied; or some populations are underrepresented in the existing evidence.

4. On Data Abstraction (I)
Abstracted data should
accurately reflect information reported in the publication
remain in a form close to the original reporting (so that disputes can be easily resolved)
provide sufficient information to understand the studies and to perform analyses
Abstract what is needed (avoid over doing it); data abstraction is labor intensive and can costly and error prone
Different question may have different data needs

5. On Data Abstraction (II)
Involves more than copying words and numbers from the publication to a form
Clinical domain, methodological, and statistical knowledge is needed to ensure the right information is captured
Interpretation of published data is often needed
Quality assessment of articles belongs in this step
Appreciate the fact that what is reported is sometimes not necessarily what was carried out

6. What Data to Collect? Guided by key questions and eligibility criteria
Anticipate what data the summary tables should include, what data will be needed to answer questions, and conduct meta-analyses
Data extraction follows the PICO format and include study design
Population
Intervention or exposure
Comparators (when applicable)
Outcomes and numbers
Study design
[Second bullet]
Go back and re-examine the key questions. Then generate a template of the summary table and your report outline to figure out exactly what you plan to write. You may not be able to do this step until you have been immersed in the literature for a little while. Once you have fields for your summary tables, you will know what your evidence tables should look like.
Your PICO criteria belong in a new slide - it's what is necessary no matter what the specifics of a summary table look like.
You need to explain where these data are headed: that is, explain the differences (and similarities) between a summary table, an evidence table, and a data abstraction form. This is something that new investigators have trouble with. It is also important to explain the relative placement of these products and expectations about their structure and content.

7. Data Elements: P, I, C
Population-generic elements may include patient characteristics such as age, gender distribution, and disease stage. May need more specific items according to topic
Intervention or exposure and comparator items depend on the abstracted study
RCT, observational study, diagnostic test study, prognostic factor study, family-based or population-based genetic studies, etc.

8. Data Elements: O
Outcomes should be determined a priori with Technical Expert Panel
Criteria often are not clear as to which outcomes to include and which to discard
Mean change in ejection fraction or proportion with increase in ejection fraction by > 5%
May be useful to record different outcome definitions and consult content experts before making a decision

9. Data Elements: O
Apart from data on outcome definitions, you need quantitative data for meta-analysis
Dichotomous (deaths, strokes, MI, etc.)
Continuous variables (mmHg, pain score, etc.)
Survival curves
Sensitivity, specificity, Receiver Operating Characteristic (ROC)
Correlations
Slopes
The exact type of data that has to be extracted for meta-analysis depends on the details of the topic at hand. Sometimes 2 x 2 cell counts need be recorded. Other times one has to record other statistics, including means and standard deviations, p-values or ranked LOD scores (for non-parametric meta-analyses), correlation coefficients, distributional parameters, etc. We will not discuss these issues here.

10. Data Elements: Study Design
Varies by type of study
Some information to consider collecting when recording study characteristics for RCTs
Number of centers (multi-center studies)
Method of randomization (adequacy of allocation concealment)
Blinding
Funding source
Intention to treat (ITT), lack of standard definition

11. Clarifying EPC Lingo
In the EPC program, we often refer to the following types of tables:
Evidence tables are prettified data extraction forms. Typically, each study is abstracted to a set of evidence tables.
Summary tables synthesize evidence tables to summarize studies. They contain context-relevant pieces of the information included in the study-specific evidence tables.
Evidence tables are typically found as appendices.
Summary tables are often found in the report, and occasionally in appendices.

12. Developing Data Abstraction Forms (Evidence Tables)
No single generic form will fit all needs
While there are common generic elements, in general, form needs to be modified for each topic or study design
Organization of information in PICO format highly desirable
Well-structured form vs. flexible form
Anticipate the need to capture “unanticipated” data
Iterative process, needs testing on multiple studies by several individuals

13. Common Problems when Creating Extraction Forms (Evidence Tables)
Forms have to be constructed before any serious data extraction is underway
Original fields may turn out to be inefficient or unusable
In practice, reviewers have to
Be as thorough as possible in the initial set-up
Reconfigure the tables as needed
Dual review process helps fill in gaps
Being as thorough as possible with the experts/project leads when identifying fields for inclusion will help mitigate this, but it is inevitable that new material will come to light as you wade through the studies. Reworking the tables will be necessary. The dual review process ensures that nothing is left out if new fields are added after the first round of reviews go out (which is very common).

14. Example First draft Second draft
Drafts of TBI and depression tables illustrate how tables change—columns renamed or relocated, fields moved from column to column, fields added or completely removed, etc.
It is not important to understand exactly which fields changed place or were removed. The important thing to note is that extraction tables change after pilot testing to suit the perceptions or needs of the extractors.

15. Example
Final draft
Some corrections will just be simple: copy and paste text edits (i.e., “design” moving from one column to another in this example); others will require rereading the papers and looking for new information (i.e. the addition of “irritability,” “aggression,” and “suicidality” as fields). Again, the dual review process helps when new fields are introduced, but sometimes core staff members and project coordinators will be leaned on more to ensure tables are complete.

16. Common Problems when Creating Extraction Forms (Evidence Tables)
Lack of uniformity among outside reviewers
No matter how clear and detailed the instructions, data will not be entered identically from one reviewer to the next
Solutions
Evidence Table Guidance document—instructions on how to input data
Limit the amount of core members handling the evidence tables to avoid discrepancies in presentation
This lists some of the biggest/most frustrating problems.

17. Example From the Vanderbilt EPC Evidence Table Guidance document
The “Country, Setting” field: provides a list of possible settings that could be encountered in the literature
Academic medical center(s), community, database, tertiary care hospital(s), specialty care treatment center(s), substance abuse center(s), level I trauma center(s), etc.
The “Study design” field: cross-sectional, longitudinal, case-control, RCT, etc.
Examples from fields that will, regardless of the project, be included in the evidence table. The more thorough the guide, the less variation (and incorrectly classified material), but tables will always be returned with formatting problems, incorrectly placed data, or omitted fields.

18. Example Reviewer A Reviewer B
Two different reviewers, with the same evidence table template, same paper to review, same guidance document, and same set of instructions—but different results.

19. Samples of Final Data Extraction Forms (Evidence Tables)
For evidence reports or technology assessments with many key questions, data extraction forms may become very long (several pages)
The next few slides are examples of data extraction forms: do not study them, just fly through them
When you design your own extraction forms, improvise: there are many possible functional versions
[Technology Assessment on home monitoring of obstructive sleep apnea syndrome AHRQ, 2007, Tufts EPC]

20. Patient and Study Characteristics
Do not waste time to see the specifics of this form. Each form will differ. Each topic will differ.
This first set of two tables pertains to the characteristics of the population.
Note that there are brief reminders of what is to be extracted in each cell. There are more detailed definitions of the operational criteria of what is to be extracted and where in a separate document.
Technology Assessment Home diagnosis of sleep apnea-hypopnea syndrome

21. Characteristics of Index Test and Reference Standard
Again, do not spend time reviewing the slide. The aim is to impress upon the reader that there can be a lot of information that need to be extracted.
Here we get information on the characteristics of the index test and the reference standard, and the criteria that test readers used.
Technology Assessment Home diagnosis of sleep apnea-hypopnea syndrome

22. Results (Concordance/Accuracy)
As before, do not spend time of this slide.
These tables would record numerical information.
Technology Assessment Home diagnosis of sleep apnea-hypopnea syndrome

23. Results (Nonquantitative)
And these tables would record additional information and quality items.
Technology Assessment Home diagnosis of sleep apnea-hypopnea syndrome

24. Considerations in Managing Data Abstraction
How to maximize scientific accuracy under budgetary and logistical constraints?
How many people should extract data?
Should data extraction be performed “blinded” to the author, affiliations, journal, results?
How to resolve discrepancies?

25. Typical EPC Evidence Reports
Systematic review of a topic
5 key questions (e.g., prevalence, diagnosis, management, future research)
Analytic framework, evidence tables, summary tables, meta-analyses, decision models
12 months from start to completion of final report
Screen 5,000 to >10,000 abstracts
Examine several hundred full-text articles
Synthesize 100 to 300 articles
100 to >200 pages in length
[Second bullet]
Some reports have more, some reports have fewer.
[Fourth bullet]
Actually, the critical thing is that the whole report may take 12 months, but the production of a draft report needs to happen within the first 9, so really it's probably 3 to 4 months of dedicated abstraction time. Another issue is that the entire timeline may become less in reports that are prepared under pressure.

26. Metaworks Inc. project summary (EPC I)
Estimating Time to Conduct a Meta-analysis from Number of Citations Retrieved
Metaworks Inc. project summary (EPC I)
37 meta-analysis projects
Mean total number of hours: 1,139 hours
Median: 1,110 hours (216 to 2,518 hours)
Pre-analysis activities (literature search, retrieval, screening, data extraction): 588 hours (standard deviation 337 hours)
Statistical analysis: 144 hours (106)
Report and manuscript: 206 hours (125)
Administrative: 201 hours (193)
Allen IE, Olkin I. JAMA 1999;282:

27. Fixed and Variable Costs Associated with Systematic Reviews
A statistical model was fit to predict total hours for the completion of a meta-analysis as a function of the citations that were retrieved before applying exclusion criteria.
The line suggests that there is a substantial fixed cost when performing a systematic review or a meta-analysis: Even when the scope is relatively contained, there is a substantial amount of time that has to be devoted (fixed costs). Total number of hours increases as the scope expands (variable costs).
JAMA 1999;282:

28. Tools Available for Data Abstraction and Collection (Pros and Cons)
Word processing software (MS Word)
Spreadsheet (MS Excel)
Database software (e.g., MS Access, Epi-Info)
Dedicated off-the-shelf commercial software (e.g., TrialStat)
Homegrown software

29. Who Should Abstract Data and How Many People?
Domain experts vs. methodologists
Single or double independent abstraction followed by reconciliation vs. single and independent verification
Blinded (to authors, journal, results) data abstraction?
[First bullet]
A content expert knows the jargon of the field and can identify subtle differences in treatment options and whether the definitions of outcomes across studies are consistent or not. However, when a content expert has no methodological background, he or she may be unable to back-calculate numerical data that are reported differently (e.g., a 95% CI from a p-value), or may miss important limitations of the studies. Quality assessment can be meaningfully performed by people with good understanding of methodological issues.
[Second bullet]
Independent data extraction in duplicate is the best way to ensure quality in data extraction, but it is also very expensive. Independent verification of abstracted data is faster, and therefore less expensive. However, reviewers may fail to identify mistakes, or may make the same mistakes.
[Third bullet]
The rational is that extractors will be more objective (especially in their quality assessment) if they are undistracted by the impact factor/fame of the journal, their personal biases about the treatments, or the fame of the authors. Limited empirical explorations on whether blinding extractors to authors or journals affects the quality of data extraction and methodological assessment suggest that there is little evidence in favor of blinding extractors. The Pennsylvania meta-analysis blinding study group studied blinding vs. nonblinding of extractors to studies, authors, and treatments, and “concluded that blinding is not necessary when conducting meta-analyses of RCTs”. Further, this is very impractical and time consuming (reformatting papers to blind them took on average 7.7 hours; the blinding process it self took on average 1.3 hours per paper.)
Berlin J. Does blinding of readers affect the results of meta-analysis? Lancet 1997;350:

30. Challenges in Data Extraction
Problems in data reporting
Inconsistencies in published papers
Data reported in graphs
We will not discuss the additional common issues in any detail. We will illustrate examples in the following slides, and then briefly go through additional common issues.

31. Examples of Data Reporting Problems in the Literature
“Data for the 40 patients who were given all 4 doses of medications were considered evaluable for efficacy and safety. The overall study population consisted of 10 (44%) men and 24 (56%) women, with a racial composition of 38 (88%) whites and 5 (12%) blacks.”
[Verbatim]
These numbers are full of inconsistencies that are not apparent unless you actually try to extract simple information. Going sentence by sentence:
At least 40 patients in total
Overall there are 34 men and women – what happened to the other at least six souls?
There are 43 black and white people
Back-calculating the denominator (total number of people) from each proportion is even more amusing. In the order of appearance, the proportion of
men comes from 23 people
women from 43 people
whites from 43 people
blacks from 42 (or 43) people
Seeing these patterns may enable us to see where the errors occurred, but we cannot actually do a lot about that.

32. Examples of Data Reporting Problems
This is obviously a typo, but it could be a marker of poor proofreading.

33. Examples of Data Reporting Problems
[ANIMATED SLIDE]
[CLICK]
The mean age is outside the age range. It seems that the weight range was also put as age range

34. Inconsistencies in Published Papers
Let’s extract the number of deaths in two arms, at 5 years of follow-up. 34

35. Results Text Overall Mortality
[…] 24 deaths occurred in the PCI group, […] and 25 in the MT group […]
[Verbatim]
PCI
(205)
MED
(203)
Dead 24 25 35

36. Overall Mortality (Figure 2 in Manuscript)
PCI
(205) MT
(203)
Dead 24 25 28 35
[The paper clearly states that there is no censoring] 36

37. Clinical Events (Table 2 in Manuscript)
PCI
(205) MT
(203)
Dead 24 28 32 25 35 33 37

38. Digitizing Data Reported in Graphs

39. Data Are Often Presented in Graphical Form
We want to dichotomize measurements for a 2 x 2 table:
Cutoff should be 15 (events per hour) in each axis.
This information is not reported in the paper, but can be extracted from the graph: count the dots!
Ayappa I et al. Sleep. 2004 Sep 15;27(6):1171-9

40. Using Digitizing Software
Engauge Digitizer,
an open-source software:
Each data point is marked with a red “X,” and the coordinates are given in a spreadsheet.
The systematic reviewer will import an electronic version of the figure (scanned, copied from the PDF) into the software, and after setting the axes’ ranges and scales (linear, logarithmic, or other), marks the points in the scatter plot with the red X’s. The software then converts the marks to coordinates.
digitizer.sourceforge.net

41. Reconstructing the Plot to Count Classification at Specific Cutoffs
This is the same plot that we saw in the previous slides, reconstructed from the digitized data. Essentially, we wanted to count the dots to construct a 2 x 2 table for classifying the number of points above and below the 15 events per hour cutoff. The colored areas correspond to the cells of the 2 x 2 table.

42. Reconstructing a Bland-Altman Plot
Further, it is easy to perform additional analyses, like a difference vs. average plot (Bland-Altman plot) to compare measurements with the two methods. Once one has digitized data, it is easy to perform analyses from scratch.
Note that the point of this description is to show the versatility of data extraction methods.
[The original study reports a similar plot, but additional information was needed for this review, and digitizing and re-analysis was preferred.]

43. Additional Common Issues
Missing information in published papers
Variable quality of studies
Publications with at least partially overlapping patient subgroups
Variable quality of conduct and quality
Potentially fraudulent data
[Third bullet]
It is often difficult to clarify whether some publications actually include overlapping sets of patients. Usually, one of three things happens:
Overlapping data (preliminary and later reports)
Same data but different authors
Similar data (same authors) but different cohort (need to verify with authors)

44. Considerations When Contacting Authors for More Information
How important is the information likely to be?
How reliable are additional data?
How likely are you to be successful?
How much effort is required?
Where else should you look for more data?
FDA website
ClinicalTrials.gov - Results Database

45. Types of Missing Data
Detailed PICOTS information (e.g., population demographics, background diet, comorbidities, concurrent medications, precise definitions of outcomes)
Information to assess methodological quality (e.g., randomization methods, blinding)
Necessary statistics for meta-analysis (e.g., standard error, sample size, confidential interval, exact p-value)

46. A Nonexhaustive List of Common Data Abstraction Problems
Non-uniform outcomes (e.g., different pain measurements in different studies)
Incomplete data (frequent problem: no standard error or confidence interval)
Discrepant data (different parts of the same report gave different numbers)
Confusing data (cannot figure out what the authors reported)
Nonnumeric format (reported as graphs)
Missing data (only the conclusion is reported)
Multiple (overlapping) publications of the same study with or without discrepant data
This is a reminder of topics we discussed and other topics we did not discuss.

47. Why Do Such Problems Exist?
It is an eye-opening experience to attempt to extract information from a paper that you have read carefully and thoroughly understood only to be confronted with ambiguities, obscurities, and gaps in the data that only an attempt to quantify the results reveals.
Gurevitch J, Hedges LV. Chapter 17. Meta-analysis: Combining the results of independent experiments. (pg 383). In: Design and analysis of ecological experiments. Samuel M. Scheiner, Jessica Gurevich, eds. Chapman & Hall, New York, 1993.

48. Why Do Such Problems Exist?
Because so few research reports give effect size, standard normal deviates, or exact p-values, the quantitative reviewer must calculate almost all indices of study outcomes Little of this calculation is automatic because results are presented in a bewildering variety of forms and are often obscure. Green BF, Hall JA. Quantitative methods for literature reviews. Annual Review of psychology 1984;35:37-53.

49. Closing Remarks
Laborious, tedious, (could take an hour or more per article); nothing is automatic
To err is human
Interpretation and subjectivity are unavoidable
Data often not reported in a uniform manner (e.g., quality, location in paper, metrics, outcomes, numerical value vs. graphs)