Developing a Review Protocol, Searching and Data Extraction Trudy Bekkering, PhD Center of Evidence-Based Medicine & Belgian Branch of the Dutch Cochrane Center Centre for Methodology of Educational Research Katholieke Universiteit Leuven, Belgium This presentation is partly based on previous presentations of Oliver Wendt and Chad Nye

Phases Campbell Review Process 1.Register title 2.Develop protocol (project plan) 3.Undertake systematic review 4.Publish systematic review

The Review Protocol A priori statement of aims and methods of the review Research question(s), aims, methods are considered in advance of identifying the relevant literature – Conduct review with minimal bias – Access to peer review – Greater efficiency in review process (Torgerson, 2003)

Review Protocol Advantages Clear research question before the review  Avoid retrieving irrelevant papers A priori inclusion and exclusion criteria  avoid changing criteria as review progresses or studies may be included on basis of their results If decisions are explicit it enables them to be justified Develop protocol as independently as possible from the literature  avoid influence by one or two major studies, less bias (Torgerson, 2003)

Review Protocol Construction “The review protocol should ideally be conceived by a group of reviewers with expertise both in the area and the science of research synthesis” (Egger & Davey-Smith, 2001) Prior knowledge of an area can aid the review process, but can also introduce bias to the review  can be reduced by clear, consistent and logical justifications for inclusion criteria

Contents Protocol Title and authors Background Objectives Methods – Searching for studies – Selecting studies – Data extraction (including assessment study quality) – Data analysis Acknowledgements / conflicts of interests References Tables / Figures

Background Describes the context of the review: why is it important? – Description of the problem – Description of intervention – How the intervention might work – Description potential subgroups Clarify conceptual issues central to the review

Objective(s) Precise statement of objective, using PICO(S) – Which population? – Which intervention(s)? – Which comparison(s)? – Which outcome(s)? – (Which study design?) A well formulated objective focuses the review

Example objective To assess the effects of home-based programmes aimed at improving developmental outcomes for pre-school children from disadvantaged families (Miller & Eakin, 2011)

Methods How to select studies? How to search for studies? Assessment quality of included studies How to extract data? How to analyse results?

Selecting studies Define inclusion and exclusion criteria Key features: – Established a priori – Explicit (explain all terms) – Based on PICO – Include study design, if applicable All retrieved studies are listed in the review either under included or excluded together with a justification

Example inclusion criteria Types of studies Randomised controlled trials. The control group will either receive no intervention or standard care. Studies comparing two different types of home based programme without a control group will be excluded. This is an area where RCTs are technically and ethically possible. As such this review is confined to RCTs as they provide the best evidence of effectiveness. ( Miller & Eakin, 2011)

Types of participants Home based child development interventions for pre-school children from socially disadvantaged families Parents with children up to the age of school entry and who are socially disadvantaged in respect of poverty, lone parenthood or ethnic minority status. Age of school entry can vary between countries (4 to 7 years) and so the upper age range for this review will be the school entry age for the country in which the trial took place. Studies will be excluded if they aim to recruit particular clinical subgroups of parents. Example (cont)

Types of interventions Home based interventions designed to improve child intellectual and socio-emotional development through the provision of relevant knowledge and skills to the parent. The intervention is delivered by trained lay or professional family visitors.

Example (cont) Types of outcome measures Primary outcomes The following child outcomes will be included: Cognitive development (including language development and attention) Socio-emotional development (including self regulation and behavioural development) Adverse outcomes: Parents feel disempowered

Methods How to select studies? How to search for studies? Assessment quality of included studies How to extract data? How to analyse results?

Searching for studies – Preliminary Searches Supports beginning steps: Definition of key concepts & research question Use of standard reference tools and broad searches for review articles and key primary studies – Main Searches Identification of primary studies through searches of databases, Web, branching, manual searches – Final Searches Refine search terms and update original searches

Examples of Databases – Education: ERIC, British Education Index, Australian Education Index, CBCA Education, Education: A SAGE Full-text Collection; Education Full text, Linguistics and Language Behavior Abstracts. – Psychology: PsycINFO, PubMed (Medline), Ageline, Psychology: A SAGE Full-Text Collection,Criminology: A SAGE Full- Text Collection – Sociology: Sociological Abstracts, Contemporary Women’s Issues. Sociology: A SAGE Full-text Collection – Multidisciplinary: Academic Search Premier, ProQuest Dissertations and Theses, FRANCIS, Social Sciences Index, SCOPUS, Web of Science

Main Searches: Decisions – Preparation of Search Strategies What are the key concepts to be searched? How are these represented in each discipline? What are their related terms? How are these key concepts represented in the controlled vocabulary within each database to be searched?

Sample Research Question Approaches to Parent Involvement for Improving the Academic Performance of Elementary School Age Children Chad Nye, Herb Turner, Jamie Schwartz The purpose of this review is to determine the effectiveness of parental involvement in improving the academic performance of school age children in grades K-6.

Using a Thesaurus 1.From the research question, determine the main concepts to be searched (usually there are three): – Intervention: Parental involvement – Outcome: Academic performance – Population: Kindergarten or Elementary students 2.Consult the main database to be searched. 3.Look up each concept in the thesaurus for this database. – A thesaurus is an alphabetical listing of the controlled vocabulary (or descriptors) used within a subject database – A hierarchical arrangement is used so that Broader, Narrower and Related headings may be discovered – The user will be sent from invalid headings to valid headings.

Example: ERIC Selecting the ERIC Descriptors – Descriptors: Parental Involvement See: Parent participation – Related descriptors: Family involvement, Parent-school relationship, Parent role, Parents as teachers – Related keywords: parent* involvement, parent* effectiveness, parent* support, family support – Descriptors: Academic Performance See: Academic achievement – Related descriptors: Science achievement, Reading achievement, Writing achievement, Achievement gains – Descriptors: Elementary School Children See: Elementary school students – Related descriptors: Elementary education, Primary education, Kindergarten Intervention: Parental involvement Outcome: Academic performance Population: Kindergarten or Elementary students

Main Searches: Decisions Construction of the Search Statements – What terms should be searched as descriptors or as “keywords”? – What Boolean operators should be used? – Where should truncation characters be used? (e.g. parent* will retrieve parent, parents, parental) – What limiting features are available to narrow results? (e.g. use of Publication Type codes, time period, language)?

Boolean Operators AND: Both terms must be present in order for a record to be retrieved. Used to combine different concepts. e.g parent participation AND achievement OR: Either term may be present in order for a record to be retrieved. Used to search for related terms or synonyms. e.g. parent OR family NOT: Used between two terms to ensure that the second term will not appear in any of the results. e.g. literacy NOT adult (Parental involvement OR parent participation) AND academic achievement AND (elementary OR primary education)

Example: ERIC, cont’d Combining Keywords/Descriptors using Boolean operators : 1.DE=(Parent participation OR Family involvement OR Parent role OR Parent-school relationship OR Parents as teachers) 2.“Parent* involvement” OR “Parent* effectiveness” OR “Parent* support” OR “family support” 3.#1 OR #2 = 28,958 records 4.DE=(Academic achievement OR Science achievement OR Reading achievement OR Achievement gains) – 46,574 records 5.DE=(Elementary school students OR elementary education OR elementary schools OR primary education OR kindergarten) recs 6.#3 AND #4 AND #5 = 1,669 records

Limiting Your Results Using the Limiting Commands: Limiting fields contain information that is common to a large number of records within a database. These include language, document type, publication year and so. Some limiting fields will vary across databases (e.g. Classification Code, Age Group) Decisions about whether you are going to restrict: The language of the document The search to a certain time period The results to empirical studies only

Example: ERIC, cont’d Using Limits in ERIC: 1.Search by DT= research -reports 2.Combine this search with the previous one – 835/1669 recs Review Date Range Review Language One final comment… Locating ERIC documents:

ERIC record AUTHOR : Polovina,-Nada; Stanisic,-Jelena TITLE: A Study on Family-School Cooperation Based on an Analysis of School Documentation PUBLICATION YEAR: 2007 SOURCE: Online Submission. Journal of Educational Research (Belgrade), v39 n1 p ERIC DOCUMENT LINK: DOCUMENT TYPE: Journal-Articles; Reports-Research LANGUAGE: English DESCRIPTORS: Student-Behavior; Attendance-Patterns; Child-Development; Parents-; Family-School-Relationship; Parent- Teacher-Cooperation; Foreign-Countries; Elementary-Schools; Parent-Participation; Grades-Scholastic; Academic- Achievement; Parent-Influence; Parent-Child-Relationship IDENTIFIERS: Serbia- ABSTRACT: Family-school cooperation is a very complex process that can be studied at different levels in a number of different ways. This study has covered only some aspects of cooperation between parents and teachers, based on school documentation of a Belgrade elementary school. The study covered analyses of 60 Attendance Registers pertaining to 60 classes with 1289 students from Grade 1 through Grade 8 during an academic year. The unit of analysis included: parents attendance at PTA meetings and individual meetings between parents and teachers. In addition to the frequency of parents' visits to school, the relationship between such registered parents' visits and overall academic performance, grades in conduct, excused and unexcused absence from classes were also considered. The research findings indicated interference between development factors (attitude change in parent-child relationship and growing-up) and parents' informal "theory of critical grades" i.e. transitional processes in schooling. The findings confirmed that parents' individual visits to school were mainly meant to offer an excuse for the student's absence from school, while attendance at PTA meetings was linked to poor grades in conduct and missed classes (both excused and unexcused). The findings also showed that parents pursued visiting strategies which were pragmatic, less time-consuming and less emotionally draining ones. The closing part refers to discussions on practical use of the study and possible further research. (Contains 4 graphs.) [This article is the result of the project "Education for Knowledge-Based Society" No ( ), financially supported by the Ministry for Science and Environmental Protection of the Republic of Serbia.] (Author)

Next Steps Repeat these steps for each database to be searched.

Additional Retrieval Methods – The web…to locate grey literature Use Advanced Search screens on large engines (eg. Google, Altavista, AlltheWeb, MSN Live) Consult specific sites – Branching (Pearl Growing) Scan the bibliographies of previous reviews on your topic – Manual searches Browse the Table of Contents of key journals for current years

Managing Your Results Export the results – Save as a Text file Import into a bibliographic management software: – RefWorks, – Reference Manager, – EndNotes Edit your inhouse database – Add Source code for each database searched (e.g ERIC1, PsycINFO1…) – Add notes to the records (e.g.includes vs excludes) Compile a Search History document listing the original search strategies – Use of IRMG/ECG Database worksheet

Information Retrieval: Wrap Up Importance of information retrieval process – Not a “one-shot”deal – Requires expertise in the planning and implementation of searches – Consulting with the Trials Search Advisor or an Information Specialist is highly recommended Use a bibliographic management software – Store, manage and organize results Must have ability to replicate review – Documentation of entire process, including search strategies used for each database, decisions taken, etc.

Methods How to select studies? How to search for studies? Assessment quality of included studies How to extract data? How to analyse results?

Data Extraction What is data extraction? Process of reading through a study and extracting the relevant information from each study The reviewer fills out an data extraction sheet with the appropriate information taken from that study Data extraction sheet may be on paper or electronically

Extraction Form Use close-ended items as much as possible (end goal may be to build a database) Design for easy use – Cluster items that deal with similar themes and format so that they are easy to complete (Lipsey & Wilson, 2001)

Extraction Form (cont.) Extraction form consists of two different parts 1.characteristics that apply to the entire study 2.results: studies typically have multiple outcomes and therefore different effect sizes

Extraction Manual Essential for data extraction process Specifies and describes what study features should be extracted Helps to minimize error and bias in the judgments of the extraction process Facilitate the consistency of extraction Should be pilot-tested with several articles and revised accordingly before actual use

Extraction Manual (cont.) Have a full definition of each item and guidelines for extraction – Assists the coder in handling ambiguous, unusual cases Bottom line: spell everything out in detail, give full definitions of the various response options, and provide guidelines for dealing with borderline cases

Assessment study quality Study quality may affect conclusions A systematic review always includes an assessment of individual study quality Different tools available Describe how to use this information – Description methodological quality – Exclude studies of poor quality? – To include in conclusions (strong versus weak evidence)

Example data extraction form

Example. Study characteristics 1.Research design Non-randomized Randomized study Pseudo-randomized Cluster-randomized 2.Setting Public school Private school 3.Etc.

Example. Population characteristics For each group: 1.Number of participants that were enrolled 2.Number of participants that were randomized 3.Number of participants that completed study 4.Age 5.Gender 6.Ethnicity 7.Baseline reading score 8.Etc.

Example. Intervention characteristics 1.Intervention offered to intervention group Intensity Duration Implementation integrity 2.Intervention offered to control group Intensity Duration Implementation integrity

Extraction study results Descriptive or other statistical information contained in studies needs to be converted into a standard metric by which studies can be compared Continuous outcomes: mean and standard deviation in the groups Dichotomous outcomes: numbers in the groups having the outcome versus the numbers in the groups not having the outcome

Extraction Form Example See example in your package

A. Publication Source 1. Journal2. Dissertation 3.Thesis 4. Unpublished Paper 5. Book 6. Other: ________ B. Subject Characteristics (page(s): ____________) Group Pretest Posttest Attrition Age Grade(s) Males (n) (n) (n) (n)(n) (n) Tx: _22___ _22__ ______ ______ ______ _____ Cp: ______ _____ ______ ______ ______ _____ Co: _22___ __18_ __ 4___ ______ ______ _____

C. Socioeconomic Status (page(s):___________ 1.Low 2. Middle3. Upper 4. Low-Middle 5. Middle Upper6. Labeled Mixed 7.Unclear8. ___________9. NR Comments:__________________________________ Etc., etc.

Bias in data-extraction “Data extraction bias”: reviewer introduces bias into the review – Studies in accord with own view – Different judgments of quality or methodological adequacy to different studies – Reviewer’s awareness of the study authors, the journal, the results  blinding the reviewers – Using multiple reviewers – Having detailed extraction manual – Validating the data extraction Checked by another person Two people working independently (much less common but likely to be more reliable) (Petticrew & Roberts, 2006)

Reliability of extraction Reliability has two dimensions: 1.Consistency of a single reviewer 2.Consistency between different reviewers To check  draw subsample of studies, have reviewers extract them again and compare the results For single reviewer reliability: this person needs to extract studies again after sufficient time has passed so that no memory exists of original extraction For between reviewer reliability: different reviewers extract the same sample without reference to what the other has done (Lipsey & Wilson, 2001)

Reliability of extraction (cont.) Once the appropriate double extraction has been done on a sample, the two sets of results are compared item by item Calculate reliability coefficients: – For continuous variables use Pearson’s r (e.g., participant’s age, effect size values) – For categorical variables use percentage agreement procedure or Cohen’s kappa (e.g., type of intervention, outcome measure, design) Report actual reliability, then resolve discrepancies (Lipsey & Wilson, 2001)

Training the reviewers Effective reviewer must – Understand extraction protocol in detail – Have the knowledge and skills to properly read and interpret research report – Have background in methodology – Be familiar with specific research domain Pilot review is recommended – Carefully read the manual – Extract pilot studies and compare results – Discuss disagreements and improve manual

Protocol on methods Protocol contains Data extraction form, including manual Method of extraction – Paper or electronically (which software) How many reviewers How are reviewers trained How to resolve disagreements

Summary Review protocol important first step when undertaking systematic review Helps to focus and structure the review Limits the scope for bias Use standard format – C2 format is recommended

_Resources_for_Reviewers.php

Examples of C2 Review Protocols are available on:

References / more reading Egger M & Davey-Smith G. (2001). Principles of and procedures for systematic reviews. In M Egger G Smith & D Altman (Eds.), Systematic Reviews in Health Care: Meta-analysis in Context (2 nd ed., pp ). London, UK: BMJ Books. Frost LA & Bondy AS. (2002). The Picture Exchange Communication System. Newark, DE: Pyramid Educational Products. Lipsey MW & Wilson DB. (2001). Practical meta-analysis. Thousand Oaks, CA: SAGE. Miller S & Eakin A. Home based child development interventions for pre-school children from socially disadvantaged families (Protocol). The Campbell Library, Petticrew M & Roberts H. (2006). Systematic Reviews in the Social Sciences: A Practical Guide. Malden, MA: Blackwell Publishing. Torgerson C. (2003). Systematic Reviews. London, UK: Continuum. Resources Campbell Collaboration: education_articles/ECG_Resources_for_Reviewers.php Resources Cochrane Collaboration: handbookhttp:// handbook