1. The Campbell Collaborationwww.campbellcollaboration.org C2 Training: May 9 – 10, 2011 Data Evaluation: Initial screening and Coding Adapted from David B. Wilson and Mark W. Lipsey

2. C2 Training Materials – Oslo – May 2011www.campbellcollaboration.org Overview Coding protocol: essential feature of systematic review Goal: transparent and replicable – description of studies – extraction of findings Forms should be part of C2 protocol

3. C2 Training Materials – Oslo – May 2011www.campbellcollaboration.org Topics Eligibility criteria and screening form Development of coding protocol Assessing reliability of coding Common mistakes

4. C2 Training Materials – Oslo – May 2011www.campbellcollaboration.org Study Eligibility Criteria Flow from research question Identify specifics of: – Defining features of the program/policy/intervention – Eligible designs; required methods – Key sample features – Required outcomes – Required statistical data – Geographical/linguistic restrictions, if any – Time frame, if any Also explicitly states what is excluded

5. C2 Training Materials – Oslo – May 2011www.campbellcollaboration.org Study Eligibility Screening Form Develop a screening form with criteria Complete form for all studies retrieved as potentially eligible Modify criteria after examining sample of studies (controversial) Double-code eligibility Maintain database on results for each study screened Example from MST review in handouts

6. C2 Training Materials – Oslo – May 2011www.campbellcollaboration.org Screening Form Effects of Multisystemic Therapy (MST) Initial Screening Form 1.0 1. Is this paper about MST (perhaps in addition to other topics)? 1. No 2. Yes 99. Can ’ t tell 2. What kind of paper is this?1. MST outcome evaluation 2. Review of MST outcome studies 3. Descriptive, correlational, or case study 4. Theoretical or position paper, editorial, or book review 5. Practice guidelines or treatment manual 6. Other 99. Can ’ t tell

7. C2 Training Materials – Oslo – May 2011www.campbellcollaboration.org Effects of Multisystemic Therapy (MST): Eligibility Screening Form 1.2 1. Does this study include two or more parallel cohorts? Note: This means that at least two groups that received different treatments were assessed at the same point in time. 1. No 2. Yes 99. Can ’ t tell IF NO THEN STOP 2. Is this a randomized experiment?1. No 2. Yes 99. Can ’ t tell IF NO THEN STOP 3. Does this study include a licensed MST program?0. No 1. Yes 99. Can ’ t tell IF NO THEN STOP 4. Does this study include youth ages 10-17 with social, emotional, or behavioral problems? 0. No 1. Yes 99. Can ’ t tell IF NO THEN STOP

8. C2 Training Materials – Oslo – May 2011www.campbellcollaboration.org Screening Coding Guide for “Internet-based Interventions for English Language Learners” Report Characteristics 1. First author (Last, initials) 2. Journal 3. Volume 4. Pages Inclusion Criteria 5. Are participants English language learners? Yes/NoIf no stop 6. Does each student in the intervention group have Internet access at school and/or at home? Yes/NoIf no stop 7. Does the study English language skills/proficiency as an outcome measure? Yes/NoIf no stop 8. Are the participants in the U.S.A.? Yes/NoIf no stop 9. Is the study a between-group? Yes/NoIf no stop

9. C2 Training Materials – Oslo – May 2011www.campbellcollaboration.org Coding practice exercise 1 For the articles provided, code Levels 1 and 2 from the MST coding sheet Use Brunk and either Bourduin or Henggler & Melton

10. C2 Training Materials – Oslo – May 2011www.campbellcollaboration.org Development of Coding Protocol Goal of protocol – Describe studies – Differentiate studies – Extract findings (effect sizes if possible) Coding forms and manual – Both important – Sample coding item from form – Sample manual instructions for item

11. C2 Training Materials – Oslo – May 2011www.campbellcollaboration.org Development of Coding Protocol Types of Information to Code – Setting, study context, authors, publication date and type, etc. – Methods and method quality – Program/intervention – Participants/clients/sample – Outcomes – Findings, effect sizes

12. C2 Training Materials – Oslo – May 2011www.campbellcollaboration.org Types of Information to Code Setting, study context, authors, publications date and type, etc. – Multiple publications; “study” vs “report” – Geographical/national setting; language – Publication type and publication bias issue – Publication date vs study date – Research, demonstration, practice studies – Example from MST review in handouts

13. C2 Training Materials – Oslo – May 2011www.campbellcollaboration.org Types of Information to Code Methods: Basic research design – Nature of assignment to conditions – Attrition, crossovers, dropouts, other changes to assignment – Nature of control condition – Multiple intervention and/or control groups Design quality dimensions – Initial and final comparability of groups – Treatment-control contrast treatment contamination blinding

14. C2 Training Materials – Oslo – May 2011www.campbellcollaboration.org Types of Information to Code Methods: Other aspects – Issues depend on specific research area – Procedural, e.g., monitoring of implementation, fidelity credentials, training of data collectors – Statistical, e.g., statistical controls for group differences handling of missing data

15. C2 Training Materials – Oslo – May 2011www.campbellcollaboration.org Types of Information to Code Method quality ratings (or not) More than 200 scales and checklists available, few if any appropriate for systematic reviews (Deeks et al., 2003) Overall study quality scores have questionable reliability/validity (Jüni et al., 2001) – Conflate different methodological issues and study design/implementation features, which may have different impacts on reliability/validity – Preferable to examine potential influence of key components of methodological quality individually Weighting results by study quality scores is not advised!

16. C2 Training Materials – Oslo – May 2011www.campbellcollaboration.org Cochrane risk of bias framework Focus on identifying potential sources of bias in studies: Selection bias - Systematic differences between groups at baseline Performance bias - Something other than the intervention affects groups differently Attrition bias - Participant loss affects initial group comparability Detection bias - Method of outcome assessment affects group comparisons Reporting bias - Selective reporting of outcomes

17. C2 Training Materials – Oslo – May 2011www.campbellcollaboration.org GRADE system for method quality Quality of evidence across trials Outcome-specific Considers: sparse data, consistency/inconsistency of results across trials, study designs, reporting bias, possible influence of confounding variables Software available at: www.ims.cochrane.org/revman/gradepro www.ims.cochrane.org/revman/gradepro Also see: www.gradeworkinggroup.org: www.gradeworkinggroup.org

18. C2 Training Materials – Oslo – May 2011www.campbellcollaboration.org Types of Information to Code Program/Intervention – General program type (mutually exclusive or overlapping?) – Specific program elements (present/absent) – Any treatment received by the comparison group – Treatment implementation issues integrity amount, “dose” – Goal is to differentiate across studies – Examples

19. C2 Training Materials – Oslo – May 2011www.campbellcollaboration.org Types of Information to Code Participants/clients/sample – Data is at aggregate level – Mean age, age range – Gender mix – Racial/ethnic mix – Risk, severity – Restrictiveness; special groups (e.g., clinical) – Examples

20. C2 Training Materials – Oslo – May 2011www.campbellcollaboration.org Types of Information to Code Outcome measures – Construct measured – Measure or operationalization used – Source of information – Composite or single indicator (item) – Scale: dichotomous, count, discrete ordinal, continuous – Reliability and validity – Time of measurement (e.g., relative to treatment) – Examples

21. C2 Training Materials – Oslo – May 2011www.campbellcollaboration.org Types of Information to Code Findings – Compute effect sizes when possible – May need to aggregate data or reconfigure findings Add back the “dropouts” Compute weighted means of subgroups (e.g., boys and girls) – Code data on which computations based (common situations) – We will look at this part of the coding in the next section

22. C2 Training Materials – Oslo – May 2011www.campbellcollaboration.org Development of Coding Protocol Iterative nature of development Structuring data – Data hierarchical (findings within studies) – Coding protocol needs to allow for this complexity – Analysis of effect sizes needs to respect this structure – Flat-file (example) – Relational hierarchical file (example)

23. C2 Training Materials – Oslo – May 2011www.campbellcollaboration.org Data extraction Double data extraction Cohen’s kappa Agreement on key decisions – Study inclusion/exclusion, key characteristics, risk of bias, coding of results Pilot-test and refine codes!

24. C2 Training Materials – Oslo – May 2011www.campbellcollaboration.org Example of a Flat File Note that there is only one record (row) per study Multiple ESs handled by having multiple variables, one for each potential ES.

25. C2 Training Materials – Oslo – May 2011www.campbellcollaboration.org Example of a Hierarchical Structure Note that a single record in the file above is “related” to five records in the file to the right Study Level Data File Effect Size Level Data File

26. C2 Training Materials – Oslo – May 2011www.campbellcollaboration.org Coding exercise 2 For either Borduin or Henggler & Melton, please code the Level 3 items (do not do the outcomes and effect sizes) Report back: what was easy/difficult?