Coding Manual and Process

Illustration, and Introduction of metafor Teacher expectancy data (Raudenbush and Bryk) Reading data into R Acquiring the metafor package Simple analyses using metafor What is metafor actually doing?

The Coding Manual Begin by keeping a list of possible covariates you might be interested in But eventually narrow because meta-analytic sample sizes tend to be small, with insufficient degrees of freedom to support too much data. Begin preparing a draft coding manual Consider a web-based coding manual Saves confusing multiple electronic files that coders give you periodically Allows coders to work at home Automatically saves and forwards data to you.

Components of Coding Manual The goal is codes that are useful in describing and predicting effect sizes. An ID system (to be discussed shortly) Substantive codes, e.g., Treatment subtype Methodological Codes, e.g., Kind of comparison group (TT v TC) Effect size codes (effect size and sample size per group) How to deal with missing data Plausible Guesses Distinguishing “No” from “Unknown” Code missing data as NA for analysis in R Partial example next page, full example CROPS

Structure of Coding Manual In meta-analysis, the effect size is the basic unit being described The structure of the manual depends on whether your studies are group comparisons or correlational.

Typical Structure for Group Comparisons ID Variables that are constant in a study e.g., publication status, year published, methods like use of random assignment Variables associated with Group 1 E.g., characteristics of participants, of treatment, of treatment providers Variables associated with Group 2 All of Group 1 codes, plus an indicator for whether the comparison is Treatment-Treatment or Treatment-Control Variables associated with the outcome measure E.g., self-report or not, training of observers, masking, construct being measured Variables associated with effect size E.g., method of computation, sample size.

Typical Structure for Correlational Studies ID Variables that are constant in a study Variables associated with Variable 1 E.g., what is it measuring, how measured, who measured, etc. Variables associated with Variable 2 Variables associated with effect size

An ID System I keep ID bibliography in Microsoft Word Purpose: to locate particular places in the data set To correct errors (e.g., found when cleaning data) To recode (e.g., to change a particular study) Components: Study ID number Comparison ID number (effect sizes are always on comparisons) Dependent Variable ID number Any other information you want to record To record reasons for excluding studies initially used. Sample ID system in Handouts (Part on next slide)

Training of Coders Crucial to train coders, especially If large numbers of studies (so you can’t do it all yourself) If inferences are complex (e.g., should alternating assignment be coded as random?) If coders are undergraduates (or at that level) Random spot checks on all codes by PI

The Coding Process Do effect size coding separate from all other coding Inter-Rater Reliability (IRR) Decide whether to code some or all studies more than once (advisable if you are not the coder). Some or all studies? If some, at least 10-20? Can use third coder (PI?) to resolve disagreements among the first two coders. Report IRR analysis (kappa; see handout)

The Hierarchical Structure of Meta-Analytic Data In between-group treatment outcome studies, Some data are constant over the whole study (e.g., year of publication, publication status), and so only needs to be coded once Some data vary for each treatment condition (e.g., length of treatment, dosage). Some data vary for each outcome measure (e.g., self- vs other report, reliability) Some data vary for each effect size (e.g., n per group, computation method), and so needs to be coded for every effect size So:

Square vs Hierarchical Data Sets The effect size is the basic unit (each line) Square Data Sets What we commonly think of as a data set Every variable coded for each effect size even if redundant Efficient if number of studies (k) or effect sizes is small. Hierarchical Data Sets Code study level variables once for each study Code treatment variables once for each treatment Code outcome variables once for each outcome Code effect size variables once for each effect size Then use “match files” to duplicate all the missing codes from what would have been the square data set Most efficient when k or number of effect sizes is large, but more complex to do.

Data Entry Depends on square or hierarchical data sets If square, one file If hierarchical, one file for each “level” Option #1: First do paper codes, then enter into computer More room for transcription errors Option #2: Direct data entry into, say, SPSS Data Editor, or an Excel Worksheet with a manual as a guide. Or directly into a web-based manual.

Example of Excel Coding Sheet