(3) Identifying Effect Size (ES) for each study

Overview General Information to keep in mind:  The goal is to convert each study to a single effect size (ES) number that you type into the excel file  Sometimes calculating the ES will be straightforward and easy  Sometimes the study will not provide detailed information to calculate the ES so we will have to make decisions (guesstimates) to arrive at best guess  Keep a record of every decision you make (in excel file) because we will be reporting that information (and last few slides of PPT explains some of the decisions

Two Preliminary Steps: (1) convert to “r” or “d” ? View #1 Empirical research can take many forms (e.g., IV can be dichotomous and/or continuous IV, DV can be dichotomous and/or continuous DV, two variables relationships, etc) and the form of research you are analyzing helps determine which metric may be best: r – Correlation Coefficient - Johnson & Eagly, 2000 suggest using r when the studies composing the meta- analysis primarily report the correlation between variables,Johnson & Eagly, 2000 d – Standardized Difference - Johnson & Eagly, 2000 suggest using d when the studies composing the meta- analysis primarily report ANOVAs, t-tests between groups.Johnson & Eagly, 2000

Two Preliminary Steps: (1) convert to “r” or “d” ? View #2 Rosenthal & DiMatteo, 2001 discussion the advantages of using r over d: Rosenthal & DiMatteo, 2001  Primary studies contain both “correlation”-based and “between-group”-based studies, so how to choose?  Converting ds to rs sometimes loses information but converting rs to ds does not.  r is more easily interpreted. (FYI – flip side is that d is harder to interpret and always a larger number than r)

Two Preliminary Steps: (2) which software? You need to decide:  Free versus $?  Does it have the functions you want/need?  How long to learn how to use it? If have money/time:  “Comprehensive Meta Analysis” If don’t have money/time:  Wilson (Practical Meta Analysis) website See for links to 11 different types of statistical software -

Identifying ES for each study (1) Download “ES Calculator” (see our website)

Identifying ES for each study (2) For each study, identify data from “Results”

Identifying ES for each study (3) Find corresponding data from “Calculator”

Identifying ES for each study (4) Input information to get ES

Identifying ES for each study (5) Put ES into excel file

Two independent raters of ES Double-checking  Two raters of each effect size  Each rates independently of the other  Compare and resolve discrepancies  You will find that rating ES is an iterative process in which you will double-check your own findings over and over again as you learn more information and become more adept at the process

Issues about identifying ES Overarching goal:  “A conservative estimate of effect size was used whenever studies did not report specific statistics.”

Issues about identifying ES What if…  p-value is reported only as “p <.05” Solution:  treat as if p =.05

Issues about identifying ES What if…  only reports that it was “non-significant” Solution:  treat as ES = 0

Issues about identifying ES What if…  only reports grand N, not n for conditions? Solution:  treat as equal sample size in each condition(s)

Issues about identifying ES What if…  two conditions, reports frequencies condition1 = 75% condition2 = 69% grand N = 200 Solution:  ???

Issues about identifying ES What if…  only reports p-value and sample size Solution:  ??? (is there a website for this?)

Issues about identifying ES What if…  does not provide enough info to calculate ES? Solution:  Could try contacting author, but unlikely to work  Keep a record of which studies don’t contain sufficient information because will report that in Method section such as: “Studies were excluded if they did not contain enough statistical information to calculate an effect size (citation, citation, citation, citation, etc)”