1. LECTURE 2 EPSY 642 META ANALYSIS FALL 2009

2. CONCEPTS AND OPERATIONS CONCEPTUAL DEFINITIONS: HOW ARE VARIABLES DEFINED? Variables are operationally defined by assigning a measurement to the variable; height is measured by a ruler or by a 5 item rating scale of low to high CONSTRUCT: CONCEPT EMBEDDED IN A THEORETICAL STANCE OR PERSPECTIVE Intelligence may differ in theories as single or multifactor constructs; measurement by the Stanford- Binet or Wechsler Intelligence Scale for Children will yield different score(s)

3. CONCEPTS AND OPERATIONS Studies need to be clear on the conceptual definitions Ex. Willson (Willson, V. L. (1983). A meta-analysis of the relationship between science achievement and science attitude: Kindergarten through college. J. Research in Science Teaching, 20, 839-850.) noted that attitude was conceived of as “interest” in the 1930s and 1940s, differed from attitude measured using the psychological development of the construct of attitude in the 1960s Meta-analysis needs to include these definitions in the coding and analysis process

4. CONCEPTS AND OPERATIONS Operational definitions within studies need to be examined and coded- are they consistent with the conceptual definitions, and if not, how and why do they differ? As the selection and reading of studies progresses, make room for revision in all phases of this process What studies to add- do you include studies that use the operationally defined measures, but were not focused on your concept? Ex. Pretest sensitization: do I use studies that “studied” sensitization or studies that included pretests? If the former, 200 studies, if the latter 100,000- or is there a middle ground?

5. Defining and Delineating the Research Topic Outcome construct definition – Importance to the field to know what has been learned – How big is it? How many potential studies? – Conduct preliminary searches using various databases Refining the construct – How much resource is available? Eg. 1000 studies = 2-3 years work – Are there specific sub-constructs more important than others? Select them or one of them – Are there time-limitations (no studies before 19xx) – Are there too few studies for the given construct, should it be broadened? Too few-> less than 10?

6. Defining and Delineating the Research Topic What is the typical research approach for the topic area? All quantitative All qualitative Mixed quantitative and qualitative Are there sufficient quantitative studies to provide evidence for findings? Can qualitative studies be included as a separate part of the study? How?

7. Within-Unit vs. Between Unit Within-unit vs. Between-unit processes: is or can the focus be on unit growth or is it a “snapshot” that may be evaluated at different time points, but the units are different at each time point. Ex. IQ growth differs if one assesses a group of people every 5 years for 50 years (or subsets that are measured over fairly long periods of time that overlap in age) vs. measuring 10 groups of people at age 20,25,…70 right now The conclusion about IQ and age is very different in the two sets of studies

8. Clinical/case studies vs. Group Studies Ideographic vs. Nomothetic conclusions Is the intended result to be for groups or for individuals? Are there possible interaction effects across levels of inference (student vs. classroom vs. school) – what level will be the focus for the study? What is a case study in one article will be part of a group study in another: a school may be a case study or it may be part of an experimental design

9. Determining Research Outlet Does the proposed journal – publish research on the construct? – Publish reviews or meta-analyses? Is there a journal devoted to reviews that your project would fit with? Has a recent similar meta-analysis been published? If so, will yours add anything new? – Ex. Allen, et al (under review) evaluated articles on first grade retention after 1990 focusing on the quality of the research design in each study to determine if the effects were different from a fairly recent meta-analysis by Jimerson (2001)

10. Meta-Analysis as an interactive, developing process View meta-analysis as evolutionary – As studies are reviewed and included, purpose and scope may change Assume initial conceptualizations about both outcomes and potential predictors may change over time – Definitions, instruments, coding may all change as studies are found and included Plan for revisions to all aspects of the meta-analysis

11. CONCEPTUAL RELEVANCE AND SELECTION START BROAD, THEN REDUCE PROVIDE FOR INTER-SCREENER RELIABILITY SEE TABLE 2.1 IN COOPER

12. CAUSAL INFERENCE MAKING CAUSAL INFERENCES ABOUT A TREATMENT CAUSING AN OUTCOME MUST BE BASED ON STUDIES THAT TEST THAT HYPOTHESIS IT IS IMPERMISSIBLE TO USE DIFFERENT STUDIES TO MAKE THE INFERENCE THAT DID NOT MAKE THE TREATMENT COMPARISON WITHIN THE STUDIES Study X reported a 10 point gain on test A for Treatment 1 with a random sample of children in Smallville Study Y reported a 1 point gain on test A for Treatment 2 with a random sample of children in Smallville We cannot conclude Treatment 1 is better than 2

13. FINDING STUDIES Searches Selection criteria

14. Searches Traditional literature review methods: – Current studies are cumulated Branching backward search uses the – Reference Lists of current studies Electronic searches – Google, Google Scholar, PsyInfo, research library catalogs (for major research institution libraries) – Searches of major journal article titles and abstracts (commonly available now through electronic libraries) – Querying listserves, informal research networks, “in-group” researchers, prominent researchers Abstract vs. full content searches- electronic, pdf, hard copy Author requests: email or hard copy requests for newly published articles or other works not found in typical search outcomes

15. Selection Criteria In or out: – Any quantitative data available? Descriptive data- means and SDs for all groups of interest? Analysis summaries- F- or t-tests, ANOVA tables etc. available that may be utilized? Iterative process: outs may come back in given broader definitions of a construct Duplicated articles/data reports? Decide on which to keep (earliest? Most complete?) why were multiple articles prepared? New groups included that can be used? Keep records of every study considered- excel or hard copy, for example

16. Selection Criteria Useful procedure: Create an index card for each study along with notes of each to refer to Organize studies into categories or clusters Review periodically as new studies are added, revise or regenerate categories and clusters Consider why you organized the studies this way- does it reflect the scope of research, construct organization, or other classes?