Texas A&M University EPSY 642 FALL 2009 Victor L. Willson, Instructor

EPSY 642 Meta Analysis of Social Science Research Syllabus FALL 2009 TUESDAYS 4:45 – 7:35 Victor L. Willson, Professor:Office: by appt 704 Harrington /fax: Course description: Theory and methodology of current state of the topic of meta-analysis of research documents. Course objectives: Students will learn meta-analysis methods by conducting a meta-analysis on a circumscribed topic of their choice. Students will be required to present their findings at the end of the course and produce a research paper on their topic. Students will be encouraged to develop a potentially publishable research article from their work. Components for the project will be reported by students during the semester. Texts: Lipsey, M. W., & Wilson, D. B. (2001). Practical Meta-Analysis. Sage. Cooper, H. (2010). Research Synthesis and Meta-Analysis. 4 th Ed. Sage

Date Topic Reading Assignment due SEP 1Introduction: Purposes, Approaches HC 1 Activity: Online Google Scholar search 5 articles: samples description, variable list 8NO CLASS 15Problem finding, Study SelectionLW2, HC Study coding, menu elementsLW4, 5, HC 4, handout 29Outcome variable effect computation LW3, HC 6, handouts OCT 6 Study Design variablesHS 2, HC 5 13Correlational study analysis LW 6-7 Experimental study analysisLW 7 #1 27Findings cumulation and reportingHC 6-9, LW 8 #2 handouts & examples NOV 3Advanced Modelinghandouts# Individual meetings #4 Nov 10, #5 Nov 17 24Presentation of student papers using powerpoint DEC 1continuation of presentations as needed

Grades: Based on completion of intermediate assignments (20%) and quality of paper (80%) U= < 60% S = 61% - 100% Intermediate assignments: 1 Proposed meta-analysis topic with preliminary list of at least 20 articles 2Preliminary list of proposed variables to be extracted from articles 3Proposed Coding menu for variables 4Preliminary analysis of outcomes using Q method, funnel plot of one outcome variable 5Preliminary covariate analysis using QB and QW method (at least 1 covariate) Project and paper: conduct a meta-analysis using the procedures of this course on a topic of your choice; limit the number of studies to be finally included to no more than about 20 but at least 8 as appropriate to the topic. Produce a finished paper in APA format summarizing the procedures and results; produce a powerpoint of your paper for presentation. In order to receive a grade a presentation must be made on the dates listed; otherwise an incomplete will be given and a presentation to me and a group of students must be made before a final grade can be submitted. Final papers may be turned in after the last day to turn in grades for the semester with a grade of I given and no penalty, but after a presentation has been made and all intermediate assignments are completed. After one semester a grade of F will be given if these conditions are not met.

Note: The handouts and web-based files used in this course are copyrighted. By “handouts” I mean all materials generated for this class, which includes but is not limited to syllabi, quizzes, exams, lab problems, in-class materials, review sheets, and additional problem sets, in paper or electronic form. Because these materials are copyrighted, you do not have the right to copy the handouts unless I expressly grant permission. As commonly defined, plagiarism consists of passing off as one’s own ideas, words, writings, etc. which belong to another. In accordance with this definition, you are committing plagiarism if you copy the work of another person and turn it in as your own, even if you should have the permission of that person. Plagiarism is one of the worst academic sins, for the plagiarist destroys the trust among colleagues, without which research cannot be safely communicated. If you have any questions regarding plagiarism, please consult the latest issue of the Texas A&M University Student Rules, under the section “Scholastic Dishonesty”. Honor Code and refers the student to the Honor Council Rules and Procedures on the web: “An Aggie does not lie, cheat or steal, or tolerate those who do.”

Purposes for Meta-Analysis Cumulate findings of studies on a particular topic Examine homogeneity of outcomes Estimate effects of independent variables on outcomes in a standardized format – Evaluate moderator and mediator effects on outcomes – Differentiate different types or classes of outcome effects

Historical background Criticism of traditional narrative reviews of research Exasperation in social sciences with constructs measured different ways in terms of determining consistencies Need to formulate theoretical relationships based on many studies

History part 2 Early 1970s efforts focused on significance testing and “vote counts” of significance Glass (1976) presented a method he called “meta- anlaysis” in Am. Ed. Research Assn. presidential address Others proposed related methods, but Glass and colleagues developed the most widely used approach (Glass, McGaw, & Smith, 1981)

Meta-Analysis as Survey Research Research articles as unit of focus Population defined – Conditions for inclusion of articles Data requirements needed for inclusion Completeness of data available in article or estimable Publication sources available, selected Sample vs. population acquisition – Availability of publications and cost – Time to acquisition (how long to wait for retrieval)

Strengths of Meta-Analysis Definition of effect and effect size beyond “significant or not” Focus on selection threats in traditional reviews (bias in selection of articles for review) Systematic consideration of potential mediators and moderators of effects Data organization of articles for public review

Weaknesses of Meta-Analysis Methodologically sophisticated and expensive Potential ignoring of contextual effects not easily quantified; eg. historical/environmental placement of research Potential improper mixing of studies Averages hiding important subgroupings Improperly weighting studies with different methodological strength/rigor

Research focus for meta-analysis Defining and delineating the construct Determining a research outlet Meta-Analysis as an interactive, developing process

Recent Criticism Suri & Clarke (2009) : Advancements in Research Synthesis Methods: From a Methodologically Inclusive Perspective (Review of Educational Research, pp ) They propose 6 overlapping approaches: 1. Statistical research syntheses (eg. meta-analysis) 2. Systematic reviews 3. Qualitative research syntheses 4. Qualitative syntheses of qualitative and quantitative research 5. Critical impetus in reviewing research 6. Exemplary syntheses

Some critical comments on Suri & Clarke (2009) Systematic reviews- original Glass criticisms hold: what is the basis for inclusion and exclusion; why are certain articles privileged? Qualitative research syntheses- how can these be done with situated contexts, small samples, environmentally- developed variables, sources, etc.? Will there be a review for every reader, or for every researcher? Same limitation as all qual research Qual syntheses of quant and qual research- potentially doable, with an alternating order: qual first to focus emphases in the quant analysis, or quant first to be validated with the qual studies of particular environments and populations- do they fit/match in reasonable ways? Critical impetus- code words for critical theory/Marxist etc. Answer is already known, why do the research? Exemplary syntheses- what is the purpose?

Defining and Delineating the Research Topic Outcome construct definition – Importance to the field to know what has been learned – How big is it? How many potential studies? – Conduct preliminary searches using various databases Refining the construct – How much resource is available? Eg studies = 2-3 years work – Are there specific sub-constructs more important than others? Select them or one of them – Are there time-limitations (no studies before 19xx) – Are there too few studies for the given construct, should it be broadened? Too few-> less than 10?

Defining and Delineating the Research Topic What is the typical research approach for the topic area? All quantitative All qualitative Mixed quantitative and qualitative Are there sufficient quantitative studies to provide evidence for findings? Can qualitative studies be included as a separate part of the study? How?

Determining Research Outlet Does the proposed journal – publish research on the construct? – Publish reviews or meta-analyses? Is there a journal devoted to reviews that your project would fit with? Has a recent similar meta-analysis been published? If so, will yours add anything new? – Ex. Allen, et al (under review) evaluated articles on first grade retention after 1990 focusing on the quality of the research design in each study to determine if the effects were different from a fairly recent meta-analysis by Jimerson (2001)

Meta-Analysis as an interactive, developing process View meta-analysis as evolutionary – As studies are reviewed and included, purpose and scope may change Assume initial conceptualizations about both outcomes and potential predictors may change over time – Definitions, instruments, coding may all change as studies are found and included Plan for revisions to all aspects of the meta-analysis

ACTIVITY Get into groups of 3 or 4 Use GOOGLE SCHOLAR to find research articles from the following: Group 1:elementary school grade retention Group 2: teenage depression treatments Group 3: IQ and socioeconomic status Groups 4+: one of above Select 5 articles that have empirical data For each article define the population being samples List with specific names the variables used in the study Discuss within your group the similarities and differences of the populations and variables; consider difficulties you would have in summarizing research for these Plan to report to the class what you found