Qualitative Meta-Analysis and Data Synthesis of Recent Research on Test Accommodations for Students with Disabilities: Summary and Implications John F. Olson, Ph.D. Olson Educational Measurement & Assessment Services NCSA Annual Meeting Minneapolis, MN June 2012

Overview: Study of Recent Research on Test Accommodations In 2011, conducted a “qualitative meta-analysis and data synthesis” of recent accommodations research in large-scale student assessments 1.Reviewed literature for relevant studies 2.Collected existing data on studies that examined impact of test accommodations, preferably those using a random-control groups experimental research design 3.Analyzed/summarized data and entered it into a new and expanded summary table from CTEAG database 4.Summarized study in a brief report 2

Research Procedures Searched NCEO Online Accommodations Bibliography (searchable accommodations database on NCEO website) −identified several dozen experimental or quasi-experimental studies in published reports Reviewed literature for relevant studies, such as those noted in key reports or sources summarizing recent accommodations research, e.g., in the Bibliography for the CTEAG Guidebook and in NCEO Technical Reports Obtained copies of published reports 3

Studies and Data Sources Focus was on relatively recent (since 2000) 2 x 2 randomized- control group studies that had test data (Note: they were not that easy to find) Categorized studies into −Useable quantitative data −Partially useable quantitative and/or qualitative data −Not useable or relevant data Found 17 empirical studies with useable quantitative data for inclusion in project −Note caveats on ability to find appropriate studies or data (costs, limited stats reported, no data for calculating effect sizes, etc.) 4

Analysis Obtained data from studies (Ns, Means, Std Deviations) for both groups of students – SWD and SWOD – under accommodated and non-accommodated conditions Analyzed effect sizes from studies (if possible) −Note: Effect size calculations are effective in summarizing different studies; it provides a practical way to measure the strength of the relationship between two variables and the effectiveness of a treatment/effect. Obtained stats from ANOVA (if available) and data on any other analyses that were done Summarized information into an expanded accommodations database summary table 5

Information included in Expanded Summary Table on Effects of Test Accommodations Title of study (publication), name(s) of author(s), and date Accommodation studied Grades and content area tested Sample sizes for SWD and SWOD Information on analysis of the accommodation’s impact on items and test forms, dimensionality, and test construct F-values and p-values from ANOVA (main and interaction effects) Effect sizes (if obtainable) General findings from study 6

(see Summary Table – separate file) 7

General Comments on Findings Validity studies conducted by the CTEAG project were among the most comprehensive ones that were found investigating effects of test accommodations Looking across all studies that were summarized – in general, findings are mixed −Differential boost found in some studies but not others −Signif main effects found in 7 of 17 studies −Signif interaction effects found in 6 of 17 −Large ES found in a some studies, small to no effects in many others 8

Implications for States and Others Meta-analysis provides more evidence for states to use and reference on validity of test accommodations from a variety of sources and types of accommodations Information from this study supplements and expands that from the CTEAG project, i.e. the project database No one study is able to answer all the questions concerning validity of test accommodations Researchers have a new summary table that will be easily expandable to include results from other studies of accommodations as they become available in the future 9

Questions & Comments For more information, contact: John F. Olson Olson Educational Measurement & Assessment Services (home/office) (cell)