Establishing the Validity of Test Accommodations for Students with Disabilities: A Collaborative of State-based Research CTEAG Project Summary of Accomplishments and Dissemination of Final Products John F. Olson, Ph.D. Principal Investigator & Project Coordinator, CTEAG Project CCSSO NCSA and Joint ASES-TILSA Meeting Detroit, MI June 23, 2010

CTEAG Project - Overview Goals Conduct research that focuses on validity of test accommodations and interpretations of scores from accommodated state assessments Use a common research design to conduct the studies and to analyze data for comparisons across states and accommodations Build a shared body of research-based validity evidence on test accommodations across multiple states Share findings from project that provide states with additional evidence (e.g., to submit to the USED for peer reviews) in support of their decisions on the use of various accommodations 2

CT EAG Project Structure Lead State – Connecticut Other Participating States – Kentucky, Michigan, and Nevada Additional State Input and Involvement – from ASES and TILSA members Project Management Team – group of SCASS advisors, experts, and consultants that oversaw and managed all validity studies and reporting activities CCSSO -- provided logistical support, staffing, and consultants to do tasks for the project

CTEAG Research Questions 1.Does the accommodation affect item and test form statistics? 2.Does the accommodation change the content structure of the test, i.e., is there an impact on the test structure under accommodated and standard administrations? 3.Does the accommodation increase performance of students? Does it increase performance more for SWD than it does for SWOD?

CT EAG – Validity of Accommodations Project State Participation – Details of Studies 5 Topics/IssuesCT #1CT #2KYMINV AccommodationComputerized text reader (computer voice) Computerized text reader with human voice and other mods made since CT#1 Read aloudEnhanced directions and fewer response options Read aloud by a trained individual (e.g teacher or aide) Grade75467 Content AreaReading comprehension MathReadingELAMath Administration:  Test Date  Study Design  Items/Forms development May Computer- based test forms with 20 MC items per form. Spring Computer-based test forms. Used old secure math test with 26 MC items per parallel form. Mar Developed forms from a pool of non- state items that were similar in content to KY forms with 22MC items per form. Oct 2008 and Jan Used released items – 24 MC and 1CR per form. Nov-Dec Used 20 MC items per form.

Summary of Results John Olson Establishing the Validity of Test Accommodations June In most cases, accommodations did not alter item stats (difficulty, discrimination) Accommodations also did not change test form stats/characteristics (mean item difficulty, reliability) Accommodations did not have an effect on content structure of test forms Accommodations did not change dimensionality of tests

Summary of Results (continued) John Olson Establishing the Validity of Test Accommodations June Overall, results from ANOVA were mixed Accommodations increased performance of students in KY study; enhanced directions increased performance in MI Differential boost effect found in some cases, i.e., SWD scores increased more than SWOD in CT#2, KY, and MI studies Key areas of focus – Differential boost effects Increasing the accessibility of the assessment

CTEAG Project - Accomplishments Assisted and guided states in conducting 5 different validity studies on test accommodations Analyzed all data from the studies and reviewed results with the PMT and states (and sometimes with states’ vendors) Summarized results in a series of detailed state reports Shared updates and information on progress of project at 8 ASES and TILSA meetings from 2007 to 2010 Completed and disseminated 3 reports and a Project Database Presented information on the project at 4 national conferences and meetings ( ) Conducted 3 evaluations of the work, surveying states, and summarizing the info in an Evaluation Report C 8

CTEAG Final Reports/Products Guidebook for Test Accommodations Validity Studies -- provides overall summary of results/findings and guidance to states on validity studies in a nontechnical manner [report was distributed to ASES and TILSA in Feb. 2010] CTEAG Technical Report – provides technical information on the project and details of each state’s study, e.g., designs, methods, analyses, statistics, results, etc. 9

CTEAG Final Reports/Products Lessons Learned from State Validity Studies – provides information on the background and context in which the validity studies were conducted, issues that states may need to consider, and lessons learned by the four participating states CTEAG Project Database -- provides additional info from the five studies, such as, samples, variables, data files, and statistical data for use in further research 10

Dissemination Activities Presentation Session at 2010 NCSA on June 22 −State participants present results and perspectives from conducting validity studies −Representative from USED discusses importance of findings from federal perspective Dissemination Workshop at Joint ASES-TILSA meeting on June 23 −Share all remaining products and summary of accomplishments −State participants discuss lessons learned and how data and findings can be used to benefit states −Expert advisors and USED representative discuss implications at national level 11

Final Dissemination Activities Distribute the reports/products widely to states, the USED, and others Place PDFs of all final products on websites (CSDE, CCSSO/EAG) Issue formal announcement (press release?) and Chiefline newsletter article Coordinate with project partners (NCEO and NASDSE) to share information with others Submit final report on project to USED 12

Questions & Comments For more information, contact: John F. Olson President/Founder Olson Educational Measurement & Assessment Services (home/office) (cell)