Copyright © 2006 Educational Testing Service Listening. Learning. Leading. 1 College Admissions Testing: Performance, Validity and Use Cara Cahalan-Laitusis Center for Validity Research ETS Research and Development Division

Copyright © 2006 Educational Testing Service Listening. Learning. Leading. 2 A Few Facts Students with disabilities are increasingly applying for and attending college. About half of students with LD who receive testing accommodations on the SAT do not disclose their disability to the college’s office of disability services. Some colleges no longer require admission test scores, but very few refuse to look at them. More colleges and college programs are focusing on specific disability subgroups (LD, Downs Syndrome, Autism).

Copyright © 2006 Educational Testing Service Listening. Learning. Leading. 3 Today’s Presentation Flagging of test scores Changes in accommodation policy from High School Exit Exams to Admissions Tests Changes in documentation of disability from High School Exams to Admissions Tests Research on extra time and the SAT

Copyright © 2006 Educational Testing Service Listening. Learning. Leading. 4 What is Flagging? Controversial policy of reporting test scores with a notation of “non-standard administration.” Usually does not indicate what type of non-standard administration. Nearly all flagged scores are for disability-related testing accommodations that alter the construct being measured.

Copyright © 2006 Educational Testing Service Listening. Learning. Leading. 5 Standards on Flagging “important information about test score meaning should not be withheld from test users who interpret and act on test scores, and irrelevant information should not be provided.” From Standards for Educational and Psychological Testing, 1999, p. 105.

Copyright © 2006 Educational Testing Service Listening. Learning. Leading. 6 Recent Policy Changes on Flagging for Extra Time Educational Testing Service discontinues flagging policy for extra time in The College Board announced discontinuation of flagging policy in 2002 after convening a blue ribbon panel. ACT announced discontinuation of flagging policy shortly after College Board’s announcement. Other testing organizations considered making changes but not all followed suit.

Copyright © 2006 Educational Testing Service Listening. Learning. Leading. 7 What Types of Accommodations Result in Flagged Test Scores? Extra time on some graduate admissions tests and licensure tests. Removal of test sections (e.g., listening section of language proficiency test for deaf test takers). Other accommodations that interfere with the construct being assessed.

Copyright © 2006 Educational Testing Service Listening. Learning. Leading. 8 Results of Survey on Flagging and Admission Decisions (Mandinach, Cahalan, & Camara, 2002) Most admissions officers saw the flag and assumed the student had a disability and received extra time. Most admissions officers and High School guidance counselors wanted the flag to remain and felt that it advantaged the student to disclose their disability.

Copyright © 2006 Educational Testing Service Listening. Learning. Leading. 9 Additional Results (Mandinach, Cahalan, & Camara, 2002) College disability service providers wanted the flag removed because they felt it forced self disclosure. Guidance counselors and admissions officers were concerned that removal of flag would increase manipulation of the system by wealthy parents.

Copyright © 2006 Educational Testing Service Listening. Learning. Leading. 10 Impact of Decision to Stop Flagging Stricter documentation review guidelines. Definition of construct being measured is more precise. Some accommodations are no longer allowed (or rarely allowed and flagged) because of changes to the construct.

Copyright © 2006 Educational Testing Service Listening. Learning. Leading. 11 Accommodation Policies State Assessments vs. Admissions Inconsistency in the “Big 4” –Extra Time –Read aloud of Reading tests –Calculator for Math tests –Computer/Spell Check/Scribe for Writing tests

Copyright © 2006 Educational Testing Service Listening. Learning. Leading. 12 Extra Time Not considered accommodation in several states and liberal time limits for all students in most states. Accommodation on all admissions tests and still flagged on some graduate admissions tests.

Copyright © 2006 Educational Testing Service Listening. Learning. Leading. 13 Read Aloud for Reading Tests Modification in most states (many states allow it and then do not aggregate test scores). Allowable accommodation on most admissions tests.

Copyright © 2006 Educational Testing Service Listening. Learning. Leading. 14 Calculator For Math Tests States are split on allowing calculators on math tests and some limit them only to non-calculation test sections. Most admissions tests allow calculator for ALL students.

Copyright © 2006 Educational Testing Service Listening. Learning. Leading. 15 Computer for Essay Allowable accommodation on most state assessments although spell check and grammar check are limited. Strictly regulated allowable accommodation on some admission tests due to score comparability between computer and paper essays. Method of assessing ALL students for other admissions tests.

Copyright © 2006 Educational Testing Service Listening. Learning. Leading. 16 Scribe for Essay Allowable on most state assessments but some states a very strict with regard to implementation (e.g., regulations on spelling every word and indicating all punctuation marks and capitalizations). Allowable accommodations on most admissions tests (varied guidelines on spelling and punctuation).

Copyright © 2006 Educational Testing Service Listening. Learning. Leading. 17 Reasons for Inconsistencies State Assessments are criterion referenced and Admissions tests are norm referenced. –Criterion-referenced tests compare performance to a set of standards. –Norm-referenced tests compare performance to the performance of other students.

Copyright © 2006 Educational Testing Service Listening. Learning. Leading. 18 Reasons for Inconsistencies State and Admissions tests measure different constructs. –Speed is usually not a construct on state assessments. –Basic skills (spelling, calculating and decoding print) are usually not constructs assessed on admissions tests.

Copyright © 2006 Educational Testing Service Listening. Learning. Leading. 19 Research on Accommodations The SAT is the most researched test with regard to: –Predictive validity of scores for students with disabilities. –The impact of extra time for students with and without disabilities. –The amount of time used by students with and without disabilities.

Copyright © 2006 Educational Testing Service Listening. Learning. Leading. 20 Documentation Requirements In most states any student with an IEP or 504 plan is allowed to use accommodations on state assessments. Most states also have policies on accommodations for English-language learners. In some cases accommodations are available to all students.

Copyright © 2006 Educational Testing Service Listening. Learning. Leading. 21 Documentation Requirements for Admissions Tests In some cases documentation must be submitted to receive accommodations on the SAT. Here are two examples: –the student has not used the accommodation for the past 4 months –the student requests a computer for the essay Nearly all graduate admission tests require the test taker to submit documentation of their disability and justification for the disability (e.g., psycho-educational evaluation not less than 5 years old).

Copyright © 2006 Educational Testing Service Listening. Learning. Leading. 22 Documentation Requirements for Admissions Tests (continued) Admission testing programs do not agree on what qualifies as a disability. –Some graduate and licensure testing programs strictly define major impairments that substantially limits one or more of a person's major life activities. –Some testing organizations have adopted conservative definitions of this ADA law and deny accommodations if the test taker is not substantially limited relative to the AVERAGE person. –Response to Intervention Model for diagnosing learning disabilities are currently not used in documentation for admissions testing.

Copyright © 2006 Educational Testing Service Listening. Learning. Leading. 23 College Board Funded Research Predictive Validity of Scores Psychometric Properties –Factor Analysis –Differential Item Functioning Flagging and Admissions Decisions Impact of Extra Time Computer Use for Essay* Reviews of State Accommodation Policy *for students without disabilities

Copyright © 2006 Educational Testing Service Listening. Learning. Leading. 24 Predictive Validity of Prior Versions of the SAT Three studies have examined the predictive validity of the SAT for students with disabilities: –Cahalan, Mandinach & Camara (2002). –Ragosta, M., Braun, H. & Kaplan, B. (1991). –Bennett, R.E., Rock, D.A. & Kaplan, B.A. (1987).

Copyright © 2006 Educational Testing Service Listening. Learning. Leading. 25 Results of Predictive Validity of Prior Versions of SAT SAT scores (with HSGPA) are a good predictor of college performance for students with disabilities. LD students who receive extra time on the SAT get slightly lower first year grades (FGPA) than predicted by the SAT (by -.12 on 4.00 scale). –When SAT and HSGPA are used to predict FGPA the LD/Extra time group performs as predicted.

Copyright © 2006 Educational Testing Service Listening. Learning. Leading. 26 Impact of Extra Time Bridgeman, Trapani & Curley (2003). Mandinach, Bridgeman, Cahalan- Laitusis & Trapani (2005). Cahalan-Laitusis, King, Cline & Bridgeman (2006).

Copyright © 2006 Educational Testing Service Listening. Learning. Leading. 27 Results Bridgeman et al Extra time benefits for students without disabilities are: –>30 points from time and a half on SAT-M –>10 points from time and a half on SAT-V –High scoring students benefited from extra time more than low scoring students did. –No increase in test scores for students scoring 400 or lower. –Racial/Gender differences did not change with extra time.

Copyright © 2006 Educational Testing Service Listening. Learning. Leading. 28 Results Mandinach et al % extra time with section breaks resulted in highest scores for both math and verbal. Extra time had greater impact on math than verbal. Extra time: –Benefited medium and high ability students (with and without disabilities). –Had no effect on low ability students without disabilities. –May hinder performance of lower ability students with disabilities (particularly without section breaks).

Copyright © 2006 Educational Testing Service Listening. Learning. Leading. 29 Results (continued) Mandinach et al Amount of extra time provided (rather than ability) is best predictor of how much time students use. Low ability LD/ADHD students had most variability in amount of time remaining for last subtest.

Copyright © 2006 Educational Testing Service Listening. Learning. Leading. 30 Results Cahalan-Laitusis 2006 LD/ADHD students took 8% longer than non-disabled students. –Ranging from 4-14% more time per section. ADHD only students generally did not take longer than non-disabled students. Critical Reading section was the only section where some students with reading-based LD were unable to complete the section.

Copyright © 2006 Educational Testing Service Listening. Learning. Leading. 31 Summary of Extra Time On average extra time helps more in Math than Verbal. On average medium and high ability students (with and without disabilities) benefit from extra time. On average low ability students without disabilities do NOT benefit from extra time. On average low ability students with LD and/or ADHD may perform worse with extra time and no section breaks.

Copyright © 2006 Educational Testing Service Listening. Learning. Leading. 32 Summary of Extra Time (continued) On average low ability students with LD/ADHD do not benefit from extra time (even with section breaks). On average LD students do use more time than students without disabilities. On average ADHD students do not use more time than students without disabilities. All of these findings are based on averages (some students will perform better and some worse).

Copyright © 2006 Educational Testing Service Listening. Learning. Leading. 33 Research Available – NCEO Accommodations Bibliography

Copyright © 2006 Educational Testing Service Listening. Learning. Leading. 34 Implications of Policy and Research for Students with Learning Disabilities

Copyright © 2006 Educational Testing Service Listening. Learning. Leading. 35 Understand Your Learning and Test-Taking Needs Research has shown that not all accommodations will improve test scores (and some may decrease test scores) so try taking practice tests with and without accommodations to see what works for you. Know what accommodations you need (both in the classroom and on tests) and WHY so you will be able to advocate for yourself in college.

Copyright © 2006 Educational Testing Service Listening. Learning. Leading. 36 Be Prepared Testing accommodations in K12 are far more liberal than those offered after K12 so consider which accommodations you really need versus those that you have become accustomed to. Make sure you leave high school with an updated psycho-educational evaluation. Understand that your documentation still may not qualify you for accommodations if your disability does not “substantially limit a major life activity.”