N C E O National Center on Educational Outcomes In Search of Consistency: Guidelines for the Administration of Accommodations by Third Parties

N C E O National Center on Educational Outcomes Presenters Ann Clapper, National Center on Educational Outcomes Amanda Blount Morse, National Center on Educational Outcomes Janet Filbin, Colorado Department of Education Scott Trimble, Kentucky Department of Education

N C E O National Center on Educational Outcomes Session Overview 1.What are third parties? 2.Why are guidelines important? 3.What is the current status of state policies on third parties? 4.How are states addressing the issue of third parties? 5.What are key elements of the process and the product?

N C E O National Center on Educational Outcomes Who are we talking about? Reader Reads test directions, items, passages, and/or prompts orally to students who are unable to decode text visually. Scribe Writes down what a student dictates through speech, sign language, pointing or by using an assistive communication device. Sign Language Interpreter Translates test directions and/or items from spoken English into American Sign Language or other method of sign language. The sign language interpreter also allows students to respond to written questions in sign language.

N C E O National Center on Educational Outcomes The Importance of Guidelines Standardization is an essential feature of educational assessments designed to support comparisons among participants. Strict adherence to guidelines detailing instructions and procedures for the administration of accommodations is necessary to ensure that the test results reflect actual student learning. If accommodations are administered inconsistently the results will not be comparable across examinees.

N C E O National Center on Educational Outcomes The Number of States With Existing Guidelines ScribeReaderSign Language Interpreter 2620

Student Assessment Program Janet Filbin Colorado Department of Education (303)

The Colorado Student Assessment Program (CSAP) Legislated in 1993 through HB 1313 Criterion-referenced assessment originally designed to obtain a point in time picture of 3 rd -4 th -7 th and 10 th graders in the area of reading and writing 5 th -8 th -10 th grade students in math 8 th grade students in science Provides state, district and building level information about how students from year to year were progressing toward state standards Requires all students have opportunity to learn state standards

Accommodations are changes made to the assessment procedures in order to provide a student with access to information and an equal opportunity to demonstrate knowledge and skills without affecting the reliability or validity of the assessment. CSAP Accommodations

Eligibility for CSAP Accommodations Any student may receive an accommodation that is allowed on the assessment if that student has received the same accommodation during daily classroom instruction for at least three months prior to the assessment. Only accommodations the student requires for the specific content area may be selected. The possible effects on the student of allowing the accommodations must be considered. A student may be allowed more than one accommodation. A student who is an English Language Learner is entitled to the same instructional and assessment accommodations as his or her English-speaking peers.

Development of CSAP Accommodations Support from NCEO Analysis of existing large scale assessment accommodations Special Populations Advisory Cross CDE Representatives District Superintendents District Assessment Coordinators District Director’s or Specialized Services Parent Centers

Standard CSAP Accommodations extended time large print Braille extended time large print Braille use of number line use of manipulatives Reading and WritingMathematics reading/signing directions in the student’s native language use of scribe signing/pointing use of augmentative device reading of the writing prompts only reading/signing directions, word problems, designated answers to problems, and glossary words and definitions in the student’s native language use of scribe signing/pointing use of augmentative device oral presentation or word problems

Early Emerging Issues Discrepancies in the use of accommodations from grade level to grade level, district to district and across populations Variability in the approaches to administering accommodations Assumed “creative license” in applying non- standard, non-approved modification to testing situations Focus on compensatory skill development in the classroom for many students

Charge Ensure access of all populations Lack of previous large scale assessment history for ELL/special education Reduce the potential for misadministration Discrepancies between buildings/districts in use of accommodations Increase clarification about the nature of the accommodation, potential users, and administration guidelines Ongoing issue of accommodations vs. modifications Confusion regarding state vs. district assessment accommodations Low teacher expectations Provide direction to districts regarding proctor qualifications Certification/training of translators Issues with signing

Stakeholders Cross Unit Collaboration Assessment Title 1 English Language Acquisition Special Education Association of Colorado Education Evaluators (ACEE) CSAP Accommodations Expert Panel Review District-level Administrators Title I, ELL, Content, Special Education Teachers Building Principals CDE personnel Parents CSAP Bias/Content Review Committee

Dynamic Process Analysis of data student performance aggregated special populations participation rates use of accommodations special populations by district/school interviews with educators and administrators in the field Review of existing literature and research on accommodations (Tindal (1996); Tindal and Fuchs (1999);NCEO Studies-Thurlow, et.al. ( , 2001); Thompson, et.al. (2002), NAEP Validity Studies-Weston (2002), etc.) Examination of developed training materials

Recommendations Increased guidance around specific accommodations Scribe Oral/signed/translated presentation Technology Relevant training/materials Standard large print test Non-standard accommodation application process Language simplification

Implementation Process  Statewide CSAP Administration Training  Procedures Manual  Tools for Increasing the Performance of Students/CSAP Accommodations Video  Fast Facts  Website Guidelines For Documenting Student Need/Teacher or Proctor Training Considerations for Making Decisions

4 th Grade Reading 10 th Grade Reading

% of students Changes In The Use Of Reading Accommodations Over Time

Lessons Learned What you don’t know can hurt you The road to standardization should not be less traveled Just when you think your done, your not The TAO of inservice: Once trained may not retain

Colorado’s Accommodation Guidelines and Training Information Homepage for the Colorado Department of Education Procedures manual 4/proc_man.pdf Accommodations Video/training materials index.htm

Two Basic Principles All students can learn at high levels Accountability must be inclusive of all students, including those at risk of failure.

Define third-party providers of accommodations (who can provide the services) Third-party providers of accommodations are trained by local districts, are either employees of or volunteers in service to the local district, and are not students. These are typically the same persons who provide accommodations during the normal delivery of instruction.

Participation in Assessment Students with disabilities participate in state- required assessments: – With no accommodations, – With accommodations consistent with delivery of instruction, or – In the Alternate Portfolio Assessment.

Accountability Environment Kentucky’s accountability environment is considered: – High stakes for schools. – Not high stakes for students.

Why Accommodations? Kentucky’s program is intended to evaluate the effectiveness of the instructional delivery system. Students with legally identified disabilities through the ARC may have been provided special accommodations in order to access the general curriculum. In such cases, it is critical to assess such students consistent with the instructional accommodations provided for in the IEP.

Questions Why is it important to have guidelines for third parties? What prompted your state to write guidelines for third parties?

Process Describe the process that Kentucky followed to develop the guidelines for third party providers of accommodations: stakeholders collaboration research?

Questions How was this implemented? How is it working? Suggestions?

N C E O National Center on Educational Outcomes Development Process - Current Reviewed state guidelines Identified key elements (process and product) Drafted composite list of considerations Sent draft to 4 states and conducted telephone interviews Revised draft

N C E O National Center on Educational Outcomes Key Features - Process Stake-holder involvement Collaborative effort Continually evolving process

N C E O National Center on Educational Outcomes Key Features - Product Comprehensive Aligned Clear Specific Coherent Accessible

N C E O National Center on Educational Outcomes Major Components The need for guidelines (i.e., access, test validity, standardization) Discussion and clarification of relevant terms (e.g., scribe, reader, etc.) Qualifications and characteristics of “third party administrators” (e.g., certified staff members, be familiar with the student, can’t be the parent of the student receiving the accommodation, have completed training) Detailed directions and procedures for implementing specific accommodations (read aloud, dictation to scribe, sign language interpretation) during testing An explanation of inappropriate administration practices Implications for test security/confidentiality

N C E O National Center on Educational Outcomes Specific Considerations Are there guidelines for when and where third party test administrators can review test materials? Are there procedures for third party administrators to follow if they need clarification of test items or materials prior to test day? Are students trained on the use of a scribe, reader, and sign language interpreter during testing? Are there guidelines specific to the type of question? Are there guidelines specific to content areas? Are procedures defined for addressing third-parties’ questions or concerns that arise during testing?

N C E O National Center on Educational Outcomes Specific Considerations Are there requirements for how familiar the person providing the accommodation must be with the student? Are there suggestions to guide the pre-discussion between test-taker and third-party administrator? Are provisions for the settings in which accommodations can be administered described? Are there procedures that need to be followed in order to ensure that all test administrators followed the guidelines established to protect standardization and the validity of the test?

N C E O National Center on Educational Outcomes Resources Allman, C.B. (2003). Making tests accessible for students with visual impairments: A guide for test publishers, test developers, and state assessment personnel. Louisville, KY: American Printing House for the Blind. Educational Testing Service. (2004). ETS guidelines for a test reader. Retrieved June. 1, 2004, from ETS Web site: Educational Testing Service. (2004). ETS guidelines for a test writer recorder. Retrieved June. 1, 2004, from ETS Web site:

N C E O National Center on Educational Outcomes Development Process - Planned Conduct more interviews Revise and disseminate for additional comment Write brief and develop tool

N C E O National Center on Educational Outcomes Contact Information If you have comments or questions please contact: Amanda Blount Morse Ann Clapper

N C E O National Center on Educational Outcomes Visit: or Search for NCEO