1. Slide 1 National Center on Educational Outcomes (NCEO) Balancing Test Security and Accessibility on Next Generation Online Assessments Martha Thurlow Sheryl Lazarus NCEO NCEO John Fremer Shelley Loving-Ryder Caveon Virginia Department of Education National Conference on Student Assessment New Orleans, Louisiana June 25, 2014, 2013

2. Slide 2 Overview I.The issues II.Standards and best practices documents that address cheating or test security and accessibility III.States’ test security policies IV.Test security company perspective V.State perspective VI.Summary and discussion VII.Questions National Center on Educational Outcomes (NCEO)

3. Slide 3 I. The Issues National Center on Educational Outcomes (NCEO) Test security policies and procedures need to address accommodations and other issues related to students with disabilities to help ensure the integrity and validity of a test. For example, some accommodations that individuals with disabilities and English language learners may need to meaningfully access a test: - Require that someone other than the student sees the test. - Require the use of specialized accessibility software. - Require individual or small group individual administration. Alternate assessments may have some unique test security issues.

4. Slide 4 Example: Accessibility Software National Center on Educational Outcomes (NCEO) Issue— Some students with disabilities use accessibility software during instruction. Computer-based tests sometimes require that the accessibility software be disabled (though there may be test-based accessibility features). Result— Student is denied meaningful access to assessment.

5. Slide 5 II. Standards and Guidelines that Address Cheating or Test Security Standards for Educational and Psychology Testing (APA/AERA/NCME, 1999)—Revised version will be available soon. Operational Best Practices for Statewide Large-Scale Assessment Programs (ATP/CCSSO, 2013) Testing and Data Integrity in the Assessment of Student Achievement (NCME, 2012) Other relevant documents National Center on Educational Outcomes (NCEO)

6. Slide 6 Standards for Educational and Psychology Testing The Standards indicate that: “all examinees be given a comparable opportunity to demonstrate their standing on the construct(s) the test is intended to measure. Just treatment also includes such factors as the appropriate testing conditions and equal opportunity to become familiar with the test format, practice materials, and so forth. Fairness also requires that all examinees be afforded appropriate testing conditions.” (p. 74) National Center on Educational Outcomes (NCEO)

7. Slide 7 Standards for Educational and Psychology Testing The Standards indicate that: “test users have the responsibility of protecting the security of test materials at all times.” National Center on Educational Outcomes (NCEO)

8. Slide 8 Operational Best Practices for Statewide Large-Scale Assessment Programs National Center on Educational Outcomes (NCEO) The Operational Best Practices indicate that the document addresses: “best practices for achieving test security throughout the process of all paper-based and technology-based assessments, using methods that support the needs of students with disabilities and English learners.” (p. 81)

9. Slide 9 Operational Best Practices for Statewide Large-Scale Assessment Programs, Cont. National Center on Educational Outcomes (NCEO) Chapter 19. Assessment of Special Populations/ Accessibility for All Learners Universal Design Accommodations Alternate Assessments

10. Slide 10 Testing and Data Integrity in the Administration of Statewide Student Assessment Programs (NCME) National Center on Educational Outcomes (NCEO) Policies and procedures must ensure that all students have appropriate, fair, and equal opportunities to show their knowledge, skills, and abilities. Students who need accommodations due to language differences or students with disabilities may require appropriate modifications to materials and administrative procedures to ensure fair access to the assessment of their skills.

11. Slide 11 Other Relevant Documents National Center on Educational Outcomes (NCEO) TILSA Test Security Guidebook (Olson & Fremer, 2013) Testing Integrity Symposium: Issues and Recommendations for Best Practice (NCES, 2013)

12. .. Analysis of States’ Test Security Policies Source: Test security and students with disabilities: An analysis of states’ 2013-14 test security policies (Synthesis Report 95). National Center on Educational Outcomes. Available at: http://www.cehd.umn.edu/nceo/OnlinePubs/Synthesis95/SynthesisReport95.pdf

13. Slide 13 What information do states’ test security policies contain about students with disabilities and the assessments they take? How are accommodations included in states’ test security policies? National Center on Educational Outcomes (NCEO) Research Questions

14. Slide 14 State policies downloaded for the 50 states + DC Documents analyzed included: Test procedures or administration manuals, general test security documents, accommodations manuals. Analyzed documents for both the regular assessment and the alternate assessment. National Center on Educational Outcomes (NCEO) Procedures

15. Number of States’ Test Security Policies That Addressed Selected Issues.

16. Slide 16 Duplication and Copying of Test Materials. Allowed ‘School staff members may reproduce secure test materials, in whole or in part, if this is necessary to provide a particular test accommodation for a student with an Individualized Education Program (IEP) or a Section 504 Plan.” Not Allowed “It is a violation of [test] security to: Copy, reproduce, or use in any manner any portion of any secure assessment book, for any reason. Student is allowed to use assistive technology to magnify/enlarge. The actual test booklet may not be copied or enlarged.” State Examples

17. Slide 17 Duplication and Copying of Test Materials, cont. National Center on Educational Outcomes (NCEO) “One week prior to the opening of the test administration window, the materials may be printed and then photocopied from a master copy for individual student use.” State Example: Alternate Assessment Based on Alternate Achievement Standards (AA-AAS)

18. Slide 18 Role of IEP National Center on Educational Outcomes (NCEO) “Teachers and other staff members may give accommodations or modifications that are consistent with the student’s IEP, 504, or LEP Plan and that are used in the normal delivery of instruction with the exception of those indicated as not allowable in the Accommodations Guidelines.” State Example

19. Slide 19 Training Requirements National Center on Educational Outcomes (NCEO) “Ensure all personnel involved with testing receive training on appropriate test administration, policies, and procedures including accommodations.” “Assessment proctors who are in ‘accommodated sessions’ should receive training on the appropriate way to administer that specific assessment accommodation. Test proctors who administer the assessment to students with multiple accommodations must be trained on all applicable accommodations.” State Examples

20. Slide 20 Qualifications National Center on Educational Outcomes (NCEO) “Test administrators shall be licensed personnel.” “Any person providing an accommodation must be an adult non-relative of the student. Students may not serve as scribes for other students, even if they are older students.” State Examples

21. Slide 21 Security of Test Materials National Center on Educational Outcomes (NCEO) “Secure test materials include: all customized administration materials including Braille, sign language, large-print, and loose- leaf test booklets, oral administration scripts (OASs), audio CD- ROMS, signed administration scripts (SASs), sign language DVDs, Braille Test Administrator’s Notes (TA Notes), and Braille OASs. Secure materials must be kept in secure, locked storage when not in use.” State Example

22. Test Security & Alternate Assessments 27 States have test security policies that specifically addressed alternate assessments.

23. Slide 23 Alternate Assessments, cont. National Center on Educational Outcomes (NCEO) State Examples Item-based performance tasks: “Test Books and student Answer Documents must be kept secure. Students must not be exposed to test content before the actual testing. If students have prior knowledge of test content, results of testing can give a deceptive picture.” Performance tasks: “Tasks are secure materials and should not be sent home with the student.”

24. Slide 24 Alternate Assessment Test Security, cont. National Center on Educational Outcomes (NCEO) State Examples Portfolios: “Once the portfolios contain student identifying information, student testing materials, and student work, the portfolios become secure documents and must be treated with the same care as other secure testing materials, keeping them in a locked file cabinet.” Checklist: Checklist is not a secure document. This means that the checklist may be copied.

25. Accommodations Most Frequently Addressed in States’ Test Security Policies.

26. Slide 26 Braille. State Example “Under the supervision of the District or School Assessment Coordinator, a Test Monitor administering a Braille test may access the Test Administrator Notes and the corresponding test books up to 48 hours prior to the scheduled administration.” National Center on Educational Outcomes (NCEO)

27. Slide 27 Sign Interpretation National Center on Educational Outcomes (NCEO) State Examples ‘Sign language presentation may be provided for all directions and test items and answer choices on mathematics and science tests. Test items may not be reworded, summarized, or simplified. When providing answer choices, the test administrator must be careful not to use language that might lead a student to the correct/incorrect responses.” “A student’s teacher should not serve as the interpreter/translator in a testing situation unless a second person is present to monitor for quality and fairness.”

28. Oral Delivery Accommodations.

29. Slide 29 Human Reader National Center on Educational Outcomes (NCEO) “Test administrators conducting an oral administration must be aware that they are viewing secure content and that responding to test questions, recording the information they see, scoring the test, or discussing the content of the test at any time is strictly prohibited. As a reminder of this obligation, individuals who give an oral administration of a test are required to sign a separate section of the test administrator’s oath.” “Test administrator must read in a pace and tone that is appropriate for each individual student. Careful attention must be given such that no changes in tone or inflection are detectable which might indicate a correct answer.” State Examples

30. Slide 30 Text to Speech National Center on Educational Outcomes (NCEO) “If a text-to-speech program is being utilized, I have reviewed the material to ensure the reading passages will not be read aloud, and the content of the test is pronounced appropriately and the test items are not read in a manner to give away an answer.”. State Example

31. Scheduling/Timing Accommodations.

32. Setting Accommodations.

33. Slide 33 State Approaches to Test Security National Center on Educational Outcomes (NCEO) Good Practice Test security guidelines that address accommodations and alternate assessments. Test security agreements address accommodations. See questions in report for list of things states and consortia may want to consider when balancing test security and accessibility.

34. Slide 34 National Center on Educational Outcomes (NCEO) For More Information National Center on Educational Outcomes www.nceo.info Sheryl Lazarus laza0019@umn.edu Martha Thurlow thurl001@umn.edu

35. Slide 35 National Center on Educational Outcomes (NCEO) References American Education al Research Association (AERA), American Psychological Association (APA), National Council on measurement in Education (NCME). (1999). Standards for educational and psychological testing. Washington, DC: Author. Association of Test Publishers (ATP)/ Council of Chief State School Officers (CCSSO) (2013). Operational best practices for statewide large-scale assessment programs. Washington, DC: Author. National Center for Education Statistics (NCES). (2013). Testing integrity symposium: Issues and recommendations for best practice. Washington DC: Institute of Education Sciences (IES). Washington DC: Institute of Education Sciences (IES). Retrieved from http://nces.ed.gov/pubsearch/pubsinfo.asp?pubid=2013454 National Council on Measurement in Education (NCME) (2012). Testing and data integrity in the administration of statewide student assessment programs. Madison WI: author. Olson, J.F. & Fremer, J. (2013). TILSA test security guidebook. Washington, DC: Council of Chief State School Officers (CCSSO).