Arizona Department of Education Transitioning from the Past into the Future Prevention, Detection, and Investigation Leila E. Williams, PhD Associate Superintendent for Assessment, Accountability, and Adult Education

Background Prior to 2015 – Paper Based Testing – Arizona’s Instrument to Measure Standards (AIMS) – Spike in incidents/irregularities reported (resulting in 7 investigations) – Superintendent’s concern

State Test Security Plan October 2014 initiation of a state plan Preparation – Identify the test being used for The new achievement test is AzMERIT. – Note security measures being employed by the test vendor. Test security information can be obtained from the AzMERIT BAFO. – Obtain the vision of the Superintendent/representative. – Create the Test Security Policy: Incorporate the Superintendent’s/representative’s vision. Develop the Test Security Policy using the appropriate ADE Guideline and Procedures (policy) format. The Guideline and Procedures document should include – – Requirements from Arizona Revised Statutes, Arizona Administrative Code, and Standards for Educational and Psychological Testing. – A Test Security Philosophy. – Governance (with and without a State Test Security Officer). – Procedures (in brief) that contain the Test Security Plan. – Obtain approval and adoption of the Test Security Policy. The State Test Security Guideline and Procedures (Policy) was approved/adopted on June 17, 2015.

Challenges Challenges to test security (detection, investigations, public awareness): – Using statistics to invalidate test scores (reporting problems to public). – Conducting investigations for evidence of misbehavior (resistance from within and without). – Providing information concerning the science behind detecting test security violations (acceptance of science behind techniques).

Audit Spring 2015 decided that it was critical to conduct an audit. In July 2015, Arizona Department of Education (ADE) made a formal request to have Caveon Test Security conduct an audit of their security practices. The purposes of the Test Security Audit are to: Identify any operational risks associated with the development, publication, storage, transfer, and administration of the state assessments in relation to Caveon Security Standards. Propose recommendations and actions to address identified security risks and to manage future security incidents.

Caveon Security Standards Security Plan Roles and Responsibilities Budget and Funding Standard Forms and Agreements Test and Item Design Test/Item Development and Maintenance Test Publication Test Administration Test Scores and Results Physical Security Information Security Web and Media monitoring Security Awareness and Training Security Incident Response Plan Conducting Security Investigations

Findings Strenghts – Management and Climate – Test Development – Test Administration

Recommendations Caveon provided recommendations for each of the Security Standards (High, Medium, Low). Highlight one of the High Recommendations: – ADE should develop a comprehensive Test Security Plan Manual or Handbook that brings together all policies and procedures related to test security for its high stakes assessments. It should address information security, the security of item/test development, publication, administration processes, as well as secure test design. The Manual/Handbook can be used to develop staff training, to coordinate action when security compromises occur, and to evaluate whether action taken followed the stated policies and procedures.

Test Security Manual Major Sections – Prevention – Detection – Investigations Resources (Appendix)

Next Steps Continue reviewing and adding to the Handbook—”It’s a Living Document!” Address areas recommended by audit Address issues related to consequences of test violations (i.e. policy from our administrations, AG office etc.) While we focus on being in prevention mode we do need to strengthen the investigation process.