K-12 EDUCATION: States’ Test Security Policies and Procedures Varied (GAO R) Council of Chief State School Officers National Conference on Student Assessment June 20, 2013 National Harbor, Maryland For more information, contact Jamila Kennedy, (202) or Page 1

U.S. Government Accountability Office (GAO) Independent, nonpartisan agency of Congress Conducts policy analysis and evaluates federal agency operations, typically upon request from Congress GAO’s work focuses on questions of importance to the federal government and federal policymakers 14 mission teams in Washington, DC and 11 field offices Education, Workforce, and Income Security Page 2

GAO’s Prior Work on Statewide Assessments NO CHILD LEFT BEHIND ACT: Enhancements in the Department of Education’s Review Process Could Improve State Academic Assessments, GAO (Washington, D.C.: September 24, 2009). Key findings on assessment security * : States varied in their capacity to develop their state assessment programs and oversee assessment vendors. States faced challenges ensuring the validity and reliability of alternate assessments for students with disabilities. Several gaps existed in state assessment security policies that were not addressed in Education’s review process. GAO recommended that Education incorporate test security best practices into its peer review protocols. Education disagreed with this recommendation. Page 3 *These findings were based on responses to a Web-based survey of the 50 states and D.C., interviews and document reviews, and site visits to the state educational agency and selected school districts in Maryland, Rhode Island, South Dakota, and Texas.

Introduction Statewide tests (or assessments) are used to measure students’ performance in core subjects, hold schools accountable for student achievement, and make key decisions, such as targeting interventions to underperforming schools. Thus, it is critical that test results be valid and reliable. However, in recent years, reports of school officials cheating on statewide assessments have surfaced in districts across the country. Page 4

Introduction Effective test security policies and procedures, when properly implemented, can help prevent and detect cheating and other irregularities that can undermine the validity and reliability of state assessments. Because state assessments—which the U.S. Department of Education (Education) has supported with over $2 billion since 2002—serve as the basis for school accountability and allocation of resources for targeted interventions, we prepared this report under the authority of the Comptroller General to conduct work on GAO's initiative. Page 5

Objectives 1)To what extent do states’ policies and procedures include leading practices to prevent testing irregularities? 2)What oversight do states use to help ensure that districts and schools are followingtest security policies and procedures, andhow often was cheating by school officials identified as part of this oversight? 2)On what sources do states rely for information or assistance with test security issues and what additional assistance would be useful? Page 6

Scope and Methodology Designed and administered a web-based survey of testing administrators in the 50 states and the District of Columbia. Deployed from November 2012 to January 2013 Response rate = 100% Relied on CCSSO’s and ATP’s Operational Best Practices for Statewide Large-Scale Assessment Programs guide as a basis to design the survey.* Conducted four pretests, two site visits, reviewed documents and interviewed officials from Education, national education organizations and test publishing companies. Page 7 *The Council of Chief State School Officers and the Association of Test Publishers, Operational Best Practices for Statewide Large- Scale Assessment Programs (Washington, D.C.: 2010).

Scope and Methodology The overall purpose of the survey was to determine the extent to which states’ policies and procedures included leading practices in test security in the following areas*: 1.Security plans 2.Security training 3.Security breaches 4.Test administration 5.Protecting secure materials 6.Computer-based testing States do not need to include every leading practice to have an effective test security program. We also collected information on state oversight activities and reported instances of cheating by school officials. Page 8 * The survey did not examine state or local implementation of these test security policies.

Leading Practices Extent of leading practices included in test security policies and procedures varied Page 9 Source: GAO survey of state testing directors. All states reported having at least half of the 74 leading practices in the following areas—security plans, security training, security breaches, test administration and protecting secure materials. 50% (37) Percentage of leading practices (Total = 74) Number of states % to 74% (41-55) 75% to 89% (56-66) 90% or more (67-72)

Leading Practices – Test Security Plans Page 10 Number of states Number of leading practices (Total = 9) Fourteen states reported having all (9 of 9) of the leading practices in this category, which include having procedures for keeping facilities and materials secure, and methods for transferring hard copies of test materials. Source: GAO survey of state testing directors.

Leading Practices – Security Training Page 11 Number of states Number of leading practices (Total = 11) Twenty-two states reported having all (11 of 11) of the leading practices in this category, which include developing training materials and conducting security training. However, four states had none. Source: GAO survey of state testing directors.

Leading Practices – Security Breaches Page 12 Number of states Number of leading practices (Total = 6) Twenty-four states reported having all (6 of 6) of the leading practices for security breaches; three states reported not having any. Leading practices in this category include procedures to prevent coaching or altering test responses. Source: GAO survey of state testing directors.

Leading Practices – Test Administration Page 13 Number of states Number of leading practices (Total = 32) Most states reported having over half of the 32 leading practices in this category, which covers procedures for managing possible testing irregularities. Only some states had certain practices within this category. Source: GAO survey of state testing directors.

Leading Practices – Protecting Secure Materials Page 14 Number of states Number of leading practices (Total = 16) Seven states reported having all (16 of 16) of the leading practices in this category, which includes protecting test materials at all stages of distribution, receipt and return. Fewer than half of the states had two of these 16 practices. Source: GAO survey of state testing directors.

Leading Practices – Computer-based testing Page 15 Number of states Number of leading practices (Total = 14) Twenty-eight states reported administering computer-based tests. Of these, eight reported having all (14 of 14) of the leading practices in this category. An example of a leading practice for computer-based testing is clearly documenting the use of any software or supporting devices. Source: GAO survey of state testing directors.

Page 16 Source: GAO survey of state testing directors. *This figure includes states responding that they felt “somewhat vulnerable”; “moderately vulnerable”; “very vulnerable”; or “extremely vulnerable” to cheating. Despite having a variety of policies and procedures, many state officials reported feeling vulnerable to cheating at some point during the testing process. Leading Practices Regardless of security practices, states feel vulnerable to cheating Number of states When testing students in need of accommodations

State Oversight States used a variety of tools to oversee test security and identify cheating State officials reported using the following oversight tools to ensure districts follow test security policies and procedures: Analysis of student data – 38 states Monitoring visits – 36 states Remote monitoring – 22 states Audits of district policies and practices – 21 states Hiring an outside security firm – 8 states Page 17

State officials reported that, during school years and , they detected potential cheating as a result of the following state oversight activities. Page 18 State Oversight Note: Reporting mechanisms may include online or paper-based forms for reporting security breaches by district and school officials, parents, or students. HotlinesReportingMonitoringAuditing Source: GAO survey of state testing directors. States using the activity to detect reports of cheating States not using the activity to detect reports of cheating

State Oversight During school years and : 40 states reported allegations of cheating by school officials 33 states confirmed at least one instance of cheating by school officials 32 states reported canceling, invalidating, or nullifying test scores due to suspected or confirmed cheating by school officials Page 19

States’ Sources of Assistance States received assistance with test security, but seek additional support Page 20 Source: GAO survey of state testing directors. Figure 11: Primary sources of information states rely on for assistance with test security issues States reported receiving assistance with assessment security from several sources, but most frequently from testing contractors.

States’ Sources of Assistance Officials from the majority of states reported that it would be very or extremely useful if Education would: gather information on best practices in test security (35 states), and disseminate information on best practices (36 states). Other areas where state officials reported additional assistance would be very or extremely useful: opportunities for state collaboration about assessment security issues (35 states) assistance with drafting requests for proposals for testing vendors (17 states) Page 21

Education’s Test Security Initiatives Education has led initiatives aimed at collecting and sharing information on practices and policies to prevent, detect, and respond to testing irregularities, such as cheating. January 2012 – Education published a Request for Information in the Federal Register February 2012 – Education’s NCES sponsored a symposium on testing integrity February 2013 – Education released a report discussing practices and policies in four areas of testing integrity Page 22

Closing GAO did not make any recommendations to Education, but observed that: Additional guidance and oversight will be key to ensuring that appropriate policies and procedures are adopted by schools to address new vulnerabilities from computer-based tests; Without strong policies and procedures and robust state oversight, there is a higher risk that decisions based on test results may be faulty, and lead to damaging results, including failing to identify and provide resources for underperforming schools and students most in need of academic support. Page 23

QUESTIONS?? Thank you for listening! Please remember to complete the session evaluation. Jamila Jones Kennedy Senior Analyst, U.S. GAO Phone: Page 24