Florida Test Security Measures Presented at CCSSO National Conference of Student Assessment National Harbor, Maryland June 2013

An Overview The Foundation The Directives Information and Guidance Forms and Agreements Detection Procedures Investigation

An Overview The Foundation Florida State Board of Education Rule 6A , FAC, was developed to meet the requirements of the Test Security Statute, s , F.S., and applies to anyone involved in the administration of FCAT, FCAT 2.0, and End-of-Course statewide assessments. The Foundation The Directives Information and Guidance Forms and Agreements Detection Procedures Investigation

An Overview Policy Directives Certified Test Administrators – Test administrators must be certified educators – Non-certified personnel may serve as proctors Seating Charts – Room name/number – Student names and their location in the room during testing Testing Rules Acknowledgement – Required to read and sign (or check a box) indicating the student understands the testing rules Electronic Devices Policy – No electronic devices The Foundation The Directives Information and Guidance Forms and Agreements Detection Procedures Investigation

An Overview Information and Guidance Data Forensics Memo – Provides information on the use of statistical analyses FDOE District Assessment Coordinator PowerPoint – Familiarizes districts with approach to data forensics 2011 Fall FCAT Retake Informational Webinar – Information provided to districts if student test scores are not released FCAT/FCAT2.0/EOC Invalidation Appeals Guide – Resource for conducting investigations and providing documentation for appeals Parent/Guardian Letters – One letter informs parents of important policies before testing – Another letter informs parents if their student’s test results were invalidated due to similarities The Foundation The Directives Information and Guidance Forms and Agreements Detection Procedures Investigation

An Overview Forms and Agreements Administration and Security Agreement – Training and consequences – Requires signature Test Administrator Prohibited Activities Agreement – Understand prohibited activities and consequences – Requires signature and certification number Test Materials Chain of Custody Form – Tracking test materials – Records names of people who handle/prepare secure materials The Foundation The Directives Information and Guidance Forms and Agreements Detection Procedures Investigation

An Overview Forms and Agreements Test Irregularities/Security Breaches – If a security breach is discovered, must contact the Bureau of K-12 Assessment – Specifics of the incident Security Log – Names and signatures of personnel assisting in testing activities Required Administration Information – Required information to be maintained by test administrators The Foundation The Directives Information and Guidance Forms and Agreements Detection Procedures Investigation

An Overview Detection Procedures School-Level Analyses – Improbable gains Predict score using prior year information Measure large score increases/decreases against predicted score Probable causes: Pre-Knowledge, Coaching, Student development – Similarity Measures degree of similarity between 2 or more test instances Analyze each test instance against all other test instances in the test across the state Probable causes: Answer copying, test coaching, collusion – Erasure indices Based on estimated answer changing rates: – Wrong-to-Right – Anything-to-Wrong Probable causes: tampering, “panic cheating” The Foundation The Directives Information and Guidance Forms and Agreements Detection Procedures Investigation

An Overview Detection Procedures Student-Level Analyses – Similarity analysis ONLY – Chances of tests being so similar, and taken independently = 1 in a trillion – Scores flagged at a very conservative threshold – If there is doubt that the reported score may not be valid, we cannot report it The Foundation The Directives Information and Guidance Forms and Agreements Detection Procedures Investigation

An Overview Investigation (school-level) Review list of flagged schools Retrieve and review answer documents from current and previous assessments DOE Office of Inspector General opens an investigation and sends notice to district Superintendents Districts required to conduct internal investigation of flagged schools Leadership review of investigation reports to determine schools for on-site investigations On-site investigation findings shared and reviewed by leadership Letter to districts outlining Department expectations of test security policies and procedures Updated school accountability decisions accordingly The Foundation The Directives Information and Guidance Forms and Agreements Detection Procedures Investigation

An Overview Investigation (student-level) Provide notice to districts of invalidated scores and appeals deadline Scores are invalidated until formal investigation is completed showing cause of the similarities School/District conducts preliminary investigation into the similarities Appeals guide is provided by DOE to guide districts with investigations Districts submit appeals, but appeals are not mandatory Review appeals and provide decisions to districts Update reporting files to release the results of successful appeals during late reporting rounds The Foundation The Directives Information and Guidance Forms and Agreements Detection Procedures Investigation

For more information about Florida’s Test Security Program, please contact: