Lessons Learned in Improving Test Security for State Assessments John Fremer President, Caveon Consulting Services National Conference on Student Assessment Annual Meeting, Philadelphia June 21, 2016

Context for TILSA Test Security Project Test Security Issues in the Nation and States Lots (and growing)! A “perfect storm” rose up in 2011-2012 and has continued across the country into recent years Some factors for this storm Increasing stakes, especially for teacher accountability and evaluation Work of interstate common assessment consortia Increased attention by the press and news stories Instances discovered in several large school districts

Lessons Learned by States Prevention Detection Follow Up Investigations Summary

Good Prevention Practices Designing for Test Security Test Administration Monitoring Sophisticated Use of Data Forensics Analyses Training Materials for Schools Commitment to Continuous Improvement

Context for Detection Discussion Increased Use of Data Forensics Expanded Methods Focus on Fairness and Validity Increased Professional Association Attention Research Programs and Publications Development of New Detection Tools, e.g., “Trojan Horse” Items, Digital Watermarking

Key Detection Lessons Conservative Interpretations Development of Communications Programs Development of Data Forensics Use Plans before Conducting Analyses Need to Adapt to Change in Delivery Approach

Good Detection Practices (1 of 2) Florida’s Communication Program Regulations and Legislation in Mississippi

Good Detection Practices (2 of 2) Regular Use of Data Forensics Incorporating Data Forensics into Scoring and Reporting Cycle Using Multiple Methods Similarity Aberrance Gains Erasures/Answer Changes

Context for Follow Up Investigations Discussion Highly Publicized Investigations Admissions Testing Impersonations Cases, Atlanta Public Schools Growing Willingness to Give Major Weight to Data Use of External Investigators

Key Follow Up Investigation Lessons Development of Investigation Plans Recognition of Need for Fundamental Change in Approach to Investigations Features of Well-Managed Investigations

Summary/ Recommendations (1 of 2) Need for a Comprehensive Security Plan Address Security in Accommodations at All Stages Perform Multiple Data Forensics Analyses

Summary/ Recommendations (2 of 2) Address Threats Posed by Increased Technology Use Monitor the Internet and Social Media Develop Media Plan for All Levels of Testing Program Management

For More Information Contact Info Dr. John Fremer President, Caveon Consulting Services 215–805-3007 (cell) John.fremer@Caveon.com