Test Security Company Perspective Presented by: John Fremer, President, Caveon Consulting Services NCSA – June 2014

Overview of Presentation  Cannot Let Security For “Non-mainstream" Children Be An “Afterthought”  Our Goal is Meaningful Scores – We Have to Do Many Things Right At Every Stage  Prevention  Detection  Investigations  Special Challenges Of On-Line Assessment  Sharing What Is Known Is A Key Step

Test Security at Every Stage  Prevention & Planning  Communicate Zero Tolerance  Clear & Consistent Messages  Qualified Personnel – For All Accommodations  Training  Monitoring

Prevention & Planning (Cont.)  Test Security Manager  External Test Security Audit  Formal Test Security Standards  Security Vulnerabilities Identified  Recommendations for Correcting

DETECTION  Use Multiple Data Forensics Methods  Unusual Gains  Similarity  Erasures (Answer Changing)  Person Fit Analyses

DETECTION (cont.)  Adhere to Legal and Regulatory Requirements  Importance of “test user” agreement  Clarify potential consequences of not following testing rules  May want to seek changes to laws or regulations

TEST SECURITY INVESTIGATIONS  Areas Needing Attention  Level of evidence warranting an investigation  Roles for state, district, and other staff  Time lines  Model investigations kit

TEST SECURITY INVESTIGATIONS (cont.)  Focus Primarily on Improving Security in the Future  Follow “Innocent Until Proven Guilty” approach, but be firm in your questioning  Give most attention to the “Worst of the Worst”  Consider other possible explanations for results

SPECIAL CHALLENGES OF ON-LINE ASSESSMENT  Involvement of Readers or Scribes  Selection  Training  Additional Access to Test Materials  May Need to Reproduce Test Materials  Timing  Controls & Monitoring

SPECIAL CHALLENGES OF ON-LINE ASSESSMENT (Cont.)  Absence of Formal Policies  Lack of Experience  Availability Of Additional Presentation Features  Facilitates large type  System design may interfere with needed accommodations  Need experienced assistance

SHARING WHAT IS KNOWN  Among and Across States  Operational Best Practices  TILSA Guidebook  Possible Additional Material  Learning From Others  ETS, College Board, ACT  Certification Programs  NCEO

FOR MORE INFORMATION: John Fremer Caveon Consulting Services Caveon LLC (cell)