1. Michigan Department of Education
New Requirements in ESSA and Peer review and States’ responses on test integrity and Security
National Conference on Student Assessment 2017
Pietro Semifero
Online Assessment Manager

2. Peer Review Critical Elements
2.4: Monitoring Test Administration
2.5: Test Security
2.6: Systems for Protecting Data Integrity and Privacy

3. Test Security Audit
March 2016 Caveon test security audit to identify operational risks and propose recommendations
Document review
Two days of on-site interview and inspection
Output: 31-page report on exemplary processes and recommendations for improvement
Document the old, develop the new

4. Test Security Audit Security Plan Roles and Responsibilities
Budget and Funding
Standard Forms and Agreements
Test and Item Design
Test and Item Development and Maintenance
Test Publication
Test Administration
Test Scores and Results
Physical Security
Information Security
Web and Media Monitoring
Security Awareness and Training
Security Incident Response Plan
Conducting Security Investigations

5. Test Security Audit Pre-Review Outside view, not “we know what we do”
Preparation
Evidence
Outside view, not “we know what we do”
Expands the range of the possible
You really have to make the improvements
Implementing best practices requires resources

6. 2.4 Assessment Monitoring
Random
Contract with Measurement Incorporated
Standard instructions, checklist
Covers all programs tied to state accountability
Targeted
Contracts with former State Police detectives
Standard instructions, checklist, “and look around a bit”
Sites selected based on previous year’s assessment incidents
50 sites per year. More random if fewer targeted.

7. 2.5 Test Security, 2.6 Test Material Integrity
PDIR: Prevention, Detection, Investigation, Remediation
State Test Security Handbook
Documented policies and procedures for:
Administration manuals, training
Chain of custody for paper materials
Secure online testing
Incident detection and reporting
Investigating prohibited behavior, misadministration
Consequences, contingency plans
Protecting the integrity of test materials

8. Assessment Integrity Guide
MDE publishes an “Assessment Integrity Guide” (AIG) that all test administrators must acknowledge reading before administering assessments
Goal of the AIG is to represent the expected professional conduct of educators who administer state assessments
With the advent of online testing, the AIG was extensively refreshed in 2015 with input by John Olson and John Fremer, finalized after the security audit
Reviewing for next round of updates, now that we have more experience with online testing

9. Assessment Integrity Guide
The Assessment Integrity Guide focuses on 4 major assessment integrity themes:
Prevention – standards and best practices for test integrity and security to prevent irregularities from occurring
Detection – guidelines for monitoring, reporting, and working with MDE when irregularities are found
Follow-Up Investigations – guidelines for investigating irregularities
Remediation – guidelines for schools to work with MDE to remediate issues and ensure valid results for all students

10. Assessment Integrity Guide
Useful to the field: best practices, establish expectations
Useful to the office: documented policies and procedures, manual to follow in the event of questions
Useful for review: documentation that is polished, reviewed, and approved by the State Board of Education

11. Test Security Specialist
Staff member dedicated to test security and integrity
Duties include:
Developing and updating test security policies
Developing and implementing test security training
Responding to incident reports and tips
Assessment monitoring, social media monitoring
Data forensic analysis
Investigating irregularities
Working with schools on remediation

12. Test Security Specialist Manual
Written by the Test Security Specialist: write down what it is that you actually do on a day-to-day basis
Practical, not theoretical
Reference guide for consistency
Training manual for turnover, backups
Identifies:
Duties
Procedures
Documentation
Contacts

13. Document the Old: Response Plans
Security incident response planning
Document practices for analysis and monitoring
Incident triage and investigation
Incident response matrix
Communications plan
Document everything

14. Document the Old: Formal Investigations
Defined plans for investigating more serious incidents
Define what rises to the level of an “incident”
Establish thresholds in advance rather than ad hoc
Investigation policies and procedures
Standard responses, consequences
Factors to consider in weighing appropriate remediation
Review panel for appeals

15. New Developments: Online Training
A series of four online modules through Michigan Virtual University
Walks through key topics from the Assessment Integrity Guide
For staff and schools, new and refresher
Focused on practitioners: preparation, administration, incident response (less on investigation and remediation)
Includes examples and quizzes

16. New Developments: Data Forensics
Ideal case of what data forensics we would like
Realistic case of what we have resources for
Now
Development plan
Division of labor between contractors and staff
Identify values that trigger investigations or invalidations
Identify and improve internal understanding of forensics
It is not “psychomagic”
Political support for data-based invalidations