Test Security June 26, >

> 2009 Data Forensics Report July 2009  PA Department of Education Director of Bureau of Assessment and Accountability resigned  Had worked with contractor to produce a Data Forensics report  Report arrived just as Director was leaving  Newly appointed Director received the report but found it difficult to use  Simply didn’t have the time or staff to access every grade (3-8 and 11) in every school (3,182) in two content areas (reading and math) 2

> 2009 Data Forensics Report July 2011  Philadelphia Notebook inquires if Pennsylvania Department of Education (PDE) had ever conducted a Data Forensics Report  PDE turns over the report to the Notebook  Notebook discovers a summary at the end of the Technical Report  Publishes names of schools that were “flagged”  PDE forms a Data Forensics team 3

> PDE Data Forensics Team Investigations  Asked every Superintendent/CAO to investigate the cause for being flagged  Unusually high numbers of wrong to right erasures  Unusually high percentage increase in students Advanced/Proficient  Lots of negative responses from Superintendents/CAOs  Changes in number of subgroup populations that put the subgroups below the minimum N  Some flagged schools were so egregious they were turned over to the Office of the Inspector General for investigation 4

> PDE Data Forensics Team Investigations  PDE didn’t want to rush to judgment  Investigations still ongoing  PDE requested Data Forensics reports for 2010 and 2011  Reports have not been released because of ongoing investigations  In 2012, PDE requested only erasure analyses  Only concrete evidence of possible violations 5

> PDE Data Forensics Team Erasure Analyses  Schools were flagged if  At least 10% of the student population in a grade had five (5) or more wrong to right erasures in a particular content area  State average was around one wrong to right erasure per student  If a school met the 10% criteria, PDE asked the school for a list of the Test Administrators matched to the students  If a Test Administrator had 20% or more of his/her students with wrong to right erasures, the Superintendent/CAO was asked to investigate and report the results of the investigation to PDE 6

> PDE Data Forensics Team Preventive Measures  Change in training  Old training model  PDE trained District Assessment Coordinators  District Assessment Coordinators trained School Assessment Coordinators  School Assessment Coordinators trained Test Administrators  New training model  All of the above PLUS  All Test Administrators have to complete an online Test Administrators’ training 7

> PDE Data Forensics Team Other changes  Scrambled items  Cell phone and electronic devices policy  Suggested that children’s teacher not administer the assessment to his/her children  If not possible, have another person in the room (Proctor)  Excessive login report  Signed certificates  District Assessment Coordinator Test Security Certificate  School Assessment Coordinator Test Security Certificate  Principal Test Security Certificate  Test Administrator Security Certificate  Proctor Test Security Certificate  General Test Security Certificate  Student Code of Conduct 8

> PDE Data Forensics Team Additional Test Security Measures  Materials inventory  Materials must be kept in a locked location  Test distribution and return procedure  PDE Monitoring  Any suspected violations must be reported to PDE immediately  Materials Accountability Form  Missing materials report 9