2017 National Conference on Student Assessment Kathy Moore, Policy Advisor Kentucky Department of Education Office of Assessment and Accountability

Details—Details--Details Peer Review KY submitted documentation June 2016 USED letter received January 2017 KY responded February 2017 Details—Details--Details No further evidence was requested for the security element after the initial submission. KDE:OAA:ksm: 6/2017

Information was provided electronically and hardcopy Information was provided electronically and hardcopy. An extensive Table of Contents served to organize the large amount of information submitted. KDE:OAA:ksm: 6/2017

The level of detail provided on the summary was down to the page and section number of documents. A few questions were received on non-security pieces. Most were easily answered by simply referencing or clarifying the evidence submitted. Some required further documentation. KDE:OAA:ksm: 6/2017

Peer Review Critical Element 2.5 Prevention of Irregularities Regulations Regulation Trainings Face to face and videos Manuals Confidentiality Training Nondisclosure Signing Secure Materials Storage Test Certification Forms Security audit Detection of Irregularities Regulations Seating Charts Onsite Monitoring Forensic Data Analysis Reported Allegations Remediation of Incidents Regulations Board of Review Re-training Lowered Scores Licensure Allegation Incident History Record Investigate Irregularities Regulations Allegation Reporting Personal Interviews Documentation Seating Charts The 4 components of Critical Element 2.5 and how Kentucky addresses each. The Administration Code regulation drives our administration and security practices. KDE:OAA:ksm: 6/2017

Prevention of Irregularities Regulations Regulation Trainings Face to face and videos Manuals Confidentiality Training Nondisclosure Signing Secure Materials Storage Test Certification Forms Security audit KDE:OAA:ksm: 6/2017

Regulations KAR 703:080 Administration Code Anyone with the potential to be involved with testing must receive annual training. KAR 703:070 Inclusion of Special Populations Everyone involved in providing accommodations must be trained. Training materials are available online. http://education.ky.gov/AA/distsupp/Pages/Adm inCode.aspx Two regulations directly guide security around handling materials, administration practices, overall security. The Administration Code covers acceptable, unacceptable, and expected practices for test administration. The regulation applies to all state-required and state-administered assessments. The Inclusion of Special Populations covers the acceptable, unacceptable, and expected practices for providing accommodations during test administration. The regulation applies to all state-required and state-administered assessments. The training materials include PowerPoints, videos, directions for presenting the training, activity handouts. The materials are designed to be presented by District and/or Building Assessment Coordinators, but may also be used as a self-paced training. District Assessment Coordinators are responsible for ensuring that anyone with the potential to be involved with testing is trained prior to working with materials or administration. KDE:OAA:ksm: 6/2017

Regulations Standards for the regulation Professional Ethics test prep and student information Educational Defensibility students and schools right to valid and reliable test scores test prep activities Student Ownership work done entirely by student student honor code The Administration Code regulation was developed based on 3 standards which are included in the regulation. These sum up all that needs to be done for a successful administration. Following the details of the regulations and paying attention to details will lead to a clean administration without problems needing to be filed and investigated by the state. KDE:OAA:ksm: 6/2017

Regulations Office of Assessment and Accountability Emphasis Test Security—checks and processes to monitor appropriate administration of state-required tests including seating charts, room location codes and a student honor code. Data Quality—collaborative process between the Office of Assessment and Accountability (OAA) and school/d Throughout the year in a variety of ways, emails, face-to-face trainings, phone calls, etc., KDE emphasizes the need for correct data. Reminders about keeping the student information system up-to-date and how the data from the system is used to populate other systems used for accountability are often reiterated. Multiple opportunities to review the data related to the assessments has proven to be a highly successful means to more accurate final data leading to valid and reliable accountability results. KDE:OAA:ksm: 6/2017

Test Administration Manuals The first thing list in manuals is security. KDE:OAA:ksm: 6/2017

Confidentiality Training Prepared and conducted by schools Nondisclosure http://education.ky.gov/AA/distsupp/Pages/Forms.aspx Secure Materials Storage Secure at all times Limited access Double locked in classrooms Students may not transport Confidentiality is done at KDE as well as in schools. Signatures on nondisclosure forms are required by everyone dealing in any way with testing and at KDE. Some new construction in schools included secure test storage areas. KDE:OAA:ksm: 6/2017

Test Certification Forms (BAC and DAC) http://education.ky.gov/AA/distsupp/Pages/Forms.aspx Vendor Contractual Requirements General affidavit nondisclosure Data security and breach protocols Including KDE Memorandums of Understanding Storage and retention of test materials Signing the form indicates that testing went well without incident or that any problems were addressed and handled appropriately. KDE:OAA:ksm: 6/2017

Security Audit 12/12 13 KDE:OAA:ksm: 6/2017 Standard High Priority Dated Started Date Completed Notes Medium Priority Low Priority 13.) Security Awareness and Training Revise training materials in a timely manner to reflect changes in policies or procedures (pg. 21).   Currently done in conjunction with Administration Code and Inclusion of Special Populations trainings. Require field sites to adhere to set procedures for training to rectify variations currently in the field (pg. 21). 8/13 2/14 New langauge in administration code to require interactive training. Develop a new video to be used to supplement training (pg. 21). 11/12 New videos and materials will be developed once regs are official. Videos updated 2/14 The Kentucky Department of Education contracted with Caveon to conduct a department wide security audit. Caveon staff were on site to interview staff from the Office of Assessment and Accountability regarding the routine practices of not only the assessment office, but the security practices of the department regarding employees the physical location and handling of secure materials. A few District Assessment Coordinators and Building Assessment Coordinators were also interviewed. Findings and recommendations are categorized and rated high, middle, or low priority. A spreadsheet internal to KDE/OAA is set up in the same manner with the addition of dates of implementation and notes. 13 KDE:OAA:ksm: 6/2017

Detection of Irregularities Regulations Seating Charts Onsite Monitoring Forensic Data Analysis Reported Allegations KDE:OAA:ksm: 6/2017

Seating Charts Basic form provided on KDE webpage and e- mailed directly to DACs. http://education.ky.gov/AA/distsupp/Pages/ Forms.aspx Some vendors provide their version. We recommend to adapt the form so that what is submitted reflects the actual room arrangement and testing scenario. KDE:OAA:ksm: 6/2017

Onsite Monitoring--School Visits Randomly and purposely chosen Materials used during visit provided ahead of visit District Assessment Coordinator notified 30 minutes or less before arrival at school View process for handling materials and storage areas. View classrooms, but do not enter while testing is in session Schools at all levels, demographics, and geographic areas are included in the selection process. One staff member from the communications branch is paired with one staff member from the data branch. All consultants from the communications branch have teaching experience and many from the data branch have education connections—wife teaches, parents were administrators. KDE:OAA:ksm: 6/2017

Forensic Analysis Erasure analysis Individual review of student response booklets Student/staff interviews Primary vendor contracts with Caveon for erasure analysis. KDE is looking to improve understanding and use of the data. KDE:OAA:ksm: 6/2017

Allegations Reported to KDE by district/school staff, self-reported, students, parents, public Phone, e-mail, letter, or online at http://education.ky.gov/AA/distsupp/Pages/Allegation- Reporting.aspx Investigated and processed by KDE Allegations Coordinator Staff Attorneys Other KDE Staff from the legal office and commissioner’s office Can include data forensic analyses Allegations can be reported to KDE with full disclosure or anonymously. http://education.ky.gov/Pages/default.aspx# KDE staff investigates the allegation through the collection of information such as written notarized statements, interviews (adults and students), state-required testing documentation. The Board of Review members are all department staff, who may or may not have an education background. KDE:OAA:ksm: 6/2017

Investigate Irregularities Regulations Allegation Reporting Allegation Documentation Seating Charts Personal Interviews KDE:OAA:ksm: 6/2017

Allegation Documentation Steps to reporting an allegation http://education.ky.gov/AA/distsupp/Documen ts/2015- 16%20Steps%20to%20Follow%20When%20a% 20Testing%20Allegation%20or%20Irregularity% 20Occurs.pdf Forms http://education.ky.gov/AA/distsupp/Pages/All egation-Reporting.aspx Notarized statements are required. KDE:OAA:ksm: 6/2017

Personal Face To Face Interviews One or more district staff One or more school staff One or more students Depending on the severity of the incident, interviews may be done face-to-face on site or by phone. Testing is not delayed due to a student refusing to sign or a parent requesting that the student not sign the honor code. The signature serves as a reminder to students of the expectations during testing. KDE:OAA:ksm: 6/2017

Remediation of Incidents Regulations Board of Review Re-training Lowered Scores Licensure Allegation Incident History Record KDE:OAA:ksm: 6/2017

Board of Review Board of Review Appointed by the Commissioner—6 KDE staff Trained annually on Administration Code and Inclusion of Special Populations by Office of Assessment & Accountability staff Makes recommendation to Commissioner of Education Can result in score change for schools (not students) Can result in loss of teaching certificate for up to 2 years by the Educational Professional Standards Board (KRS 161.120) Not all members have teaching backgrounds, but all are KDE employees. The work and results is monitored by the Kentucky Legislative Research Commission. KDE:OAA:ksm: 6/2017

Retraining KDE offers KDE offers 6 regulation code trainings per year 2 November 2 December 2 January Districts/Schools provide Districts/schools are responsible for the initial training using the provided materials or adapting their own to best suit their needs. When addition training is recommended by the Board of Review the district/school has the option of providing the training or assigning the individual to attend a KDE provided session. KDE:OAA:ksm: 6/2017

Consequences Scores lowered Loss of teaching license Impacts school and district accountability Individual student not impacted Loss of teaching license Educational Professional Standards Board Recommendations for lowering of scores or teacher names to be sent to the Educational Professional Standards Board are made by the Board of Review. KDE:OAA:ksm: 6/2017

Incident History Allegation Example Inappropriate Assistance/Intervention by Staff Leading students to answers, making corrections to student work Test Security (Staff) Discussing item specifics Test Security (Student) Removing a test booklet from testing location Out of Order (Staff) Not following test schedule Out of Order (Student) Working in prior sections or ahead Special Education Giving inappropriate accommodations or not providing accommodations Student Action Using cell phones Missing Test Booklets Misplacing or failing to collect books by test administrator Other Giving feedback, allowing students to change answers after testing Allegations are categorized and the number of incidents reported are logged. Trends are monitored and emphasis is placed on problem areas during spring face to face meetings which target procedures for test administration. KDE:OAA:ksm: 6/2017

Kentucky Department of Education Office of Assessment and Accountability 300 Sower Blvd. Frankfort, KY 502-564-2256 Kathy Moore, Policy Advisor Kentucky Department of Education Office of Legal, Legislative and Communication Services 300 Sower Blvd. Frankfort, KY 502-564-4474 Patsy Kenner, Executive Staff Advisor/Allegations Coordinator http://education.ky.gov/Pages/default.aspx# KDE:OAA:ksm: 6/2017