DATAG The Importance of Good CBT Data December 7, 2018 DRAFT

Glossary of Terms Acronym /Initialism Definition BEDS Code Code: Basic Education Data System Code (SED Education Organizational Code) BOCES Boards of Cooperative Educational Services CBT Computer-Based Testing ELA English Language Arts LEA Local Educational Agency L2 Level 2 State Data Warehouse (WNYRIC) NYSSIS ID New York State Student Identification System Identifier OODP Out of District Placement Students OSA Office of State Assessment SAFT Stand-Alone Field Testing DRAFT

Agenda CBT Overview CBT Data Requirements CBT Systems CBT Data Problems & Solutions Ordering CBT Organizing CBT NYSSIS IDs Not Tested Codes CBT and OODP DRAFT

Computer-Based Testing, NY Year 1 2015-16 All schools given the option to administer stand-alone field tests on computers 902 schools administered field tests on computer More than 60,000 students took the field tests on computer Year 2 2016-17 All schools given the option to administer operational tests with CBT in any subject, any grade level 184 schools administered operational tests with CBT More than 28,000 students took the operational tests on computer 819 schools administered field tests on computer Year 3 2017-18 More than 650 schools administered operational tests with CBT More than 145,000 students took the operational tests on computer More than 1000 schools administered field tests on computer DRAFT

CBT Schools by Region RIC/Region 2017 CBT Schools 2018 NERIC 31 144 CNYRIC 53 110 WNYRIC 27 85 MHRIC 5 58 EduTech/Wayne Finger Lakes 12 47 Monroe/MAARS 6 43 SCRIC 19 41 Eastern Suffolk 15 32 Big 5 NYC, Rochester, Syracuse, Yonkers 4 49 Nassau 20 LHRIC 17 MORIC 1 9 Greater Southern Tier (GST) 2 7 We work closely with our RIC colleagues about CBT data issues: RIC Working Group established in 2017 Our RICS are our first line of CBT Support in the field

CBT Data Timeline DRAFT

CBT Data Requirements Data requirements for the 2019 CBT Administration: https://cbtsupport.nysed.gov/hc/en-us/articles/360020403871-New-11-29-18-CBT-Data-Requirements-Grades-3-8-ELA-and-Math DRAFT

CBT Data Issues We’ve already tackled a lot of CBT data issues and know the “importance of being good data”; bad data creates issues such as the following: Student data cannot be loaded correctly to Nextera Admin for CBT administration Student scores data does not flow to the district of accountability We’ve developed solutions for myriad CBT data issues, but there is always more that we can learn and do. “I have the simplest tastes. I am always satisfied with the best.” - Oscar Wilde DRAFT

CBT Systems Nextera Admin Nextera Test Delivery System Online test administration system Multi-tiered, role-based system Manage students and student-test taking Nextera Test Delivery System Secure Browser Students access and complete tests Downloaded to students’ devices Additional practice with previously released state question New York Question Sampler: https://ny.nextera.questarai.com/tds/#practice Nextera Admin Video Tutorial: https://cbtsupport.nysed.gov/hc/en-us/articles/115001411643-Video-Learn-about-Nextera-Admin-features-functionality DRAFT

Ordering CBT: Problem Scenario: school orders tests under the wrong BEDS Code: High School orders all tests for the district with multiple schools in the district High School orders Grade 3 tests for the Elementary School We identify students to populate Nextera Admin for CBT administration by the BEDS Code + Grade Level Enrollment Outcome: L2 cannot locate any Grade 3 student enrollment records associated to the High School BEDS Code Students cannot be loaded to Nextera Admin for CBT Administration DRAFT

Ordering CBT: Solution We added validation checks in two places: CBT Team reviews CBT order reports daily to identify schools that have ordered outside of the grade level We contact principals to advise them to adjust their test orders In December, L2 sends reports to the CBT Team to distribute to the RICs that identify schools with missing student enrollment records for a grade level The RICs contact the districts and schools to upload missing student records DRAFT

Organizing CBT: Problem In the first year of operational CBT, we loaded students to Nextera Admin according to their instructional classes: SIRS Staff_Student_Course template Loaded students into Nextera Admin according to ELA and Math class enrollment Loaded all ELA and Math teachers: Could only load one teacher per class If a class had more than one teacher, the teacher with the “lowest” TEACH ID number was loaded Primary Teachers were often missing from the data load Teachers who were not involved with CBT were loaded to Nextera Admin Related Article on CBT Support: https://cbtsupport.nysed.gov/hc/en-us/articles/360001366872-Why-aren-t-my-students-available-in-Nextera-Admin-according-to-their-instructional-classes- DRAFT

Organizing CBT: Solution In the second year of operational CBT, our RICs and Districts suggested we load students into Grade-Level Enrollment “containers” in Nextera Admin: Created phony classes Created phony teachers No teachers were loaded Nextera Admin teacher accounts must be created by the district or school DRAFT

NYSSIS IDs: Problem(s) The NYSSIS ID is the student’s User ID for logging into the CBT exam and is created for a student when a student is first enrolled in a New York State (NYS) district/school: How can a student who moves into NYS a few days or during the CBT administration window log into the CBT exam without a NYSSIS ID? Students can be manually entered into Nextera Admin after the Pre-ID student data load, which allows for the potential of human error when typing a NYSSIS ID to create a student record. DRAFT

NYSSIS IDs: Solution Fast-Track NYSSIS ID Ahead of the first CBT administration (2017), we developed a process for districts to immediately create a placeholder NYSSIS ID for allowing a student to log into the CBT exam: Can be created within minutes Only available to CBT schools Only available one week before and during the operational CBT windows All Fast-Track NYSSIS IDs begin with “9999” The district must still follow the standard process to have a NYSSIS ID created for the new-to-NYS student The district must replace the Fast-Track NYSSIS ID with the student’s permanent NYSSIS ID in Nextera Admin Fast-Track NYSSIS ID DRAFT

NYSSIS IDs: Solution DRAFT

Not Tested Codes: Problem It is the responsibility of the school to mark students in Nextera Admin with Not Tested Codes when testing on computer: Can be completed at a “later date” than at the moment the students are testing (potentially different workflow within the school from print answer sheet workflow) A missing Not Tested Code, such as Medically excused, can result in a student’s score not being counted properly DRAFT

Not Tested Codes: Solution We remind our schools to verify student Not Tested Codes are set before the data is pulled for scores, and we ask the RICs to help with these reminders. Summer Cleanup remains an option Schools should NEVER use a print answer sheet when a student is testing on computer or refusing a CBT test session. DRAFT

OODP: Problem Solution Working through issues related to Out of District Placement (OODP) student populations and data flow is an example where NYSED, L2, and the RICs worked very closely together to develop a working solution: Added new data fields to the Nextera Admin student data population file layout Developed new workflows for ordering CBT Developed new workflows for scoring CBT Flexibility for BOCES with Virtual Locations DRAFT

OODP Student Data We load students based on enrollment + demographic data, identifying students with the following three data elements: Home District (“district of accountability”) Testing District Testing Location (School) BOCES program schools roll up to the primary BOCES: L2 data warehouse Nextera Admin DRAFT

OODP Student Data Flow L2 is already deriving OODP status for your students. Identification: When the first 8 digits of the LEA District BEDS Code does NOT match the first 8 Digits of the Building BEDS Code (and building BEDS does not end in“0888” or “0777”) THEN L2 updates the last 4 digits of the Building’s BEDS Code to “OODP.” This is internal to L2. OODP = Student not identified as OODP DRAFT

OODP Student Data Flow OODP Students load to Nextera Admin under the testing district and testing location so that the OODP school has the ability to administer the computer-based tests to the student. DRAFT

OODP & CBT Ordering: The testing location of the student is the location responsible for ordering the test. Scoring: The testing location of the student is the location responsible for scoring the test. DRAFT

OODP & Scores Scores flow back to the district of responsibility. Current processes in place for paper OODP student scores should be followed for CBT student scores. As a result of the student data elements, district of accountability, etc., the scores flow back to the district and not to the BOCES or “testing location.” DRAFT

District Moves We had a lot of district moves last spring during the CBT administration. The best way to handle student district moves remains an area to improve for CBT with the following known issues that can result in complications for scores data flow : Not editing a student Local ID once a student is manually moved in Nextera Admin No data fields in Nextera Admin for the additional OODP home district information Not exiting a student at the previous district And more …. DRAFT

CBT Listerv Subscribe to the CBT Listserv for weekly messages about what’s new with CBT! Send an email to cbtsupport@nysed.gov: First Name / Last Name Email Address Name of Educational Organization BEDS Code DRAFT

Questions about CBT Transition? Contact CBT Customer Support CBTSupport@nysed.gov https://CBTSupport.nysed.gov