1 Regents Scanning CFN Training Sessions Updated April 28, 2011

22 Objectives The purpose of today’s session is to EDUCATE you on the new Regents scanning procedures, and EMPOWER you to turn-key the process at the school level. After the training, you will understand: >Why the DOE is implementing the new Regents scanning process and the benefits of the changes; >The overall scanning process to be performed by schools, and step-by-step procedures for each stage; >How schools should plan for Regents exam week, and special situations to prepare for; >What documents and resources are available for training and support, including how to use the demo IT environment to train schools; and >Best practices for turn-keying the scanning process at schools.

33 Agenda >New SED Regents requirement & DOE scanning solution >Regents scanning A–Z  Process overview and timeline  Step-by-step review  Live demonstration >Turn-keying to schools  Lessons learned from the pilot program  Demo testing environment  Group activity: case study and discussion >NY SED Special Study – June 2011 Regents >Ongoing support and resources >Q & A

44 SED Regents Requirement & DOE Solution >Comprehensive English >Earth Science >Global Studies & Geography >Integrated Algebra >Living Environment >U.S. History & Government SED Requirement: Beginning in June 2011, the SED will require all districts to collect and submit item-level data for the following Regents exams:* DOE Solution: Leverage existing technology and systems currently used by schools to capture and report item-level data. New scanning process highlights include: >Answer documents: student-specific documents will now be generated and printed at school level >Scoring: teachers mark Constructive Response items only >Data capture: scan answer docs with attendance scanners >Reporting results: real-time results available in ATS immediately after scanning; automated data transfer from ATS to STARS *Other Regents exams & RCTs will use the traditional administration and scoring process.

55 SED Regents Requirement & DOE Solution >Time savings through automation of manual processes  Less overall time required for scoring  No further data entry requirements for aggregate scores >Increased accuracy in reporting test results  Elimination of manual scoring of MC questions  Section scores totaled automatically  Automatic application of conversion tables >Improved instruction  Item-level data available for use in informing instruction at student, teacher, school, and district levels >Audit trail  Answer document image-capture and line-item results will aid in special inquiries Benefits to schools under the new process:

Pre- Planning STARS ATS & Scanner ATS STARS (& Beyond) 1.Schools receive training and begin preparations for Regents Scanning process. 2. Schedule students for Regents and produce exam invitations in STARS. 3. STARS data flows to ATS. Schools test scanner and generate and print pre-slugged answer docs in advance. 6.Schools scan answer docs for upload to ATS. 7.ATS instantly applies scoring key & conversion chart for scale score. Results available immediately. 8.ATS passes data nightly to STARS for report cards and transcripts, and for reporting to NYSED. Exam Week & Scoring 5.Schools administer exams. Answer documents for walk-ins are printed on-demand. Scoring committee marks Constructive Response items and bubbles in scores on answer documents. Attendance Scanner Re-Scanning (As Needed) Regents Scanning A-Z: Process Overview 6 Logistics Planning Manual Score Verification 4.Schools conduct Regents week planning activities for logistics, including assignment of roles, and performing checklist items in the week before exams are administered. 7A. Score verification6A. Re-scan docs. Overnight Data Transfer Overnight Data Transfer

77 Timeline of Key Steps DatesKey Step ASAP ALs begin scheduling sessions to train schools on the new process May 1 – 31 Relevant personnel at schools assign roles, conduct planning, review materials, test scanner, etc. May 1 – 31 Schools clean and test scanner and ensure adequate supplies (toner, paper, etc.) By May 27Schools register majority of students for Regents in STARS June 1–10Schools print answer documents in ATS for most students June 8–14Regents week logistics planning & final preparation June 15–24Regents administration June 15–24Scanning and data validation

88 Step 1: STARS >Main changes from the traditional process:  No changes – the process is exactly the same. >Main tasks:  Register students for Regents exam, and generate exam invitations. >Task owner: typically school Program Chair >Best practices:  Complete registration process for majority of student by May 27.  Middle Schools: use proper exam codes in STARS for 8 th graders. If appropriate, notify the high school where students will take exam.  Only register students expected to take exams (though when in doubt, err in favor of registration).  Use “Master Schedule” function for creating a single section, and “Master Schedule Builder” for multiple sections. >Supporting materials: Regents Scanning Handbook, School Progress Reports available to Assessment Liaisons >Support: CFN Data & IT Liaison

99 Steps 2–3: Printing Answer Docs in ATS >Main changes from the traditional process:  Answer documents are no longer included in the Regents exam booklet to be torn out.  Pre-slugged student-specific answer documents are printed in advance, or generated on-demand at school level.  No generic answer documents may be used.  Answer docs are 1 page for Algebra & English; 2 for all others. >Main tasks:  Test scanner to ensure readiness, adequate supplies of toner and paper.  Print answer docs (for pre-registered students). >Task owner: typically Pupil Accounting Secretary >Supporting materials: Regents Scanning Handbook, School Progress Reports >Support: DIIT Help Desk,

10 Traditional & New Answer Documents Traditional Process Regents Scanning Individualized bar code Student MC answers CR teachers’ grades Alt. language & accommodations Pre-populated student info Scorers’ info

11 Steps 2–3: Printing Answer Docs in ATS >There are five options for printing answer docs: 1.“Not previously printed” (Batch printing for registered students) Prints batch of answer documents that have not been printed 2.“Print by student ID” (For DOE walk-in student) Prints an answer sheet for any student one at a time. This option may be used for another DOE student taking the exam at your school. (Use ATS SBIO screen to look up student ID.) 3.“Print by name (non-pub)” (For non-DOE walk-in student) Prints an answer sheet for a non-DOE student who is taking the exam at your school. 4.“Charter school option” (For schools not using STARS) Batch printing option for schools who are unable to register students in STARS. 5.“Reprint answer documents” Reprint previously printed docs by student ID or answer document page number.

12 Steps 2–3: Printing Answer Docs in ATS Sorting Options >Use RGPR screen for printing options 1, 2, and 3.

13 Steps 2–3: Printing Answer Docs in ATS >Best practices for printing answer documents  Answer documents must be printed on white paper only. Use the same stock paper as is normally used for attendance forms. Using colored paper or heavy stock may result in scanning errors.  Print answer docs for pre-registered students by June 10.  After printing batches of answer documents, a “test page” will print. Scan the test page to confirm ink quality for batch.  When printing, you may sort print jobs by teacher, section, or room number.  Ensure that adequate supplies of paper and toner are on- hand.

14 Step 4: Logistics Planning for Regents Week >Main changes from the traditional process:  Educate relevant staff members on new procedures and responsibilities for Regents scanning, and organize staff roles. >Main tasks:  Distribute test rosters and page one of answer docs to appropriate staff in advance.  Train exam proctors on new procedures and policies.  Ensure adequate supply of toner, paper, pens/pencils.  Establish process and task owners for dealing with walk-ins. (See slide “Instructions for Walk-Ins”)  Review Implementation Checklist >Best practices:  Roles and responsibilities are clear and communicated to all relevant staff before Regents Week.  Perform all checklist items. >Supporting materials: Regents Scanning – Implementation Checklist >Support: CFN Assessment Liaison

15 Step 5: Exam Week & Scoring >Main changes from the traditional process:  Answer docs for “walk-ins” printed on-demand on test day.  Students use pre-slugged answer documents; pencil for MC, pen for essays and DBQ questions.  Teachers mark Constructive Response items only using pencil. >Main tasks:  Generate answer documents for “walk-in” test-takers. (See next slide.)  Proctors administer exams according to best practices and emphasize new procedures (e.g. completely fill in bubbles, no stray marks, sign student declaration when finished). >Best practices:  Good preparation will lead to smooth exam administration >Support: CFN Assessment Liaison

16 Step 5: Instructions for “Walk-Ins” Schools should make preparations and assign roles to handle walk-in students during Regents week. StepDOE StudentNon-DOE Student Confirm student’s IDCheck photo ID School notificationCheck student record against notification sent from her home school Must present signed letter on school letterhead with exam specified Print answer doc in ATS“Print by student ID”* so that data flows to STARS “Print by name” for non-pub walk-in student Enter exam codeUse appropriate exam code *If Student ID is unknown, use ATS SBIO menu to look it up. >Inform appropriate staff how to direct walk-ins upon arrival, and appoint staff to print answer docs using ATS. >Print answer documents by following the below instructions: >Detailed instructions on handling walk-ins is included in the Handbook and FAQ.

17 Step 6 & 6A: Scanning Answer Documents >Main changes from the traditional process:  Scanning will capture item-level data and images of answer documents (and any ancillary re-scans) for audit trail.  Immediate results available in ATS; conversion chart automatically applied for instant score. >Main tasks:  After notification by DOE, scan docs using attendance scanner.  Docs with errors in reading bar codes will be reprinted with an error message. Docs with bubble read errors will be indicated on DFS confirmation sheet. Re-scan affected documents.  Review ATS reports for “Warnings” observed in Teacher section. >Task owner: staff members who process attendance scanning >Best practices:  Scanning tips available in Regents Scanning Handbook  Ensure that scanner has been cleaned in advance. >Support: DIIT Help Desk,

18 Step 6 & 6A: Scanning Tips >Ensure the scanner is clean; a dirty scanner may result in inaccurate results. See Tips for Scanning appendix for instructions on cleaning. >When scanning 2-page documents, it is recommended to batch all page 1 documents and scan separately from page 2 documents. >Schools may scan the first page of two-page documents immediately, before scoring is complete for page two. >If only one page of a 2-page document is scanned, it will be shown in ATS as an “Incomplete Scan” until the second page is processed. >Use the ATS Status Report to review warnings. >After scanning, a confirmation page will print indicating pages with scanning errors. The scanner will reprint pages for which the bar code could not be read. >File the scanned batches together with the DFS confirmation page. This will assist in retrieving papers as needed in the future. >Daily cutoff for overnight data transfer to STARS is 9pm on weekdays and 4pm on weekends.

19 Step 7: ATS Reports & Results >ATS reports available immediately after scanning:  Use Document Status Report to ensure that all answer docs have been properly scanned, and to flag warnings.  Use Item Report to see students’ scores and inform promotion status of individual students. Review report for “warnings/multiples/omits” by using “Include only W/M/O” option.  Use Item Distribution Report to inform decisions on present and future instruction. >Detailed instructions on how to access the ATS reports are available in the Regents Scanning Handbook. >Given the length and format of the ATS reports, they are best viewed when printed. You may not see all the data in the ATS screen. >Schools may download the ATS reports into Excel for further analysis.

20 Step 7: ATS Reports (REDS - Status Report) >Report uses:  Displays score, accommodations, warnings, for each student.  Shows summary of documents scanned, outstanding, and warnings.  Use report to track outstanding unscanned and incompletely scanned document. Accom. Scores Scan-specific info Document statuses

21 Step 7: ATS Reports (REDS – Item Report) >Report uses:  Displays line-item data for each student.  Shows specific questions to review (warnings).  Use to flag Teacher section questions to investigate (e.g. omits, multiples). TEACH SECSTUDENT SECTION Correct answers Multiple answers Incorrect answers CR raw scores

22 Step 7: ATS Reports (RADI – Item Dist. Report) >Report uses:  Shows school-level aggregated data for each question.  Use data to flag specific subject matter, question types, etc. for instruction. ‘*’ denotes correct answer

23 Step 7A: Re-Scoring & Score Verification >Main tasks:  AS OF MAY 24, 2011 – THERE IS NO RESCORING OF ANY REGENTS EXAM INCLUDING ALL MATH AND SCIENCE EXAMS with scaled scores of  Verification of scanned results for approximately 5% of answer docs.  Complete Manual Scoring Data Validation Form.  Consult the FAQ as Regents week approaches for updates on scoring procedures. >Best practices:  Two people at school level perform data verification >Supporting materials: Regents Scanning Handbook >Support: CFN Assessment Liaison

24 Step 8: Accessing Results in STARS >Main changes from the traditional process:  Data flows to STARS automatically; no manual data entry required.  Results will be automatically applied to transcripts and report cards from STARS. >Main tasks:  Provided scanning is completed by 9pm – on day after answer documents are scanned, schools may generate a STARS reports for all registered students who sat for Regents, regardless of test location. >Best practices:  Any future score changes (through re-scanning, DOE answer key alteration, etc.) will flow through to STARS automatically. >Support: CFN Data & IT Liaison

Live Demonstration

26 Turn-Keying: Demo Environment >The demo environment is available from May After May 31, the demo period will end and all demo data will be removed. >Detailed instructions on how to access and use the demo environment are available through a Demo Script. You will use your existing ATS username and password. >Demo scope: >We highly encourage schools to take advantage of the demo environment to gain familiarity with the new processes and to help ensure smooth Regents administration. A demo environment is available for training purposes. Schools are encouraged to use this to prepare for Regents Week and gain an understanding of the required processes. DEMO STARS (Scheduling) ATS (Printing) ATS (Scanning & Reporting) STARS (Report Card & Transcripts)

27 Turn-Keying: Lessons Learned from Pilot >Schools with advanced preparation performed well during January Regents  Preparations include: communication of new processes and policies, defined staff roles, on-time scheduling/printing of answer documents, prep for “walk-in” students, etc. >Many schools needed additional support and training around the technical systems.  Many schools did not utilize ATS reports (Document Status Report, Item Report, Item Distribution Report) for scanning validation and instructional benefits. The demo environment is intended to assist with this for the June implementation. >Scanning technology was validated through the pilot. DOE conducted a pilot program in January to administer the new process at select schools. The most valuable lessons learned from the pilot are:

28 Turn-Keying: Case Study & Discussion >Review of Regents Scanning Implementation Checklist. >Case Study exercise. >Discussion: in groups of 3-5, discuss the following questions:  How do you plan to effectively turn-key this new initiative to schools?  Who will you include in the school-level training?  What challenges do you anticipate?  What items need to be emphasized or modified for different size schools within your network?  How will you support your schools in using the new item- level data that is produced from this process?  What questions do you have about the process itself or its implementation? (Record for Q&A)

29 Turn-Keying: Roles & Responsibilities Assessment LiaisonsData IT Liaisons Overall point of contact for schools for Regents Scanning initiative Ensure schools and network staff have appropriate access levels in ATS and STARS Responsible for coordinating and delivering training to relevant school staff within network Support schools with questions on printing answer documents. Accompany AL during school questions. Ensure schools are properly equipped for Regents scanning (e.g. paper, toner, printing capability, STARS scheduling) Support schools in scheduling students for Regents in STARS and with all technical questions relating to ATS Verify schools have performed data verification of students exams Provide any additional training to schools on ATS or STARS, as requested Be on-hand during Regents week to provide support as needed Monitor STARS and ATS before & during test administration for data integrity issues and general activity. Be on hand for support during Regents week. Escalate any unanswered policy or process questions/issues Escalate any technical policy or process questions/issues

30 Turn-Keying: Network & School Checklist Main Action ItemsTimeline CFNs coordinate trainings with schoolsMay 1-31 Schools notify and train relevant staff, and make preliminary preparations (check toner, paper supplies) May 1-31 Schools utilize demo environment to run mock exam processMay 2-31 Schools schedule students for Regents in STARSBy May 28 Schools print answer documents for majority of studentsBy June 10 Schools establish & communicate Regents week plan (roles, logistics)June 1-14 Schools administer exams, scan documents, conduct data validationJune *Refer to Implementation Checklist for a complete list of tasks to be performed by schools.

31 SED Special Study – June 2011 Regents >Affected Regents exams:  Comprehensive English  Algebra 2/Trigonometry >Affected schools:  See NY SED Special Study for June 2011 Regents – Affected Schools handout >Process highlights:  The special study process will supersede any NYC DOE administrative process – including Regents Scanning for the Comprehensive English exam.  CFNs will receive instructions on how affected schools must administer these exams. The NYS Education Department is conducting a state-wide special study which will affect select schools for two June 2011 Regents.

32 On-Going Support & Resources >Documentation and updates:  ARIS: >Technical support (STARS & ATS):  CFN Data & IT Liaison >Scanner and printer:  DIIT Help Desk, >Overall process support:  Borough Assessment Implementation Directors (BAID) >Regents week emergency inquiries:  Patrick Purcell  Marc Bush  Kirby Fields The following resources are available to support implementation and execution of Regents scanning. *Please note that guidance from DOE on unresolved items will be issued on a rolling basis.