1. Oral Exam Rating Process May 23, 2016 NATIONAL CENTER FOR STATE COURTS

2. 2 | Page Reminders:  Registration lists to NCSC at least 21 business days in advance of testing  Register candidates with name on government-issued ID (driver’s license, passport, etc.) Preparation for the Oral Exam

3. 3 | Page  Each NCSC Oral Exam must be rated by two NCSC-approved raters  NCSC-approved raters are listed on the NCSC Rater Expert List Oral Exam Rating Process: Overview

4. 4 | Page Oral Exam Rating Process: Overview

5. 5 | Page  Two raters rate each oral exam  At least one rater must be an Approved Lead Rater  If no Approved Lead Rater is available, a Rater Supervisor must be used * All newly trained raters will need to work with Approved Lead Rater or Rater Supervisor Oral Exam Rating Process: Overview

6. 6 | Page  Coordination Options:  Contract directly with raters  Contract directly with a third-party vendor  Contract directly with raters in some languages and use a third-party vendor for other languages Coordinating Rating Services

7. 7 | Page  Preliminary steps: Check availability of raters (before announcing exam) Develop contracts/work agreements Security requirements Time requirements Payment Coordinating Rating Services

8. 8 | Page  Marked Scripts  Scoring Dictionary  Rater Results Form  Candidate Audio Recording  Scoring Unit Suggestion Form Materials for Raters

9. 9 | Page  Must be transmitted via secure method  Rater folders on SharePoint  Secure shipping method (Fed-Ex, UPS) Transmission of Materials

10. 10 | Page  Notify NCSC of raters to be used  NCSC provides Program Managers with access to individual rater folders on SharePoint  Program managers post materials to individual rater folders on SharePoint Transmission of Materials via SharePoint

11. 11 | Page Transmission of Materials via SharePoint

12. 12 | Page Marked Scripts with Rater Markings (one script per rater for each candidate) Rater Results Forms (one per candidate) Candidate Audio Recording Scoring Unit Suggestion Form(s) Materials Returned by Raters

13. 13 | Page  Audit all materials  Make sure Rater ID numbers and Candidate ID numbers are on scripts  Make sure rater markings match on both scripts  Ensure rater markings on scripts match scores on Rater Results Form  Check math Program Manager’s Role: After Rating

14. 14 | Page Program Manager’s Role: After Rating

15. 15 | Page  Determine rescores (if needed)  Report results to candidates  Prepare final materials for NCSC  Report any rater issues to NCSC Program Manager’s Role: After Rating

16. 16 | Page Materials to deliver to NCSC:  Marked scripts from each rater (two per candidate)  Candidate audio recording  Rater Results Form (one per candidate)  Scoring Unit Suggestion Forms  Candidate Oaths (and any Rater Oaths needed) Program Manager’s Role: After Rating

17. 17 | Page Provide final information on candidate testing status (no-shows, cancellations, etc.) Check-in all final files on SharePoint Confirm NCSC has all marked scripts and audio files before deleting Complete rating and upload all test materials to NCSC prior to your next test administration cycle Communication to NCSC

18. 18 | Page Questions? Q&A Session