What are Best Practices in Large Scale Testing? A New Book: 2013 Edition Outlines Those Practices Operational Best Practices for Statewide Assessment Programs Update – 2013 Edition 1

Introductions 2 Presenters  Christopher Hanczrik – State of WA, Office of Superintendent of Public Instruction (Co-Chair)  Pat Roschewski – Data Recognition Corporation  Sarah Susbury – Virginia Department of Education  Lisa R. Ehrlich – Measured Progress (Co-Chair)

Outline 3  Background  Chapter 0 - Assessment Program Design (Pat)  Chapter 9 - Test Administration (Sarah)  Chapter 15 - Technology-based Assessments and Technical Support (Sarah)  Chapter 22 - System and Program Interoperability (Lisa)  Chapter 8 – Security (Christopher)

Background – 2010 Edition 4  Began in June of USDOE, ATP, CCSSO, AAP  A collaborative effort between CCSSO and ATP - Testing Directors and Service Providers together at the same table  Focus was predominately to define and describe Best Practices in operational fulfillment of paper-based large-scale, high-stakes assessment - Includes examples and implementation suggestions - Articulate what to do but not how to do it - These are not standards but practices - Voluntary, inclusive and nonproprietary - Released July 2010 and available for purchase on ATP/CCSSO websites (thousands of copies sold)

2013 Edition 5  Convened new working group in January 2011  Focus was review existing document and expand or augment technology-based and operational assessment topics  Author new and/or significantly expand existing chapters to address technology specific topics - Interoperability - Assessment Program Design - Security - Technology-Based Assessments - Accessibility focus, especially for technology-based assessments - Online Reporting

2013 Edition - Status 6  Completion of review draft – September 2012  Stakeholders review October – November Internal review by ATP “Heads of Houses” and staffs, as well as CCSSO, State DOEs, EIMAC, Assessment Directors.  Public review February – April 2013  Release of 2013 Edition – expected August

How do vendors use the OBP? Edition of the Operational Best Practices is typically used in a number of different ways. Training  Onboarding of incoming team members  Recurring “recertification” training for existing team members Program Design - Test Design- Pilot Testing - Delivery Model- Distribution Requirements - Item Bank Development- Reporting Plan

8 Process Development  Internal QA procedures across each department  Security protocols throughout the program cycle  Program reporting requirements both internally and externally Client/Service Provider Dialogue  Provides foundation and continued reference source How do we use the OBP?

Pat Roschewski Senior Director of Education Solutions Data Recognition Corporation Chapter “Zero” – Assessment Design 9

10  The chapter’s name – defines its purpose  Had I only known...  Designed for state directors – new and veteran

Chapter “Zero” – Assessment Design 11  Checklist of topics, considerations – pre-RFP  Each “topic” forecasts a chapter  The chapter order...assessment cycle  New name of Chapter “0” Pre-Chapter: State Considerations for Assessment Program Design

Sarah Susbury Director of Test Administration, Scoring, and Reporting Virginia Department of Education Chapter 9. Test Administration Chapter 15 - Technology-based Assessments and Technical Support 12

Chapter 9 - Test Administration 13  Paper-based, Technology-based, and Mixed Mode Testing  Calendars and Schedules  Defined Roles at the State, District, and School Levels.  Test Administration  Technology  Test Administration Directions for Students  Test Environment Protocols  Materials and Procedures for Training and Test Administration

Chapter 15 - Technology-based Assessments and Technical Support 14  Previously Chapter 15 was named “Online Assessment and Technical Support”  Content of the chapter transitioned from aspirational to best-practices  Infrastructure  Devices and Software  Assistive Technology

Chapter 15 - Technology-based Assessments and Technical Support 15  Security  Technical Support  Configuration of Testing Environments  Training  System Performance and Monitoring  Change Management  What to do, not how to do it.

Lisa Ehrlich COO Measured Progress Inc. Chapter 22. System and Program Interoperability 16

Interoperability 17  System - Hardware - Operating Systems - Application software  Assessment System - Data - Reports  Networks and Data transport

Industry Standards 18  Content  Data/Reporting  Integration Components - Seamless - Content – TEI - Delivery - Online reporting's  Student Registration - Student related data - Accessibility - SIS

Standardized Processes for Assessment Systems 19  System Components - Hardware – Servers, interface devices - Software – operating systems, applications, SIS, delivery, APIs - Network – WAN, LAN, Wireless, Cloud  Migration Paths for legacy components

Data/Content Interoperability Considerations 20  Item Development  Item Banking  Delivery  SIS  Accessibility  Scoring  Results

Interoperability Standards 21  SIF – School Interoperability Framework - XML data formant  IMS Global - QTI [Question and Test Interoperability Specification] - APIP [Accessibility Portable Item Protocol]  CEDS – Common Education Data Standards  AIF – Assessment Interoperability Framework

Christopher Hanczrik State of WA Office of Superintendent of Public Instruction Chapter 8. Security 22

Chapter 8. Security 23  So what is so new about test security?  Paper and technology-based assessments addressed together and separately

Chapter 8. Security 24  Define and design a comprehensive security plan  Internal processes  Computer systems  Electronic transfers  Test administration

Chapter 8. Security 25  How to reduce the likelihood of a security breach  How to deal with a security breach when it occurs

Chapter 8. Security 26  Protect your assessment program, your organization –client and service provider

Chapter 8. Security 27  Develop the Plan  Implement it!!

How does this book benefit us? 28  All states are in transition - Assessment Consortium - New ESEA legislation - Enhancements to system  Service provider and state perspectives – learning from each other - Partnership  “Hot button issues” – security, technology-based testing

How else does this book benefit us? 29 Additional players joining the large scale, high-stakes assessment environment: - Data systems & network specialists - Hardware & software developers - Technology leaders - New assessment leaders Additional need for current and up-to-date information: - “examples and implementation suggestions” - “non-proprietary views” - “what to do but not how to do it” - operational best practices

Questions? 30

Thank You If you should have any questions, please contact or Please complete the session evaluation that has been distributed to you. 31