1. Oregon’s Online Assessment Tony Alpert Oregon Department of Education CCSSO- NCSA June 21, 2010

2. Increase in Oregon Online Testing *Math general education tests used as an example **461 paper math tests administered in 2008-09 **

3. Oregon’s Contract –Oregon Department of Education Contracts with Oregon teachers to write all of its test items Conducts the psychometric work Develops and documents the system supplies Tier I help desk services Maintains a database with “official scores” Oregon supplies the official student, teacher, and school level reports online

4. Oregon’s Contract (cont) –Oregon’s Vendor (AIR) Provides an item bank application that holds Oregon’s items Supplies the test delivery application including the adaptive algorithm Supplies an online reporting system that incorporates tiered security Delivers data to Oregon for Oregon to incorporate into it’s database

5. Who’s doing what? –Project Management for Vendor and State. If the state doesn’t meet its deadlines, vendor may not meet theirs –Quality Control Item quality control including item rendering consistency with past format Unit testing, system testing –Help Desk and Training for State, District and School Personnel

6. Trust but verify –Application Specifications including use cases and mock-ups –Must include minimum system requirements –Test plans that encompass each element of functionality and on each “flavor” of browser/operating system combination –Test plan reports that describe whether the tests were successful –Plain language is critical –User acceptance testing in school environments is powerful

7. Browse with Security –Secure browser development should be completed by July of each year to allow districts to install during the summer. Push back development timelines accordingly –Lock down supported browsers and platform versions earlier rather than later –Decide whether to allow auto-update or not

8. Preventing Problems –Build a system that is designed to work within the technology available to your schools –Limit the use of propriety 3 rd party software –Limit bandwidth use by reducing item size –Include detailed Service Level Agreements in the RFP and resulting contract Include System Latency System down time Item presentation (Correct Item at the Correct Time) Help Desk Response Time/Resolution Data delivery (quality, timeliness) Quality of deliverables

9. Preventing Problems (cont) –Ensure vendor has a comprehensive approach to change management including Test, Development and Production environments Whom has access to change the production environment (Change Review Board) Version control and documentation –Ensure vendor’s system is reliable Fail over requirements Backup requirements Business continuity Security –Limit the volume of work sub-contracted

10. Fixing Problems –Push development timelines as early as possible –System to track known issues during the development period and during operational testing –Communicate clearly (and early) as to when support for hardware, platforms and operating systems are anticipated to end –Ensure problem resolution process is clearly described –Consider methods for emergent communications

11. Policy Stuff Reset vs Restart tests? Testing in non-traditional environments –Alternative programs –Home school –Students with long term medical illnesses Online/Virtual schools? Resolving over multiple testing events Duration between start and end of the test

12. System Enhancements –Annual lessons learned sessions –Include specific expectations for enhancements Federal compliance reviews Enhancements provided to other clients Annual improvements to the interface based on school and district requests