1. CALIFORNIA DEPARTMENT OF EDUCATION Tom Torlakson, State Superintendent of Public Instruction California Computer Based Testing Tryout in Science CCSSO Conference on Student Assessment June 21, 2013 Eric E. Zilbert, Interim Director Assessment Development and Administration Division

2. TOM TORLAKSON State Superintendent of Public Instruction 2 Purpose of the Study Collect feedback and data useful for state in transitioning to CBT and CAT. Provide students and schools an opportunity to try CBT. Develop and administer new technology- enhanced (TE) items; evaluate the feasibility of artificial intelligence scoring. Conduct special studies of items and tests.

3. TOM TORLAKSON State Superintendent of Public Instruction Test Design 3 Science Tests –Grades 5 & 8 Science –High School Biology Item types –40 traditional single select multiple- choice items –20 technology-enhanced items Drag-and-drop & hotspot functionality Simulations Artificial intelligence scoring

4. TOM TORLAKSON State Superintendent of Public Instruction Sample 4 21,473 students, 193 schools, 133LEAs Represented all areas of the state Included all demographic subgroups

5. TOM TORLAKSON State Superintendent of Public Instruction Key Findings Of those LEAs and schools that participated, the majority indicated: –Their school was “somewhat” or “very well” prepared for CBT. –The technology functioned as expected. –The CBT system was easy to use. The majority of students expressed satisfaction with the experience. –Instructions, pictures, and item types were easy to understand and intuitive. 5

6. TOM TORLAKSON State Superintendent of Public Instruction Key Findings Reported problems during testing were primarily related to: – Software Issues: needed to update software, difficulty logging on, difficulty with pausing and reactivating student testing sessions. – Server Problems – both local and offsite Inhibited dragging and dropping. Booted students from the system. – Lack of resources, including computers, technical help, and Wi-Fi connections. 6

7. TOM TORLAKSON State Superintendent of Public Instruction Key Findings Multiple choice items performed similarly in the CBT compared to the previous paper administration (difficulty, discrimination). Artificial intelligence scoring was successfully used for: –Graphs, equations, and numeric-entry responses, and –Short constructed-response items. 7

8. TOM TORLAKSON State Superintendent of Public Instruction For Further Information Eric E. Zilbert ezilbert@cde.ca.gov CBT Tryout Report www.cde.ca.gov/ta/tg/sr/documents/cbttryoutrpt.pdf 8