1. Open Source Innovations for Better Assessment Brandt Redd CTO NCSA 22 June 2015

2. Smarter Balanced at a Glance 23 Members: 21 States, US Virgin Islands, Bureau of Indian Education Represent more than 14 million students. Formative, Interim, and Summative assessments aligned to the Common Core State Standards for English Language Arts and Mathematics. 2

3. Achievements in the Last 12 Months New home at UCLA Released open source assessment software (Assessment Production, Test Delivery, Scoring, Reporting, Digital Library) Summative tests delivered to 6.5 million students in 18 states. Millions of interim tests delivered. More than 570,000 registered teachers on the digital library. Nearly 200 colleges accept Smarter Balanced assessment for placement purposes with more on the way. 3

4. Devices Used in Spring 2015 Tests Fraction of 38 million test sessions using each device type. 5.6% Windows 8 38.5% Windows 7 0.1% Windows Vista 2.3% Windows XP (percent of all devices) (Preliminary Results as of 22 May 2015) 4

5. Innovative Features Constructed Responses: Graphing, Equations Accessibility Features: Text-To-Speech, Refreshable Braille, Tactile Graphics, more Computer Adaptive Tests (CAT) 5

6. SmarterApp.org Specifications Core Standards (Format and API) Item Format Item Metadata Content and Test Packaging Test Results Reporting API Data Extracts More 6

7. Interoperability Standards and Specifications used by SmarterApp Common Core State Standards SmarterApp Assessment Item Format Common Education Data Standards (CEDS) –Smarter Balanced Test Results Loaded into “Align” IMS Content Packaging SAML and OAuth (Single sign-on) Smarter Balanced Profile of IMS QTI/APIP* LRMI and Learning Registry* Accessible Rendered Item format (ARI)* *Planned 7

8. Interoperability Challenge Each member state chooses its service provider to administer, score, and report the Smarter Balanced tests. 8

9. Technical Challenges Deploying open source across vendors Packaging assessments for multiple vendors Collecting data in multiple formats from multiple states and vendors Efficient assessment production system Sustaining pace of innovation 9

10. Open Source Tools for the Assessment Lifecycle 10 Test Blueprint Standards Item Specifica- tions Item Authoring Item Banking Test Authoring Test Package Test Specifica- tions Test Delivery Item Scoring Test Scoring Test Results Hand Scoring Reporting Individual & Aggregate Reports Data Warehouse Assessment Design Assessment Production Assessment Delivery Reporting & Analytics

11. Assessment Production Current Status –Contracted out for Assessment Production Support in 2014-2015 and 2015-2016 –Open Source Item Authoring, Item Bank, and Test Authoring tools on SmarterApp.org Roadmap –Integrated assessment authoring system based on software production systems –Software RCS for Item Pool –Open Source and Multivendor Item Authoring –Open Source Test Authoring and CAT Simulation 11

12. Assessment Delivery Current Status –Summative tests delivered to more than 6.5 Million Students in Spring 2015. –18 States and 5 Test Delivery Vendors –3 States used the Open Source Test Delivery System (TDS) Enhancements for 2015-2016 –Increased robustness, convenience, and capacity in the TDS –Universal Secure Browsers –Accessibility Updates Enhancements for 2016-2017 –Horizontal server scaling –New Accessible Rendered Item (ARI) format –Summative Handscoring –Graphing Calculator –Accessibility Updates 12

13. Reporting & Data Warehouse Current Status –Used by 4 States in 2014-2015 –Some hosted by Smarter balanced, others hosted by vendors Roadmap –Updated “Test Results Transmission” format –Public Reporting –Clearer and more customizable Individual Student Reports –Links to the Digital Library 13

14. Digital Library of Formative Materials Current Status –More than 2,200 curated resources –More than 575,000 registered teachers –Community features – content submission, sharing of experiences, feedback Roadmap –Better integration with the Learning Registry –Improved Search –Local Community Groups –More Content –More Community Features 14

15. Accessible Rendered Item Format “Like a PDF for Assessment Items” Facilitates innovative new item types. Complimentary to QTI/APIP Leverages JavaScript for accessibility and scoring Supports multiple item authoring systems – each specializing in certain item types Rendering modes: –Response Capture –Inline feedback (for formative use) –Response rendering (for scoring and review) xAPI and Caliper compatibility 15

16. Opportunities For Assessment Technology Vendors –Watch for Smarter Balanced RFPs –Incorporate SmarterApp technology into your solutions for interoperability and cost savings –Deploy technology for Smarter Balanced states For Assessment Developers –Leverage production and delivery tools –Consistent delivery platform across subject areas For States and Districts –Consistent secure browser technology across all assessments –Build state and local assessments on the same technology platform as Smarter Balanced 16

17. Q&A Resources –http://www.smarterbalanced.orghttp://www.smarterbalanced.org –http://www.smarterapp.orghttp://www.smarterapp.org Specifications including test results 17

18. Sample Individual Student Report 18