Development of Assessment Literacy Knowledge Base Kathleen Flanagan Assessment Research Coordinator, Massachusetts Department of Elementary and Secondary Education June 2016

Assessment System Features Massachusetts Department of Elementary and Secondary Education

What is Assessment Literacy? Massachusetts Department of Elementary and Secondary Education

Building Assessment Literacy LEAs have a great need to build capacity for assessment literacy More assessment engines available More data available More demands in the generation and use of local data (e.g., teacher evaluation) Different assessment literacy needs (e.g., data team leader vs. classroom teacher) Better, more accessible guidance is needed to generate better assessments and to make better use of assessment data to make decisions Massachusetts Department of Elementary and Secondary Education

Massachusetts Assessment Literacy Guidance Materials Provide users of assessment and reporting systems with key assessment literacy and data use information Support high-quality assessment and data use practices in LEAs Provide educator-facing materials on assessment literacy and data use Develop go-to reference materials for educators that will grow and evolve to encompass more topics, greater specificity over time

Assessment Literacy Materials Development Co-authored by NCIEA’s Charlie DePascale and Karin Hess Elicited contributions from: Massachusetts educators Graduate students (test construction class, Boston College) Massachusetts Department of Elementary and Secondary Education

Demonstration: Writing Content with MA LEAs *The basic logic in establishing scoring reliability for OR items is that the scoring would be consistent across scorers – in other words, an individual student’s response would be scored the same regardless of who was doing it. *When a single person is scoring a student’s work, reliability is enhanced by: Prepare a scoring guide that establishes points for each level of response Grade one question at a time to ensure uniformity in scoring Block the identify of the student when scoring the response Massachusetts Department of Elementary and Secondary Education

Revised Scoring Reliability Section Massachusetts Department of Elementary and Secondary Education

Materials Description Comparative Advantages of Item Types Type of Item Selected Response (Objective Items) Constructed Response (Subjective Items) Performance/Portfolio/ Observational Items Sampling of Curriculum Samples a lot of curriculum in a short period of time Samples less curriculum than selected response items; takes longer examinee administration time Item Development Requires the development of many items Fewer items are needed Fewer items are needed, but the items are written to break out the components of the task Rigor Can sample the range of Bloom's Revised taxonomy from Remembering to Evaluating. Takes skill to write items at the higher levels of rigor Constructed Response items should be written for higher levels of rigor Performance Items can range the levels of rigor although some of them should represent higher-level demands Complexity Low to moderate complexity Can range from low to high complexity Tasks should reflect moderate to high levels of complexity Scoring Objective scoring -- efficient with a scoring key Subjective scoring -- requires the use of rubrics/scoring papers and scorer training Subjective scoring -- requires the use of rubrics; students can participate in scoring Currently, 125 pages of body text with graphs, tables, illustrations Hierarchical nesting with larger topics broken out into digestible parts Specific heading styles (e.g., fonts and sizes) Multiple examples for each topic area ~5 pages glossary ~ 5 small datasets ~ 80 pages of links Excerpt, Grammatical Cuing Excerpt, Cycle of Inquiry

Table of Contents Section 2 Section 1 Massachusetts Department of Elementary and Secondary Education

Why Build a Knowledge Database? Discrete Documents Knowledge Database Can quickly overwhelm audiences Dense single documents Disparate small documents Interrelationships unclear Collection cultivated by single organization Static documents hard to update Navigational tools in a database can provide customized information to audiences Interrelationships clear in content and organization of materials Can be cultivated by multiple organizations, (e.g., cross-state use of materials) Updates made in real-time; scalable Massachusetts Department of Elementary and Secondary Education

Example: Social Science Research Knowledge Base Publication Style Illustration Left-hand navigation for document, nested topics Hierarchically arranged short topic presentations with unified graphics and illustrations Many illustrations – most are embedded as links Structure is scalable, allows for additional topics

Literasee Repository developed and hosted by the NCIEA Charlie DePascale Damian Betebenner State-level authorship and control over collections Technical expertise provided by NCIEA Publications generated using Github GIT=Version control system (for collaborative, multiple versions) HUB=Project repository Allows authors to connect to other web-based material Massachusetts Department of Elementary and Secondary Education

Why Open Content? Knowledge database contents available to all users (can request that attribution to original authors/states, etc., is used via Creative Commons licensing) Leverage materials across the Web Encourage fluidity of contents (revisions, updates) Massachusetts Department of Elementary and Secondary Education

Example from Literasee Massachusetts Department of Elementary and Secondary Education

A Work in Progress Complete online publication of Assessment Literacy materials Collect and revise draft per user feedback Other areas of needed guidance Data use Data visualizations (e.g., Student Growth Percentiles) Massachusetts Department of Elementary and Secondary Education