Tools and Resources to Increase Data Use Capacity Rebecca Cohen, AACC & CRESST/UCLA Ellen Osmundson, AACC & CRESST/UCLA Neal Gibson, Arkansas Department of Education CCSSO Leadership Conference September 8, 2008 Milwaukee, Wisconsin

Overview Background Data Use Web Based Tools Key Capacities Map Inventory Formative Assessment Data Use Scenarios Collaboration with Arkansas Department of Education

Background

Assessment and Accountability Comprehensive Center Supports the use of data to achieve the goals of No Child Left Behind Offers Regional Comprehensive Centers and states a national perspective on research-based resources and access to effective models, research syntheses, toolkits, and software systems Operated by WestEd in partnership with CRESST/UCLA

What is Data Use? Data use involves systematically collecting, analyzing, and interpreting various types of data to guide educational decisions at the state, district, school, and classroom level for improved student achievement.

Data Use for Continuous Quality Improvement Website: 18-month, iterative process, drawing on current educational research and consultations with expert scholars, and national and state leaders in the area of data use for school and student improvement Conceptually based representations, tools, and resources of the domain of data use in educational organizations (classrooms, schools, districts, states)

What is on the Data Use Website Defines a process for data use linked to foundational core capacities Organizes and makes available research and research-based guidance Presents criteria for evaluating data tools and selecting and using diagnostic tests Offers professional development for assessing data use in organizations, and selecting and using diagnostic tests

Data Use Process Guide

Criteria for Selecting Data Tools

Criteria for Selecting Diagnostic Assessments

Professional Development Tool for Selecting Diagnostic Assessments

Library

Map of Key Capacities for Data Use Multi-faceted representation of the dependent and dynamic network of capacities crucial for effective data use Iterative process, drawing on current educational research and consultations with expert scholars, and national and state leaders in the area of data use for school and student improvement Emphasizes five key concepts, and their relationships to each other and more than twenty other sub-concepts

Leadership Significance of leadership for the process of data use in educational organizations Essential for fostering the policies for educational improvement Establishes standards for learning and data quality, and assessment use and institutional capacity and improvement Responsible for developing a culture of inquiry and collaboration among data users, establishing expectations for data use, and building capacity to support data use Facilitate communication with stakeholders

Inventory

Tool for Organizations to Assess Data Use Capacity Aligned with the 5 Key Capacities Leadership, Data Quality Assurance, Infrastructure, Communication, Action for Learning General Inventory Provides a broad overview of each capacity In-depth Inventory Provides a more detailed analysis of each capacity Options for individual or group scores Resources Aligned with library Transformation Plan Plan next steps, outline responsibilities, tasks

Develop Dept. of Education knowledge and skills associated with formative assessment

Formative Assessment

Working Definition of Formative Assessment The following working definition of formative assessment has been derived from a review of definitions of formative assessment presented in the literature. This definition provides the framework for the assessment cycle. Formative assessment is an ongoing process that provides feedback during the course of instruction to teachers and students to close the gap between current learning and a desired goal.

Assessment Cycle Outlines learning and assessment process Research-based Iterative in nature

Assessment Strategies Five key strategies of formative assessment Clarify learning goal, establish success criteria Illicit evidence of learning Provide feedback Activate student ownership of learning Establish students as instructional resources

Data Use Scenarios

Scenarios: State, District, Classroom Narratives about how education leaders use data to identify and address challenges of student achievement in their organizations Indicates where leaders are at different points in the data use process and the types of analyses appropriate to their work Includes a list of additional readings with suggestions for how education leaders can use resources to improve data use in their organizations Can be used for professional development

State Scenario Narrative The members of a State Board of Education in a small state are increasingly hearing from parents that the reason their children’s schools are receiving low scores on the statewide Academic Achievement Index, a general measure of student achievement in the core academic subjects, is because they have too many inexperienced and under-qualified teachers. The board members have charged the State Commissioner of Education with investigating this concern and making some policy recommendations based on the findings.

State Scenario Narrative (Research Questions) Research Questions The Commissioner has put together a team of four administrators from the State Department of Education to lead the investigation. Following considerable discussion, the team decides on the research questions that will guide their inquiry: What is the relationship between Academic Achievement Index scores and school percentage of experienced teachers? What is the relationship between Academic Achievement Index scores and school percentage of qualified teachers? To what extent are teacher experience and teacher qualifications associated with other school-level characteristics, including geographic location, grade-levels served, and student socioeconomic status?

State Scenario Narrative (Excerpt) Defining Research Terms After identifying the research questions, the leader of the team, Mr. Dawes, notes that they need to define the variables within each research question. “For example,” he says, “we need to figure out how to define experienced and qualified teachers. We also need to be specific about how we describe and categorize school-level characteristics to ensure that our descriptors are valid and appropriate. Having clear definitions of what we are measuring will help ensure we look at the right data, conduct the appropriate analyses, and report back to stakeholders with confidence in our methods.” Notice this section is about assessing the present state because the state leaders: Ask questions focused on how teacher and school-level characteristics relate to student performance Review the educational research literature to clarify their questions and identify appropriate data sources

State Scenario Resources (Example) Assessing the Present (Process Guide for Data Use) Professional Development Resources Citation/ISBN: [website - Fashola, O. S. (2004). Being an informed consumer of quantitative educational research. Phi Delta Kappan, 85(7), pp Summary: This article provides a straightforward description of various components of scientific research, including evaluation designs, measures of student academic achievement, and interpretation of statistical findings. The author also discusses factors that contribute to program effectiveness including clear goals, regular assessments, professional development, and support. How to use: Use as a focus for discussions about conducting quality research, including considerations for research design, implementation, analysis, and reporting. Use in conjunction with Mertens’ (2005) book and Leedy and Ormrod’s (2004) book.

Future Plans Revisions to the website Audio-visual components Updates about field work Share research through website Solicit work and feedback for website

Data Use Website in Action

Collaborative Partnerships: Formative Assessment Iowa: Iowa SD, NCCC, AACC Build capacity and support Iowa SD in roll out of Core Curriculum and formative assessment strategies Syracuse: NYSED, NYCC, AACC, SCS Build capacity in SCSD and develop a model that can be implemented in other districts in New York and other states North Carolina: NCDPI, ARCC, AACC Build support and capacity for formative assessment Contact:

Arkansas Collaboration: AACC, MC3, ADE Use the statewide database as a leverage point to think about supporting states: Disseminating data; Choosing data sources; Creating models to frame and think about data; assessing and addressing professional development needs Conduct pilot projects in select school districts Work with the state to create professional development that can be rolled out statewide Develop guidance for states

Arkansas Statewide Data System Neal Gibson

Questions/Comments Rebecca Cohen Ellen Osmundson Neal Gibson