1. The DATA WISE Process and Data- Driven Dialogue Presented by: Lori DeForest ldeforest@cnyric.org (315)433-2247

2. “ Using data effectively does not mean getting good at crunching numbers. It means getting good at working together to gain insights from student-assessment results and to use the insights to improve instruction. ” - Kathryn Boudett, Elizabeth City, & Richard Murnane, “ When 19 Heads Are Better Than One, ” Education Week, December 7, 2005.

3. The DATA WISE Process and Data- Driven Dialogue

4. Organize for Collaborative Work Build Data Teams Establish team structure to allow for data discussions Establish norms Utilize protocols Complete a Data Inventory

5. Data Teams Raise important questions about student learning and achievement Assist in organizational aspects of data Dialogue about multiple data sources Examine and interpret data Investigate ways to improve teaching and learning

6. Curriculum Councils Initial Data Teams October and November - Setting Norms Protocol - Compass Points Protocol

7. Setting Norms Protocol What norms do we need? Brainstorm Discuss Synthesize Build consensus

8. We need to build emotional safety to reach cognitive complexity. B. Wellman and L. Lipton

9. Curriculum Councils Initial Data Teams October and November - Setting Norms Protocol - Compass Points Protocol December and January - Data Inventory - Data Analysis Tools

10. Purpose of Data Inventory Summarizes all the types of data that are available and helps to determine what other data is needed Builds assessment literacy Assists in the planning of using data effectively Begins conversation about educational questions

11. Data Inventory Data Source Dates of Collection Students Assessed Purpose Current Data Use More Effective Use ELA State Asmt January (results in summer) Grades 3, 4, 5, 6, 7, 8 State Accountability purposes and to evaluate program and students’ … Program evaluation and intervention placement Data Team analyzes data to inform instruction Other Student Level-Information Race/Ethnicity Data Wish List English Proficiency Attendance Race/Ethnicity

12. Data Inventory Data Source Dates of Collection Students Assessed Purpose Current Data Use More Effective Use Running Record OngoingGrades K - 2 Evaluate a students’ reading skill and level To inform planning of guided rdg. lessons and inform new teacher of rdg level District Writing Folder 3 times per year per grade Grades K-12 Document a students’ writing progress over time To archive student work Teacher review regularly to inform instruction

13. The DATA WISE Process and Data- Driven Dialogue

14. Assessment Literacy

15. Data Analysis Tools COGNOS Report Net COGNOS Power Play Cubes Data Mentor nySTART NYS State Report Card Databases and ELA and Math Media Databases NYS State Report Card Databases and ELA and Math Media Databases Student Management System Demonstrate tools at Curriculum Councils, Grade Level and Department Meetings Offer training opportunities

16. The DATA WISE Process and Data- Driven Dialogue

17. Data Overview Determine audience Decide on educational questions Create graphic displays of standardized test results Engage in conversations around initial data set

18. 2006 English Language Arts Performance Data Sources: CNYRIC COGNOS PowerPlay Cubes, NYSED School Report Card and ELA Assessment Databases

19. 2006 Mathematics Performance Data Sources: CNYRIC COGNOS PowerPlay Cubes, NYSED School Report Card and Math Assessment Databases

20. The DATA WISE Process and Data- Driven Dialogue

21. “ Without an investigation of the data, schools risk misdiagnosing the problem. ” Data Wise, 2005

22. Data Analysis Protocol Activate and Engage Set norms Articulate predictions and assumptions Explore and Discover Begin with a single data source First describe what you see Ask questions Identify additional data needs

23. The DATA WISE Process and Data- Driven Dialogue

24. Data Analysis Writers Club

25. Informing Planning for Writers Club Use COGNOS PowerPlay to identify needs of struggling students. Item Difficulty (% of points earned out of total possible points) 2006 Grade 3 English Language Arts Assessment SCHOOLDISTRICT BOCES Region MC 01Level_3 L3_Low89.7%90.0%92.4%92.6% Level_393.8%95.5%95.2%95.1% Level_2 L2_High85.7%74.1%86.4%86.7% L2_Med100.0%85.7%81.9%80.9% L2_Low100.0% 72.5% Level_291.7%79.5%82.5%82.2% Level_1 L1_High80.0%55.6%54.0%56.6% L1_Med/0 23.5%20.5% L1_Low/0 20.0%14.3% Level_180.0%55.6%51.6%53.6% All Performance Levels94.1%92.7%90.1%89.6% MC 02Level_3 L3_Low100.0% 97.6%97.5% Level_3100.0%99.5%98.3%98.4% Level_2 L2_High100.0%85.2%95.3%95.6% L2_Med100.0%71.4%93.2%93.9% L2_Low100.0% 91.0%91.7% Level_2100.0%84.6%93.9%94.4% Level_1 L1_High80.0%88.9%72.1%74.6% L1_Med/0 23.5%17.9% L1_Low/0 20.0%14.3% Level_180.0%88.9%68.3%70.1% All Performance Levels99.2%97.4%95.6%

26. A distracter analysis may help you understand children ’ s incorrect thought processes. Distracter Analysis Item Count as values 2006 Grade 3 English Language Arts Assessment ELEMENTARY SCHOOL Choice 1 Choice 2 Choice 3 Choice 4 Blank MC 01Level_3 L3_Low126200 Level_3276300 Level_2 L2_High06100 L2_Med04000 L2_Low01000 Level_2011100 Level_1 L1_High14000 L1_Med00000 L1_Low00000 Level_114000 All Performance Levels3112400 MC 02Level_3 L3_Low002900 Level_3008100 Level_2 L2_High00700 L2_Med00400 L2_Low00100 Level_2001200 Level_1 L1_High10400 L1_Med00000 L1_Low00000 Level_110400 All Performance Levels1011800

27. Data Analysis Writers Club Physical Education program Intervention Analysis

28. 2006 Cohort ELA 8 Performance of students who performed at Level 1 or 2 on ELA 4 and remained in district (n=43) Tracking cohort performance may give you some information about program or student growth.

29. Data Analysis Writers Club Physical Education program Intervention Analysis English Language Arts Analyses

30. English Language Arts Educational question: Is student performance declining in reading comprehension but increasing in listening comprehension? Utilize COGNOS PowerPlay item analysis and Scoring Key to classify questions by subtest Compile data using formulas and functions Study trends over time Use comparative data to inform analysis

32. The graph above illustrates the importance of using comparative data to inform analysis.

33. English Language Arts Analyses COGNOS Report Net now houses analysis reports which can provide you information about student performance.

34. English Language Arts Analyses Last Name First NameSchoolTeacher Reading Comp (MC-28) Listening Writing (5) Reading Writing (5) Writing Mechanics (3)Level Total Score SampleAbbyEHEladata18331Level 3652 SampleBobEHEladata26332Level 3695 SampleCatherineEHEladata28332Level 3711 SampleDavidEHEladata27342Level 3711 SampleErinEHEladata28342Level 4721 SampleFrankEHEladata11211Level 1606 SampleGabbyEHEladata22332Level 3673 SampleHaroldEHEladata27332Level 3703 SampleIrisEHEladata15321Level 2632 SampleJacobEHEladata24443Level 3711 SampleKarenEHEladata20222Level 3652 SampleLukeEHEladata28232Level 3703

35. Sampling Principle “ Because a test is not a direct measure of a student ’ s degree of mastery of an entire domain, any conclusion you reach about proficiency in that domain is based on an inference from proficiency on the smaller sample. … Even a test that provides good support for one inference may provide weak support for another. ” Data Wise, 2005

36. Data Analysis Writers Club Physical Education program Intervention Analysis English Language Arts Analyses Math Analyses

37. Data Overview of NYS Assessment Performance Considerations when reviewing summary data Different cohort groups Different samples of items each year Different test blueprints Importance of comparison data sets

38. Go to www.emsc.nysed.gov/osa and the Report Card link to access data for similar schools.www.emsc.nysed.gov/osa

39. Note: Dual enrollments taken into account for total annual percentage Enrollment in College Level Math Courses Utilize your own student management system to analyze additional data as well.