2. DVAS Training 10-03-05

3. Find out how Battelle for Kids can help Presentation Outcomes Learn rationale for value-added progress measures Receive conceptual overview of value-added analysis View sample value-added reports See how value-added information fits with school improvement

4. Why are traditional achievement measures alone an insufficient way to assess student achievement? Rationale for Value-Added Progress Measures

5. The Changing Educational Landscape 2000’s: Standards-based education 1960’s: Mastery learning 1970’s: Behavioral objectives 1980’s: Minimum competencies 1990’s: Outcomes-based education

6. Looking at the changing educational landscape, a clear pattern exists: The focus has moved from what goes into a child’s education to what comes out of the process What’s the Upshot?

7. From Teaching Inputs Context (room, furniture, master schedule, course of study) Resources (number/quality of books, computers, materials) Capacities (knowledge of subject and teaching/learning processes, classroom control, lesson planning) To Learning Outputs State content standards and local curriculum aligned to standards Annual paper/pencil tests to measure achievement State report cards at district/building level Good teaching = High student performance The primary measure in “output focused” system is student scores on statewide achievement tests. The Shift in Focus

8. Stair-Step Expectations Most achievement measures imply: Achievement test scores are enough to show growth Students start at the same place Students progress at the same rate

9. Reality Students start at different places Students progress at different rates Educators need more than individual test scores to evaluate school’s impact on student learning

10. Need for Progress Measures To measure school effectiveness, we must pay attention to passage rates AND annual student progress

11. How do we maximize student progress each year, regardless of where they start? 85% of the public believes student progress is the best measure of a school’s effectiveness Question for Educators Today — Phi Delta Kappa/Gallup Poll 2005

12. Conceptual Overview of Value-Added Measures in Ohio How is performance data used to produce value-added information?

13. Two Value-Added Systems in Ohio Project SOAR Ohio’s Accountability System

14. Project SOAR Operated by Battelle for Kids Began in 2002 with 42 school districts Now has 106 districts and 3 charters schools Provides analysis in 5 subjects for grades 3-10 Uses state and non-state test data Uses a prediction based value-added approach Expected growth is normative (“Average Growth”)

15. Ohio’s System Operated by the Ohio Department of Education Begins as a 4th grade pilot in 2006 in all districts Provides analysis in math & reading in grades 4-8 Uses only state achievement tests Uses a mean gain value-added approach Expected growth is likely to be a fixed amount

16. What Do the Two Have in Common? Utilize the power of longitudinal data and linking together each student’s assessment data over time Compare students’ current test scores to baseline scores Provide value-added information in Web based reports Use the statistical power of EVAAS™ to produce the value-added analysis

17. Why EVAAS ™ in Ohio? EVAAS™ is a the value-added methodology pioneered by Dr. William Sanders of SAS Applies most sophisticated statistical methodologies available to ensure reliability Allows for the use of all student test data Provides valuable diagnostic information Approaches for handling different types of test data Identified by RAND and others as a preferred model Used statewide in Tennessee for 10 years

18. What is Value-Added? Value-added in simplest form is an accurate measure of the present (observed scores) minus an accurate measure of the past (baseline scores) for the same group of students. Mean Observed Score - Mean Baseline Score = Value-Added Math Scores Year 1 3rd Grade Student 1 350 Student 2 370 Student 3 360 Student 4 375 Student 5 365 Year 2 4th Grade Student 1 400 Student 2 385 Student 3 395 Student 4 405 Student 5 390 Mean Baseline Score 364Mean Observed Score 395 31 would be a crude measure of the value-added

19. Why are Two Systems Needed? When both tests are on a common scale like the Ohio achievement tests, baseline can come from the prior year’s test scores (Mean Prior Score Approach - Ohio System) When the tests are on different scales the baseline must be calculated. (Mean Predicted Score Approach - Project SOAR) Sophisticated statistics are required in both approaches to ensure that all students’ data are included, that the information is reliable, and to add predictive diagnostic power. Different approaches are needed to provide reliable baseline scores from the different kinds of tests used in Ohio

20. What are Common Scales? Vertical Scales increase in equal intervals as you increase grade levels Horizontal Scales remain the same as you increase grade levels Score at the 3rd 4th 5th 6th 7th 8th 50th percentile350 400 450 500 550 600 Ohio Achievement Tests Proficient 3rd 4th 5th 6th 7th 8th Score 400 400 400 400 400 400 When you have common scales, the prior years’ scores can be used as the baseline.

21. How Are Test Data Used When They Are Not on a Common Scale? All available test data is collected and linked for each student Districts are grouped into pools based on common testing histories Relationships between and among all tests in pool are used to create predicted baseline scores What students’ like them would typically score on this year’s test.

22. Collect all individual student data available for a minimum of three years Link each student’s annual test data together to create a longitudinal record How Do you Create a Longitudinal Record?

23. How Are Comparison Pools Created? Districts grouped into pools based on common testing histories at each grade-level cohort Example: Mike’s cohort testing history or pool

24. How Are the Relationships Between Tests Within a Pool Defined? Relationships between and among all tests in the pool are calculated and can be represented as a number or correlation. Example: using 2 years of and 4 years of Mike’s testing history

25. How Much Prior Data is Used? Up to 5 years of student test data and the relationships between tests are used to calculate predicted baseline score for this year’s subject-area tests.

26. How Are Predicted Scores Calculated? Using the test data for students with similar prior performance on common tests and the tests’ relationships to each other allows for the creation of statistically reliable predicted scores for each student in each subject Mike’s Prediction Student Prediction 2 Student Prediction 3 Student Prediction 4 Student Prediction 5 Student Prediction 6 Student Prediction 7 Student Prediction 8 Student Prediction 9 Student Prediction 10 Student Prediction 11 Student Prediction 12 Student Prediction 13 Student Prediction 14 Student Prediction 15 Student Prediction 16 Student Prediction 17 Student Prediction 18 Student Prediction 19 Student Prediction 20

27. Are All Students Used in the Analysis? Mike’s Prediction Student Prediction 2 Student Prediction 3 Student Prediction 4 Student Prediction 5 Student Prediction 6 Student Prediction 7 Student Prediction 8 Student Prediction 9 Student Prediction 10 Student Prediction 11 Student Prediction 12 Student Prediction 13 Student Prediction 14 Student Prediction 15 Student Prediction 16 Student Prediction 17 Student Prediction 18 Student Prediction 19 Student Prediction 20 Mean Predicted Score “Baseline” Mike’s Score Student Score 2 Student Score 3 Student Score 4 Student Score 5 Student Score 6 Student Score 7 Student Score 8 Student Score 9 Student Score 10 Student Score 11 Student Score 12 Student Score 13 Student Score 14 Student Score 15 Student Score 16 Student Score 17 Student Score 18 Student Score 19 Student Score 20 Mean Observes Score Your School Only students with enough prior data to create a predicted score are included.

28. Mean Student Score – Mean Predicted Score (with some statistical reliability factored in) = School Effect How Do you Estimate the School’s Effect on Student Growth?

29. Sample Value-Added Reports What information do value-added reports provide that was previously unavailable?

30. Achievement & Progress

31. High Achievement, High Progress High Achievement High Mean Scores = 89% passage High Progress Positive School Effects 2005 School Value-Added Report for OPT Math

32. High Achievement, Low Progress High Achievement High Mean Scores = 85% passage Low Progress Negative School Effects 2005 School Value-Added Report for OPT Math

33. Low Achievement, High Progress Low Achievement Low Mean Scores = 69% passage High Progress Positive School Effects 2005 School Value-Added Report for OPT Math

34. Student A Report

35. Student B Report

36. Student B Projection

37. Connecting to School Improvement Efforts How can value-added progress measures enhance school and district improvement at the: District Level School Level

38. District Improvement Efforts Identify patterns of progress across buildings, grade levels and subject areas Locate areas of strength to build upon Locate areas for improvement

39. District Value-Added Summary Report 2005 District Value-Added Summary Report 4th Grade

40. School Search Report 2005 School Search 4th Grade Math

41. School Improvement Efforts Identify patterns of progress across grade levels, subject areas and student subgroups Locate areas of strength to build upon Locate areas for improvement

42. School Value-Added Report 2005 School Value-Added Report Reading

43. Performance Diagnostic Report 2005 Performance Diagnostic for Reading 5th Grade Means

44. School Diagnostic Report 2005 School Diagnostic for Reading 4th Grade Means

45. Search for Students By Subgroup or Achievement Student Search

46. Identified At-Risk Students Student Search Results

47. In Summary, Value-Added Information Shows… How much progress students make in each subject area and grade level How much progress students at different previous achievement levels have made How students’ progress in one curricular area or program compares to their progress in another Whether individual students are making adequate progress to meet state standards Without data, all we have are opinions!

48. School Strategic Planning Cycle Pre-School Start of the Year Meetings Examine value-added and other school performance information by grade level and/or subject area Assess strengths and weaknesses and potential actions, grade level by grade level, subject by subject Celebrate progress Set 1-2 goals for each grade level, department, or team around strengths and weaknesses Create action plans and accountabilities Grade level, department and/or team meet to work on team specific goals Grade level, department and/or team meet to work on team specific goals Grade level, department and/or team meet to work on team specific goals

49. Select a lead person in the school district who understands value-added information, and can access, interpret and conduct presentations Train school leaders (principals and lead teachers) to access, interpret and conduct presentations Have school leaders share value-added information with school staff members Have school staffs use value-added information to assess annual progress and set goals for next year Implementation Checklist

50. Ohio’s Scale Up Plan

51. For more information, contact: www. BattelleforKids.org (866) KIDS-555