1. The hybrid success model: Theory and practice G. Gage Kingsbury Martha S. McCall Northwest Evaluation Association A paper presented to the Seminar on longitudinal measurement, University of Maryland April, 2005

2. What does “success” mean for a school? Consider two schools, both of which have half of the students in each grade identified as proficient or higher Which one is more successful? Consider that one of these schools caused twice as much growth as the other Which one is more successful?

3. Difficulties with Proficiency Models A model which counts the number of students in a few proficiency categories causes schools to focus on borderline students, to the possible detriment of all other students The setting of proficiency levels is an exercise which can cause inconsistent and inappropriate decisions

4. Difficulties with Growth Models Growth models need strong assessments to operate well If students grow, it doesn’t mean they will become proficient

6. A Hybrid Success Model Set a growth target for each student that will lead to proficiency Measure each student’s status Measure each student’s growth Judge a school by the success of each student

8. Elements of a Hybrid Success Model Observed Growth = X t2 – X t1 Expected Growth = the average growth of students with a beginning achievement level equal to this student. Standards Growth = the amount of growth needed to get to proficiency by a target year. (Proficiency ty – X t1 )/ (number of yrs to target) Growth Target = the greater of Expected Growth and Standards Growth

9. Hybrid Success Index (HSI) for a student HSI = 1 if Observed Growth >= Growth Target Otherwise HSI = Observed Growth / Growth Target

10. Hybrid Success Value: School or District HSV = Mean HSI

11. Advantages of the HSM Incorporates both proficiency levels and growth Gives credit for growth of all students Sets predetermined goals for both students and schools Gives a more complete picture of school and district growth

12. Comparison of HSM and Traditional Value- Added Models HSM Adds substantial information to standards approach Identifies schools that cause more growth than typical Growth of every student counts Compares schools to a constant scale Requires a single measurement scale All schools can be successful Points out needed instruction Needed growth to succeed is known in advance Value-Added Model Adds substantial information to standards approach Identifies schools that cause more growth than typical Growth of every student counts Compares schools to one another Allows multiple measures Always results in high and low schools Doesn’t point out needed instruction Growth isn’t prespecified

13. What is Needed to Implement the Hybrid Success Model? Accurate assessments A measurement scale that allows growth across years A vertical scale related to proficiency levels across grades A process to assign useful growth targets to students

14. What is needed to implement a vertical measurement scale that can measure growth across years? Very accurate measurement for each student A data system that follows students across time At least one, preferably two or more scores per year. A theory of learning that allows continuous growth A scale that doesn’t change across time

15. Vertical scaling is enhanced by adaptive testing Each form is unique Each score has small error Item parameter estimates have less error Links throughout the scale are dense and accurate

16.  Vertical Linking Block  Grade X Form Grade X +1 Form Fixed Form Vertical Linking

17.  Grade X  Grade X +1 Adaptive Continuous Vertical Linking

18. What is meant by an equal interval scale? For any 2 values of theta* on the scale, the odds ratio of success on a given item equals the odds ratio of the two scores. A 1-unit change in theta difference results in a 2.718 unit change in the odds for success These scales do NOT indicate growth measures that are equal for equal intervals of time.

19. When conditions for vertical scaling are not met, results can be disappointing Trying to merge two or more existing scales is not advised (e.g., merging existing benchmark scales). Merging scales from tests given far apart in time can be difficult to interpret (e.g. Haertel’s analysis of NAEP scales) Fixed form linking is often too weak for vertical scaling (e.g., Huynh, Meyer & Barton)

20. Advantages of Using Scale-related Growth Measures Scores can be interpreted in terms of curriculum continuum descriptors Provides practical knowledge of what the student has mastered and what needs to come next Growth can be shown both absolutely and in comparison to performance standards Meaning of growth is more accessible to teachers, parents and students

21. Growth is not uniform at all ages or abilities

22. Growth is not uniform in reading either

23. A Comparison of Three Models --149,000 students --74,788 8 th graders in 2005 --75,302 5 th graders in 2005 --Reading & mathematics scores from Spring 2003, Spring 2004, Spring 2005

24. Models of accountability compared --Percent meeting standard median of most recent state studies (smoothed across grade levels) --Standardized growth ((Observed growth - expected Growth) / SD of Exp Growth) using point growth norms --Hybrid success value Using a proportionate growth increment toward 10 th grade standards

25. Accountability for Schools MeasureSchool CriteriaPolicy Goal Percent Meeting Standard>= 57% % needed to meet 100% proficiency in 2014 Standardized Growth>=0 Greater than expected so that mean moves upward on the scale Hybrid Success>=.771 Index needed to meet 100% proficiency in 2014

26. Rate of School Success Schools/Grades Meeting Accountability Criteria AllGr 5 MathGr 8 MathGr 5 ReadGr 8 Read ModelN%N%N%N%N% % Met173464%57663%27762%56463%31770% GrowZ131749%41846%18642%42047%29365% HSV180567%64771%25156%61768%29064%

27. Agreement between Model Accountability Criteria Agreement AllGr 5 MathGr 8 MathGr 5 ReadGr 8 Read %Met/GrowZ/HSV N%N%N%N%N% Y/Y/Y101037%31535%13630%32736%23251% Y/Y/N181%00%41%20%123% Y/N/Y53320%21824%9120%19121%337% Y/N/N1736%435%4610%445%409% N/Y/Y1847%748%215%677%225% N/Y/N1054%293%256%243%276% N/N/Y783%404%31%324%31% N/N/N61323%19421%12227%21424%8318% Total2714100%913100%448100%901100%452100%

28. Final thoughts Students, teachers, and administrators need accountability goals that they can see beforehand and tie to specific action Accountability that doesn’t require growth for all students is weak accountability Accountability that doesn’t reward students and teachers for great growth is weak accountability

29. Thank You for Your Thoughtful Consideration of These Ideas Northwest Evaluation Association www.nwea.org 503-624-1951