Jim Lloyd_2007 Educational Value Added Assessment System (EVAAS) Olmsted Falls City Schools Initial Presentation of 4 th Grade Students

Suppose this box represents a group of students for a given grade level and subject area… Low High

Low … and we fast forward to the end of the given school year

An Achievement Status Consideration Considerations for Gauging Effectiveness: 1) External Standard (proficiency level) 2) Student’s Ending Point of Achievement ? (Yes/No) … sets a proficiency level students are expected to meet.

A Value-Added Consideration Considerations for Gauging Effectiveness: 1) Student’s Prior Achievement Level 2) Student’s Ending Point of Achievement … sets one end of the measurement at the students’ entry point, not at an external standard. How much gain?

Factors related to student learning - District, School, and Teacher Influence on Student Progress Following inferences were shared at the Governors Education Symposium (2004) Following inferences were shared at the Governors Education Symposium (2004) Based on 22 years of Value-Added Study, Dr. Sanders draws the following conclusions: Based on 22 years of Value-Added Study, Dr. Sanders draws the following conclusions: Variation in student academic progress can be attributed this way: Variation in student academic progress can be attributed this way: 5% attributed to District quality 5% attributed to District quality 30% attributed to School quality 30% attributed to School quality 65% attributed to Teachers quality 65% attributed to Teachers quality

Difference in student achievement levels after comparably matched 2 nd graders experience qualitatively different teacher sequences of effectiveness, TN L=Least EffectiveH=Highly Effective (Sanders & Rivers, 1996)

Difference in student achievement levels after Experiencing qualitatively different teacher sequences of effectiveness, Dallas Source: Heather Jordan, Robert Mendro, & Dash Weerasinghe, “Teacher Effects On Longitudinal Student Achievement” Beginning 3rd Grade Score (Percentile) Average Math Score 3 yrs later in Percentiles Dallas Students Assigned To 3 Highly Effective Teachers In A Row Dallas Students Assigned To 3 Ineffective Teachers In A Row

Difference in student achievement levels after experiencing qualitatively different levels of teacher effectiveness, Boston -0.6% 14.6% 0.3% 5.6% Average Student Growth Over One Year (percent) Least Effective Teachers Most Effective Teachers Source: Boston Public Schools, in “High School Restructuring,” March 9, (courtesy of The Education Trust) READING MATH

Rivers (1999) Can varying sequences of teacher quality vary student chances of passing a high stakes test? Can varying sequences of teacher quality vary student chances of passing a high stakes test? o Rivers (1999) followed students from 4 th through 9 th grade from two large urban districts. o Students were grouped into quartiles on the basis of achievement on the 4th grade standardized test. o Math teachers were identified for grades 5-8 and were linked to effectiveness levels (Low, Avg, High), based on their success in facilitating value- added achievement with a prior group of students. o Do these quality distinctions apply to another cohort of students and offer reliable information about how a student will do on a high stakes in the future?

Probability of Passing High School Exam based on Sequence of Teacher Effectiveness Experienced

Jim Lloyd_2006 How is value added different than what we’re currently doing? Current system measures only achievement (what percentage of kids pass the test). Current system measures only achievement (what percentage of kids pass the test). New system measures progress towards the standard, so can more accurately reflect the growth of ALL students, especially those who may be hard-pressed to ever reach the passing rate. New system measures progress towards the standard, so can more accurately reflect the growth of ALL students, especially those who may be hard-pressed to ever reach the passing rate.

Jim Lloyd_2006

How will this help me as an educator? Measures growth from where the student enters the grade level. Measures growth from where the student enters the grade level. Measures progress at building/grade levels. Measures progress at building/grade levels. Shows if programs/instruction are impacting student growth. Shows if programs/instruction are impacting student growth. Aligns classroom resources to help all. Aligns classroom resources to help all.

Jim Lloyd_2006 EVAAS How to view a summary report

Jim Lloyd_2006 Concepts to Remember This isn’t about proving…it’s about improving. This isn’t about proving…it’s about improving. The power of the scores is in the sample size. The power of the scores is in the sample size. The scores become less reliable as the size of the population decreases. The scores become less reliable as the size of the population decreases. We will not be looking at the individual growth of individual students because it is not reliable. We will not be looking at the individual growth of individual students because it is not reliable.

Jim Lloyd_2006 Colors to Remember A score of “0” isn’t bad because we have a 0 growth model. A score of “0” isn’t bad because we have a 0 growth model. Red isn’t good Red isn’t good Yellow is no detectable difference Yellow is no detectable difference Green is good Green is good

Jim Lloyd_2006 Terms Mean NCE Gain—the average gain that the group achieved compared to where they started. This is the growth measure for the entire group of kids who have at least 2 data points. Mean NCE Gain—the average gain that the group achieved compared to where they started. This is the growth measure for the entire group of kids who have at least 2 data points. Standard Error—a statistical calculation as to how confident we are that the score didn’t occur due to error. Standard Error—a statistical calculation as to how confident we are that the score didn’t occur due to error. Impacted by sample size and dispersion Impacted by sample size and dispersion Ohio’s model uses 1 SEM for significance Ohio’s model uses 1 SEM for significance

Jim Lloyd_2006 Terms cont. State Base Year—the average amount of growth that happened by the grade in Ohio (expressed in NCE units) State Base Year—the average amount of growth that happened by the grade in Ohio (expressed in NCE units) 2005 Mean—the average amount of achievement that the students came in with Mean—the average amount of achievement that the students came in with Mean—the average amount of achievement that the students left with Mean—the average amount of achievement that the students left with. The amount of growth is the difference between the 2005 and The amount of growth is the difference between the 2005 and 2006.

Types of Reports Composite Composite Diagnostic Subject Specific Diagnostic Subject Specific Reading Reading Math Math Performance Subject Specific Performance Subject Specific Reading Reading Math Math Subgroup specific (reading/math) Subgroup specific (reading/math) Jim Lloyd_2006

Composite Reports Jim Lloyd_2006

Composite Report Shows the total amount of growth by combining all achievement tests administered. Shows the total amount of growth by combining all achievement tests administered. We’re going to look at our composite report now. We’re going to look at our composite report now.

Jim Lloyd_2006 Building Summary Report There are summary reports for each building as well that are subject specific. There are summary reports for each building as well that are subject specific. Here is reading. Here is reading.

Jim Lloyd_2006

Subject Specific Reports Jim Lloyd_2006

Subject Specific Reports Shows the amount of growth obtained in a particular subject area. Shows the amount of growth obtained in a particular subject area. We’re going to look at a Summary Reading and Math Report now. We’re going to look at a Summary Reading and Math Report now.

Jim Lloyd_2006 What is this? Growth standard of 0 is what we would expect if the student made a year’s growth

Jim Lloyd_2006 EVAAS How to view a Diagnostic Report

Jim Lloyd_2006 Pos Gain Neg Gain This shows us the # of OFCS students in each quintile Total # of students in the testing pool is divided into 5 groups called quintiles A “0 gain model” means that if students made about a year’s worth of growth we would see a gain within +1 or -1 of the standard error

Jim Lloyd_2006 Note that without at least 5 in a group, no data is reported

These students all have plenty of room to grow…even those at the 97 th NCE.

Jim Lloyd_2006 Example— Scattered spread w/a low progress spike

Jim Lloyd_2006

EVAAS Math Reports District Reports 2006 Current 6 th Grade Students

Jim Lloyd_2006 Building Summary Report There are summary reports for each building as well that are subject specific. There are summary reports for each building as well that are subject specific. Here is mathematics. Here is mathematics.

Jim Lloyd_2006 Math Diagnostic Reports

What is the gain pattern? Remember to GAIN & SEM

Implications of EVAAS Questions What sorts of organizational strategies will lead to increasing district, building and teacher effectiveness? What sorts of organizational strategies will lead to increasing district, building and teacher effectiveness? What are the professional development implications of instituting EVAAS in Olmsted Falls? What are the professional development implications of instituting EVAAS in Olmsted Falls? What is the short and long term plan to prep the administrative team and building staff for this additional accountability system? What is the short and long term plan to prep the administrative team and building staff for this additional accountability system?Responses Teachers and principals make the difference. Build capacity! Teachers and principals make the difference. Build capacity! Make a concerted effort to get ahead of the curve and use this and other data to our advantage. Make a concerted effort to get ahead of the curve and use this and other data to our advantage. Foster and reinforce a data- driven culture across the district. Foster and reinforce a data- driven culture across the district. Build upon and expand our staffs’ capacity to implement and monitor a standards-based educational program for all students. Build upon and expand our staffs’ capacity to implement and monitor a standards-based educational program for all students.

Jim Lloyd_2006 Questions?