1. Project on Educator Effectiveness & Quality Chancellor Summit September 27, 2011 Cynthia Osborne, Ph.D.

2. Accountability System for Educator Preparation Programs (SB 174)  Standard 1: Certification Exam Passing Rates  Standard 2: Principal Survey of 1 st Year Teacher; Graduates’ Survey of Preparedness  Standard 3: Teachers’ Influence on Growth in Student Achievement in First Three Years Post Certification  Standard 4: Quality of the Field Supervision September 27, 20112

3. Purpose of PEEQ  Tasked by TEA to design a metric to determine EPP’s graduates’ influences on student achievement  Metric is required due to: o SB 174 student achievement standard (#3) o State longitudinal data system grant (SLDS II) o School improvement grant (SIG) o State fiscal sustainability fund (SFSF)  Will focus only on 1st through 3rd year teachers  Objective is to provide feedback to EPPs to improve teaching and student performance – not just for accountability September 27, 20113

4. SB 174 and PEEQ September 27, 20114 SB 174Standard III Please note that the weights of SB174 and the weights of Standard III have not been determined.

5. What Predicts Effective Teaching?  Years of experience: teachers show 2/3 of all improvement in first 5 years  General intelligence: selectivity of the college, college entrance exams, teacher licensing exams, and IQ tests  Certification in subject area (especially math): masters degree is not predictive  Rigorous observations of teachers in classroom  Most variation in teacher effectiveness is not explained by “teacher inputs” – focus has shifted to “student outcomes” September 27, 20115

6. Measuring Teacher Effectiveness in Texas  Our current system, PDAS, does not provide information to the state or EPPs on effective teachers  Over 98% of teachers are rated proficient, thus EPPs have no indicators to improve programming  NCLB standards (highly effective) are inconsistent with PDAS findings; students are not all proficient  Goal is to improve the quality of the pipeline of new teachers in the state by providing EPPs with information on how their graduates are influencing student achievement September 27, 2011

7. Measuring Teacher Effectiveness Across the Nation  States and districts are increasingly using student performance to measure teacher effectiveness  Have moved from a system of measuring “teacher inputs” to measuring “student outcomes”  31 states have initiated plans  States are moving quickly from plan to implementation; LEAs move more slowly  Wide variety exists in components measured and instruments used  Few states use teacher effectiveness for EPP accreditation – although momentum is gaining September 27, 2011

8. An Effective Metric Should Be:  Comprehensive: Includes multiple sources; not just test scores  Useful: Leads to improvement in EPPs and teaching  Valid & Reliable: Reflects student achievement and considers variations in populations of students; Problems with measurement error, test scaling, and data quality must be minimized  Integrated: EPPs held accountable for training teachers to do what teachers will be held accountable for doing  Transparent: Clear how components are measured and weighted September 27, 20118

9. Decisions PEEQ Must Make  What components should be included in a comprehensive metric to determine a teacher’s effect on student achievement? How should we weight the components?  What is the most valid and reliable method to measure student performance using test scores? September 27, 20119

10. A Comprehensive Metric Should Include:  Growth in Student Performance on TAKS (VAM)  Observations of Teachers in Classroom o Classroom Environment o Curriculum and Instruction  Principals’ Assessment of Influence on Achievement  Teacher’s Impact on Student Achievement  School or Grade Level Growth in Performance September 27, 201110

11. Weighting of Components Will Be Determined Empirically September 27, 201111

12. PEEQ’s Metric September 27, 201112

13. Measuring Student Performance of EPPs’ Graduates  Common approaches o Percent of students passing standardized test o Percent increase in students passing test  Strengths o Straight forward and transparent o Common standard and expectation of knowledge  Limitations o Does not account for a student’s prior knowledge or growth o May lead to focus on teaching to test September 27, 201113

14. Measuring Student Performance of EPPs’ Graduates  Value Added Models o Goal is to predict what a student should score and compare to actual performance – “teacher effectiveness” is the difference o Various methods used to calculate o Require a large enough sample to create comparisons o Need sufficient data to accurately predict student performance o Attribution of students to teachers is crucial o Tests must be vertically aligned to show growth o PEEQ will use a VAM, but not at individual teacher level, yet September 27, 201114

15. Measuring Student Performance of EPPs’ Graduates  Value Added Models (Continued)  Strengths o Account for past performance of students and student - and school-level characteristics that may affect teaching and student performance o Emphasis is on growth in student performance o Teachers’ past effectiveness is highly predictive of future  Limitations o Difficult to understand calculations (less transparent) o Measurement error may bias estimates September 27, 201115

16. Observations of Teachers  Must be mindful of the scale: o 75,000 beginning teachers (in first 3 years) o 1,400 school districts  Principals are required to observe beginning teachers and fill out a survey on each teacher (ASEP)  This survey provides the platform to gather information on teachers’ classroom environment and instruction (elements linked to student achievement) September 27, 201116

17. Classroom Environment To what extent is this beginning teacher able to... (4 pt scale)  Effectively implement discipline-management procedures?  Communicate clear expectations for achievement and behavior that promote and encourage self-discipline and self-directed learning?  Provide support to achieve a positive, equitable, and engaging learning environment?  Build and maintain positive rapport with students?  Build and maintain positive rapport and two-way communication with students’ families? September 27, 201117

18. Instruction To what extent is this beginning teacher able to... (4 pt scale)  implement varied instruction that integrates critical thinking, inquiry, and problem solving?  respond to the needs of students by being flexible in instructional approach and differentiating instruction?  use the results of formative assessment data to guide instruction?  engage and motivate students through learner-centered instruction?  integrate effective modeling, questioning, and self-reflection (self-assessment) strategies into instruction?  assume various roles in the instructional process (e.g. instructor, facilitator, audience)?  set clear learning goals and align instruction with standards-based content?  provide quality and timely feedback to students? September 27, 201118

19. Principal Assessment  “How would you rate this teacher’s influence on student achievement?”  10 point scale: unacceptable to exceptional September 27, 201119

20. Information to EPPs for Standard #3  EPPs will get a total score, and disaggregated information on graduates  Information to EPPs will include: o Overall effectiveness o Aggregate scores by each metric component o Scores by: Certification program Type/location of school Grade level/subject matter September 27, 201120

21. Feedback from Districts/Teachers  Want better indicators of effectiveness; have concerns about the uses of the metric  Claim that quality of field supervision distinguishes EPPs  Encourage a stronger dialog and more interaction between districts and EPPs to align teaching and practice September 27, 201121

22. Feedback from EPPs  Deans want better indicators of effectiveness of graduates  Concerned about variation in support teachers receive across districts  Concerned that information has not previously been available; would like information for two years prior to accountability  No consensus on what new teachers should know relative to experienced teachers September 27, 201122

23. We want your input!  We have a statistical advisory group and invite each EPP to designate a member.  We conduct quarterly stakeholder meetings and invite all stakeholders to participate.  We update our website regularly with new research briefs: www.utexas.edu/lbj/peeq September 27, 201123

24. Expected Timeline  Principal Survey (Standard #2) administered in summer 2011  Data linking teachers and students available fall 2011  Pilot Metric available spring 2012  Pilot Metric presented to SBEC June 2012 September 27, 201124