1. 2010 OSEP Leadership Mega Conference Collaboration to Achieve Success from Cradle to Career Challenges in Linking Special Education Teacher Evaluation to Student Performance Tony Bagshaw, Battelle for Kids Bill East, National Association of State Directors of Special Education Lynn Holdheide, Vanderbilt University/National Comprehensive Center for Teacher Quality Bambi Lockman, Florida Department of Education Dan Reschly, Ph.D., Vanderbilt University/National Comprehensive Center for Teacher Quality S4-203 and S4-208

2. About the National Comprehensive Center for Teacher Quality The National Comprehensive Center for Teacher Quality (TQ Center) is a federally funded partnership whose mission is to help regional comprehensive centers and states carry out the teacher quality mandates of the Elementary and Secondary Education Act (ESEA) as reauthorized by the No Child Left Behind Act. Partners: – Learning Point Associates – Vanderbilt University – ETS 2

3. Today’s Goals Seeks to build the capacity of participants to – Articulate the challenges identified with evaluating special education teachers through value-added and other measures of teacher evaluation. – Actively participate in the creation or redesign of teacher evaluation models that support the development of strong, valid and reliable teacher evaluation policies and practices that recognize and promote the unique contribution of special education teachers. 3

4. What do you know? Turn to someone at your table. Share anything you know about what is going on with teacher evaluation in your state – with specific emphasis on how special education teachers are evaluated

5. Changing Teacher Evaluation Policy and Practice Consensus that teacher quality impacts student achievement Teacher evaluation seen as a “policy lever” to improve teacher performance Race to the Top (RTTT) spotlight on teacher evaluation systems that focus on effective teaching and improved student achievement. 5

6. The Purpose TQ Research & Policy Brief: Challenges in Evaluating Special Education Teachers and English Language Learner Specialists (http://www.tqsource.org/publications/July2010Brief.pdf) – Identify the specific challenges in evaluating this population of teachers. – Determine the current status of state policy and practice. – Identify promising evaluation practices and instruments. – Provide guidance and policy recommendations to districts and states. 6

7. The Inquiry Review of policy/literature Survey inquiry Series of interviews with state- and district-level practitioners and researchers Data collection period: December 2009–April 2010 7 Designed in collaboration with Council for Exceptional Children (CEC) and national experts State and local survey Respondent pool: state and local directors (identified within CEC’s Council of Administrators of special education listserve) 1,143 total respondents

8. Thoughts and Experience If you were to observe an excellent teacher what might you see or hear?

9. Modification of Evaluation Processes for Special Educators Among the local administrators, reported that contractual agreement prevented modification in the evaluation process. 9 81%

10. Opinions Regarding Special Education Teacher Evaluation 10 Strongly Agree or Agree 84% 92% 32%

11. Evidence-Based Practices Meeting the needs of “diverse” learners may not attend to the following: – Special skills ( individualized education program [IEP] facilitation, collaboration, secondary transition, social and behavioral interventions, compliance with legal mandates) – Evidence-based instructional methods (direct/explicit instruction, scientifically based reading instruction, learning strategy instruction) 11

12. Multiple Measures 12 Use more than one measure. 95+% Survey inquired about current practice. Respondents indicated value-added models in future evaluation efforts.

13. Practical Example: District of Columbia IMPACT Individual Teacher Value-Added Scores Non-Value-Added Achievement Teaching and Learning Framework Commitment to the School School Value-Added Scores Core Professionalism IEP Quality Plan IEP Timeliness 13 Special Education 10% 50% 15% 5% 15%

14. Opinions Regarding Use of Student Achievement for Special Educators 14 Strongly Agree or Agree 73% 60% 21%

15. Challenges in Using Growth Models A research-derived value-added model for special educators does not exist Small student samples commonly associated with special education caseloads Student mobility Use of accommodations Students assessed on alternate standards Teacher attribution in a coteaching situation 15

16. Student Growth Measures Practical Examples Austin Independent School District, Texas – Student Learning Objectives One is targeted toward classroom performance. One is targeted toward particular skills or subgroups of students. Norwell Public Schools, Massachusetts – Progress on the IEP is factored into evaluation of special educators. Both districts are heavily dependent on teacher training and support. 16

17. Observation Protocols 17 Use the same observation instrument as that of general education teachers. Use a modified or different observation instrument. 85% 12% Align to the state’s professional teaching standards. 26% Didn’t know. 51% “Our evaluation tool was developed in the district over 40 years ago.” “Our current evaluation system is outdated and applied to nothing.”

18. Observation Protocol Practical Example Alabama Department of Education’s Professional Education Personnel Evaluation Program – Slightly modified for the following: Specialty area systems (speech paths, library specialist) Teachers of students with significant cognitive disabilities – Competencies added in certain areas (e.g., classroom is expanded to include community settings, and academic content is expanded to include functional life skills.) 18

19. Expert Opinions Regarding Evaluators 19 Strongly Agree or Agree 77% 61% 12% 60% Require training Require specialized training

20. Practical Examples Toledo’s Peer Assistance and Review – School-based teams evaluate. Norwell Public Schools, Massachusetts – All teachers are evaluated using the same instrument. – Two formative assessments are conducted: One with principal One with special education administrator – Each evaluator focuses on expertise areas. – Both work collaboratively to develop summative evaluation. 20

21. Opinions Regarding Attribution in Coteaching Setting 21 Strongly Agree or Agree 13% 85% 75%

22. About Battelle for Kids National, service- and solution-based, not-for-profit organization Started in 2001 with funding from Battelle Memorial Institute Headquartered in Columbus, Ohio Mission-driven team of education, technology, communications and business professionals

23. The Right Measures –Improving data quality and reporting The Right Practices –Providing professional development (consulting, tools and online learning) –Using value-added analysis, formative assessment and the BFK  Focus™ process The Right People –Identifying and sharing the practices of highly effective teachers –Developing strategic compensation and evaluation models –Communicating and managing change What We Do

24. Yes, within reason –No snap judgments, look across time - Repeatability reaches.8 with a three-year average –Balance the value-added data with other measures - Everything that counts can’t be counted –Beware of unclean data… education data is notoriously noisy, we are now trying to use educational data systems for purposes for which they were not designed –Be a smart consumer of data…sample size matters –Simple models vs. complex models… you make tradeoffs and to a great degree it depends on the end use of the data - “Simpler is better unless it is wrong” Is Value-Added Data the Way To Go?

25. BFKLink™ □Hundreds of thousands of assignments, and over one million students, have been successfully linked. □Thousands of teacher-level value-added reports have been generated for educators. □99.37% successful linkage rate—connecting tests to students, teachers, teams and buildings—that will allow more value-added reports to be generated. □Solves the data accuracy and the data transparency issue.

26. BFKLink™ − Getting Started Principal Set-Up Period

27. Manage Your Linkage Support Team Support team members can manage (add/remove) staff, staff classes and rosters, and modify linkages as necessary.

28. Teacher Linkage Teachers begin the linkage process by viewing a list of all their classes for tested subjects requiring linkage.

29. Reviewing and Modifying Class Rosters Class rosters must be reviewed and accurately completed.

30. Percentage of Instruction (Mobility) Mobility information is combined with % of instruction information. In Advance Mode, nine separate months of instruction are collected.

31. Battelle for Kids’ Philosophy Balance is the key: Multiple data sources/measures Across time Linking teaching instruction to students Informing instruction based on data Student success in college, workplace and life

32. www.BattelleforKids.org

33. 2010 OSEP Leadership Mega Conference Florida’s Educator Quality Update Bambi J. Lockman, Chief Bureau of Exceptional Education and Student Services Florida Department of Education 33

34. Educator Quality Topics Data Florida Educator Accomplished Practices – Status of Process – Implementation implications Race to the Top – Great Teachers and Leaders activities Opportunities for Engagement 34

35. 35 Professional Instructional Staff Florida’s Public Schools, Fall 2009 Source: EIAS, Staff in Florida’s Public Schools, Fall 2009

36. 2010 OSEP Leadership Mega Conference Florida Educator Accomplished Practices (FEAP) 36

37. FEAP Established 1998 – SBE Rule 6A-5.065 Serve as state standards for effective instructional practice Define and identify effective teaching Currently form the basis for Florida Preparation Programs Foundation for core practices for teacher appraisal systems under RTTT Plan 37

38. FEAP Process FEAP Revision webpage: http://www.fldoe.org/profdev/FEAPSRevisions/ http://www.fldoe.org/profdev/FEAPSRevisions/ Working Group - July 2010 – Charge #1: develop Draft 2 from public comments on Draft 1 and from research base – Charge #2: recommend Draft 3 after public comment from Draft 2 is completed Draft 3 - SBE adoption - November 2010 38

39. FEAP Implementation Implement the following: – District instructional personnel evaluation systems – College of education, Educator Preparation Institute, and district alternative certification teacher preparation programs: candidate and completer performance Provide Technical Assistance 39

40. 2010 OSEP Leadership Mega Conference State Board of Education (SBE) Race to the Top 40

41. SBE Strategic Imperative Improve the Quality of Teaching in the Education System Objectives: Establish the state’s expectations for quality instructional practice (FEAP) Improve the quality of preparation programs, professional development, and certification exams Align requirements for district performance appraisal to the state’s expectations and student achievement goals Provide statewide recognition and award programs that reward outstanding performance based on the state’s expectations and student achievement goals 41

42. Teacher Quality Performance Measures Percentage of classes taught by out-of-field teachers 2007-08 Overall: – 8.3% = 79,985 of 964,718 Total Classes 2008-09 Overall: – 7.7% = 83,300 of 1,078,618 Total Classes Percentage of HQ teachers 2005-06 Overall: 89.6% 2008-09 Overall: 93% 42

43. Teacher Quality Performance Measures Number of approved teacher preparation programs and program completers Approved Programs: 469 Initial Teacher Preparation Programs 33 Educator Preparation Institutes +72 District Alternative Certification Programs 574 Approved Teacher Preparation Programs Program Completers Overall: 2007-08 = 8,860 2008-09 = 9,705 Program Completers in Critical Shortage Areas: 2007-08 = 1,932 2008-09 = 2,312 43

44. 44 9,705 individuals completed a Florida approved program during the 2008-09 school year 5,090 of them were employed in instructional positions in 2009-10 5,770 1,716 1,374

45. The Next Level – From Qualified to Effective Step 1 - Evaluate and adopt teacher-level effectiveness measures: – Instruments for assessing and methods for calculating student learning growth – Indicators of effectiveness based on observable practices that are closely associated with measurable student growth Step 2 - Set performance targets for individual teacher and principal evaluations and for approval of teacher/principal preparation programs – Establish by State/districts during coming years 45

46. 46 Overall Teacher Performance in Reading and Math Combined in Title I and Non-Title I Schools 2008-09 School Type Total Number of Teachers Number of Teachers at 50% LG Mark Percent of Teachers at 50% LG Mark Number of Teachers at 75% LG Mark Percent of Teachers at 75% LG Mark Number of Teachers at 125 VT or Higher Percent of Teachers at 125 VT or Higher Title I27,34222,44882.17,80328.54,64017 Non- Title I48,12839,48782.016,82034.912,68226.4 Three Thresholds: 1.50% or more of students making learning gains 2.75% or more of students making learning gains 3.Value Table score of 125 or higher (placing teacher in about the top 20% of teachers in the state)

47. 47 Green Overall 2007-08 Completer Performance in Reading 2008-09 Note: Green highlight shows programs that met or outperformed the state Level and Subject Taught Percentage With 50% or Higher Learning Gains Percentage With 75% or Higher Learning Gains Percentage With a Value Table Score of 125 or Higher All Teachers in the State Elem. Reading 91%40%22% Middle Reading 91%24%21% High Reading 38%4%21% ITP Elem. Reading 88%32%14% Middle Reading 91%23%19% High Reading 37%1%16% EPI Elem. Reading 93%31%17% Middle Reading 91%16%15% High Reading 31%0%16% DACP Elem. Reading 83%23%14% Middle Reading 90%15%13% High Reading 35%2%19%

48. 48 Green Overall 2007-08 Completer Performance in Math 2008-09 Note: Green highlight shows a program that met or outperformed the state Level and Subject Taught Percentage With 50% or Higher Learning Gains Percentage With 75% or Higher Learning Gains Percentage a Value Table Score of 125 or Higher All Teachers in the State Elem. Math 84%34%30% Middle Math 83%44%24% High Math 91%47%11% ITP Elem. Math 81%29%23% Middle Math 79%39%15% High Math 89%40%6% EPI Elem. Math 77%31%27% Middle Math 84%41%29% High Math 82%35%6% DACP Elem. Math 85%28%24% Middle Math 82%36%19% High Math 96%43%3%

49. Opportunities for Engagement FEAP - Draft 2 and 3 input SBE - What’s Working Series (Fall 2010) Race to the Top Workgroups 49

50. The shortest distance between two points is under construction.  Noelie Altito

51. Bambi J. Lockman, Chief Bureau of Exceptional Education and Student Services Florida Department of Education www.fldoe.org (850) 245-0475 51

52. Policy and Practice Recommendations Include special education administrators in the revision/design of evaluation frameworks Identify a common framework that defines effective teaching for all teachers, differentiate as needed for special educators Integrate evidence-based practices for students with disabilities into evaluation models 52

53. Policy and Practice Recommendations Improve data quality In addition to - or, in some situations, in the absence of - appropriate standardized assessment data: – incorporate other reliable evidence of teachers’ contributions to student learning into the teacher evaluation system (i.e. progress toward accomplishing IEP objectives and student learning objectives across broad academic and behavioral domains) 53

54. Policy and Practice Recommendations Ensure that evaluator training includes: – evaluators of special educators and/or – a model of peer-to-peer observations or – a model in which evaluators are matched to specific disciplines Collaborate with teacher preparation programs to ensure that evidence-based practices are incorporated into teacher preparation coursework and professional development activities. 54

55. Evidence-based Practices in Special Education Scientifically based instruction ESEA (2002) and IDEA (2004) IES criteria and evolution to evidence-based practices Research supported evidence-based practices in special education – ABA and its many variations/application – Direct instruction, big D and little d, reading and mathematics – Formative assessments with instructional decision making – Learning strategies 55

56. 56 What Works? Research Foundations From Meta-Analysis Treatment Effect Size Applied Behavior Analysis + 1.00 Formative Evaluation: Curriculum-Based Measurement+Graphing+Decision Rules+Reinforcement + 1.00 Explicit Instruction and PS +.70 to 1.50 Comprehension Strategies + 1.00 Mathematics Interventions +.60 to 1.10 Writing Interventions +.50 to.85 Matching instruction to learning styles? 0.00 Sources: Kavale, 2005Note: These effect sizes are stable across cultural groups.

57. Scientifically Based Instruction in Reading Reading Curricula content (Snow, Burns, & Griffin, 1998) – Phonemic Awareness – Phonics – Fluency – Vocabulary – Comprehension Problem of teacher preparation Vanderbilt University/TQ Center innovation configurations: reading, classroom behavior, inclusive services, learning strategies (Reschly, Smartt, & Oliver, 2007) 57 PLUS Direct, systematic instruction Universal screening and formative evaluation

58. Learning to Read: The Great Debate (Chall, 1967) Research review 1900–1965 Early reading, K–3 Code versus meaning emphasis Phonics or whole word Code superior, especially for struggling readers Lamented the generally poor preparation of teachers to teach reading 58

59. National Council on Teacher Quality: Reading Components Taught Well 59 Percentage Number of Components 11% N=8 7% N=5 11% N=8 13% N=9 43% N=31

60. NCES Reading Report Card 2009: Categories < Basic: Less than partial mastery of prerequisite knowledge and skills fundamental to proficient work at the grade level Basic: Partial mastery of … Proficient: Solid academic performance and demonstrated competency over challenging subject matter Advanced: Superior performance Source: National Center for Education Statistics, 2009 60

61. 61 Reading 2009 Grade 4 Percentage Source: National Center for Education Statistics, 2009, Table A-12

62. Preparation of Special Education Teachers in Scientifically Based Reading Instruction in 27 Institutions of Higher Education (IHEs) 62

63. Reading Course Syllabi: Projects Explain your philosophy of literacy. Develop bulletin board to motivate children to read. Produce journal explaining personal experience in learning to read. Analyze the social justice implications of literacy. 63

64. 64 TQ Research & Policy Brief  Smartt, S. M., & Reschly, D. J. (2007). Barriers to the preparation of highly qualified teachers in reading (TQ Research & Policy Brief). Washington, DC: National Comprehensive Center for Teacher Quality. 64 http://www.tqsource.org/publications /June2007Brief.pdf

65. Current Special Education Practice With High- Incidence Disabilities Review of special education case files for the state of Washington for a court case Randomly selected 900 special education students – Ten districts were represented. – All students were in special education for at least 12 months. – Case files varied from 50 to 1,100 pages. – The review evaluated individualized education programs (IEPs) using checklist for required components and evidence of formative evaluation. – How many graphs? 65

66. Review of Special Education Case Files: Results Little evidence of systematic, direct instruction or behavior interventions using problem solving Assessment and formative evaluation nearly nonexistent (11 of 870 cases had graphs.) Lots of test protocols documenting weaknesses Little objective evidence of positive outcomes (i.e., benefits of special education are largely undocumented in high incidence.) No assessment of progress 66

67. National Mathematics Advisory Panel (2008) Report Key Findings Conceptual understanding, computational and procedural fluency, and problem solving skills are equally important and mutually reinforce each other. Students should develop immediate recall of arithmetic facts to free the “working memory” for solving more complex problems. Teachers’ regular use of formative assessments can improve student learning in mathematics. 67

68. National Mathematics Advisory Panel (2008) Report Key Findings Explicit instruction for students who struggle with mathematics is effective in increasing student learning. Teachers should understand how to provide clear models for solving a problem type using an array of examples, offer opportunities for extensive practice, encourage students to “think aloud,” and give specific feedback. 68

69. Summary: Teacher Preparation and Practice Insufficient use of evidence-based practices in teacher preparation/comprehensive professional development and practice TQ Center use of evidence-based innovation configurations to address these issues (See the TQ Center Special Education Resource List.) Improved implementation of evidence-based principles leading to improved outcomes Major Challenge: Narrowing the gap between what is known about evidence-based instruction and teacher preparation and special education practice 69

70. References Chall, J. S. (1967). Learning to read: The great debate. New York: McGraw-Hill. Kavale, K. (2005). Effective intervention for students with SLD: The nature of special education. Learning Disabilities, 13(4), 127–138. National Center for Education Statistics. (2009). The nation’s report card: Reading 2009─National Assessment of Educational Progress at Grades 4 and 8 (NCES 2010-458). Washington, DC: Institute of Education Sciences, U.S. Department of Education. Retrieved July 1, 2010, from http://nces.ed.gov/nationsreportcard/pdf/main2009/2010458.pdf http://nces.ed.gov/nationsreportcard/pdf/main2009/2010458.pdf National Mathematics Advisory Panel. (2008). The final report of the National Mathematics Advisory Panel. Washington, DC: U.S. Department of Education. Retrieved July 1, 2010, from http://www.ed.gov/about/bdscomm/list/mathpanel/report/final- report.pdf http://www.ed.gov/about/bdscomm/list/mathpanel/report/final- report.pdf 70

71. References Reschly, D. J., Smartt, S. M., & Oliver, R. M. (2007). Innovation configurations to improve teacher preparation in reading, behavior management, and inclusive practices. In C. A. Dwyer (Ed.), America’s challenge: Effective teachers for at-risk schools and students (pp. 23– 45). Washington, DC: National Comprehensive Center for Teacher Quality. Smartt, S. M., & Reschly, D. J. (2007). Barriers to the preparation of highly qualified teachers in reading (TQ Research & Policy Brief). Washington, DC: National Comprehensive Center for Teacher Quality. Retrieved July 1, 2010, from http://www.tqsource.org/publications/June2007Brief.pdf http://www.tqsource.org/publications/June2007Brief.pdf Snow, C. E., Burns, M. S., Griffin, P. (Eds.). (1998). Preventing reading difficulties in young children. Washington, DC: National Academy Press. Walsh, K. Glaser, D., & Wilcox, D. D. (2006). What education schools aren’t teaching about reading and what elementary teachers aren’t learning. Washington, DC: National Council on Teacher Quality. Retrieved July 1, 2010, from http://www.nctq.org/p/docs/nctq_reading_study_app.pdf http://www.nctq.org/p/docs/nctq_reading_study_app.pdf 71