1. PVAAS Teacher Specific Reporting: Professional Growth, Not Gotcha! 1 PVAAS Statewide Team for PDE November/December 2014 Half-Day Sessions

2. Introductions − PVAAS Statewide Team for PDE − IU Support Session Materials − PPT (electronic copy) − Supplemental Handout (paper copy) − Session Feedback − “Teacher to Teacher” − Laminated PVAAS Sample Reports – Please leave on table for next group − White envelope containing Activity Materials – Please leave on table for next group 2

3. Who is Here Today? Teachers with Teacher Specific Reports? Teachers without Teacher Specific Reports? Administrators? Other? Who is new or relatively new to PVAAS? Who has already accessed his/her teacher specific reports? 3

4. Today’s Purpose Promote your knowledge and skills in using PVAAS Teacher Specific Reports to improve performance for all students – You will be able to : Access & navigate through your reports Understand your reports Use the reports to reflect on professional practices GROWTH…GROWTH…GROWTH….GROWTH 4

5. HOW IS PVAAS USED IN PENNSYLVANIA? School accountability through the School Performance Profile (SPP) Teacher evaluation (PVAAS 3 Year Rolling Avg.) PVAAS has been and continues to serve as a tool for LEA/district and school improvement Now, annual tool for teachers to reflect on professional practices

6. Why Does This Matter to Me? Provides me with annual information about the growth of the students for whom I had full or partial responsibility last year Allows me to reflect on last year, and use “looking forward” reports to bridge to this year’s students Provides me with information to reflect on my practices in terms of how they may have impacted student growth 6

7. Recognizing the Challenges PVAAS Teacher Specific Reporting is new – (273 LEAs - SY12-13 Pilot) – PVAAS District/School Reporting is not new (SY05-06) May or may not be what you expected One source of data and one year of data Measures the academic growth of students of what our state assessment assesses Hold those thoughts - put them aside - hang with us to learn some new information. 7

8. The Challenges of Self-Reflection Requires willingness to examine one’s own beliefs & practices Requires systematic “digging deeper” process to analyze what might be contributing to what is seen on growth reports – Too often we stop with our first gut reaction - must ask array of questions/explore various hypotheses Requires learning from what’s working to apply to identified areas of need 8

9. So, what do you hope to learn today? What are your top 2-3 questions?

10. Questions 10

11. TYPES OF REPORTS PVAAS: Growth, Not Gotcha! 11

12. HOW DOES PVAAS WORK? District School Teacher Student Note: All Prior Data are Used *Writing used in District & School Reporting

13. Types of Teacher Specific Reports 13

14. A Process for Reviewing and Planning 14 Guide you through a process today to get started Page 3 in Handout

15. Teacher Value Added Summary 15 What Growth Color Indicators are you seeing, and what do those colors mean to you?

16. PA Data Group of Students Students’ Prior Data Across Grades & Subjects Estimated Prior (or Entering) Achievement of the Group of Students Measuring Growth: Students’ Starting Point 16

17. Prior (or Entering) Achievement of the Group of Students Higher Achievement Lower Achievement Higher Achievement Lower Achievement Current Achievement of the Group of Students Concept of Measuring Growth In District & School Reporting, students are weighted equally. For Teacher Specific Reporting, students are weighted by taking into account the % Total Instructional Responsibility for each student. 17

18. Why Use Prior Data Across Subjects? o Multiple correlation coefficients between PSSA subjects, and between PSSA and Keystones are strong Correlations above +0.7 or below -0.7 are considered strong o Consider 2013-2014 data: Math using all prior scores  correlation 0.83 to 0.89 Reading using all prior scores  correlation 0.80 to 0.85 Science using all prior scores  correlation 0.78 to 0.82 Keystones using all prior PSSA scores  correlation 0.77 to 0.80 T 18

19. Teacher Value Added Summary 19 Where does the growth measure come from?

20. Prior (or Entering) Achievement of the Group of Students Higher Achievement Lower Achievement Higher Achievement Lower Achievement Current Achievement of the Group of Students Concept of Measuring Growth 20

21. Teacher Value Added Summary 21 Growth Index ≈ Growth Measure ÷ Standard Error

22. Student History Report Growth is about a group of students NOT an individual student 22 This report should NOT be used or interpreted as growth for an individual student

23. Turn, Talk, & Report What does green mean? Turn, talk, and report 23

24. Teacher Value Added Summary 24 The composite is a combined measure of all tested subjects, grades, and Keystone content areas in which a teacher received a value added report.

25. Composite v. Teacher Value-Added The growth color indicators are categories or ranges of indicators of growth. We cannot “average” categories. In other words, green + red ≠ yellow; likewise green + dark blue ≠ light blue. For PVAAS, the assumption is that the achievement level of the teacher's group of students is maintained (green) UNLESS there is enough evidence in the assessment data to say otherwise. It is all about the amount of evidence in the student assessment results – the assumption is that the achievement level of the teacher's group of students is maintained (green) UNLESS there is enough evidence in the assessment data to say otherwise. The more data available, the more evidence we have to see if the group of students exceeded the growth standard OR not. There is more evidence when all data was combined for a composite score. With data from multiple subjects (or multiple years when that’s available) included in the Composite, there’s more evidence/more data. 25

26. More Information About Methodology PVAAS Methodologies, Measuring Growth & Projecting Performance – PVAAS login page, under Professional Development Virtual Learning Modules – eLearning Link – Value Added Reports – Math/Reading – Value Added Reports – Science/Keystones – Teacher Specific Reporting EVAAS Statistical Models in PA – PDE PVAAS site, Methodology Page 26

27. Example (Grade 4 Math/Rdg/Science) 27 Next: What might account for what you are seeing? To start: Observe patterns; make note of observations. What kind of growth do you see? Was this expected? Does this align with what you expected to see? Where do you think this teacher might focus first?

28. Card Sort Activity Envelope – Card Sort Find the Achievement and Growth labels on the orange paper, these are the categories Sort the cards under the correct category 28

29. The reason the group did not make growth is because the students in my group are… – Low-achieving? – High-achieving? – Economically Disadvantaged? – Students with IEPs? Usually this concern comes from having an achievement focus, NOT a focus on growth Before Looking at Your Own Reports to Dig Deeper

30. All students can learn. All students deserve to make appropriate academic progress each year. Do you agree?

31. Let’s examine some evidence! 31

32. GROWTH VS. ACHIEVEMENT Each dot represents a teacher growth measure for PSSA Reading Grade 6 in 2014 GROWTH ACHIEVEMENT in NCEs (at beginning of subject/grade, or course) PVAAS Growth Color Indicators

33. PERCENT ECONOMICALLY DISADVANTAGED Each dot represents a teacher growth measure for PSSA Reading Grade 6 in 2014 Achievement Growth

34. PERCENT SPECIAL EDUCATION Each dot represents a teacher growth measure for PSSA Reading Grade 6 in 2014 Achievement Growth

35. Common Concerns & Responses 35 Page 14 in Handout Page 21 in Handout

36. 36 What is one take-away you have so far?

37. Step 1 C 37 Still on Step 1 Moving on to part 1C

38. Teacher Value Added Summary Is this report creating any thoughts/questions about your professional practice? Think about the Domains! 1. Planning and Preparation 2. Classroom Environment 3. Instruction Think about CIAO! 1. Curriculum 2. Instruction 3. Assessment 4. Organization 38 Page 8 in Handout

39. Example (Grade 4 Math/Rdg/Science) 39 How strong is your content knowledge? How are you sure you are teaching the big ideas and the important concepts? What formative assessment strategies do you use to assess ALL student levels? How do you ensure you are clearly communicating the purpose of the lesson? Do does my instruction reflect the “I DO, WE DO, YOU DO” gradual release of responsibility? Am I sure that I am teaching (input/modeling), providing sufficient guided practice, and independent practice for each learning target? Are you using the allocated time efficiently? Are my routines and procedures intact so as to minimize transition time?

40. PRACTICE INTERPRETING REPORTS 40

41. What might account for what you see on your Value-Added reports? 41 Handout pg 5 See the Digging Deeper questions in the Appendix

42. Practice Interpreting Reports Using the Value Added Summary Report 42 You choose: Use your own Value Added Summary Report One of the laminated demo reports Or log into the demo site using a demo accounts

43. Reflection Did you start by observing patterns & trends in your data? Did you consider a variety of questions across the Domains or CIAO categories? – How many questions did you generate? Did it make you want to dig deeper into the data? What questions do you still have about this part of the process? 43

44. The Next Step in the Process 44 Just finished with Step 1 Now on to Step 2

45. Types of Information 45

46. Diagnostic Summary Report Is there evidence of more growth with certain groups of students? High-achieving vs low-achieving Students with IEPs Other groups Reflect on the components within the four domains of the Framework for Teaching! 46

47. Do NOT factor in the percent of Instructional Responsibility Reflect the growth of a group of students who may have had more than one teacher with instructional responsibility in that specific subject/grade or course. See Concern # 5 for additional information Diagnostic Reports 47

48. Diagnostic Summary Report 48 What do the colors mean?

49. Diagnostic Summary Report 49 How are students placed into groups? Math & Reading Science & Keystones

50. Diagnostic Summary Report 50 Number of Students % of Students Growth for the group

51. Looking at Subgroups Note: New subgroups were added this year – (1) Historically Underperforming and (2) Non-Historically Underperforming 504 Plan & GIEP will be added in Dec. 51

52. Diagnostic Summary Report 52 Observe patterns with achievement groups or subgroups Do you know you students to the degree that you plan/prepare lessons based on the needs of the students? Am I planning coherent instruction that differentiates for groups of students? Am I using flexible grouping to meet the needs of specific groups of students? Were you the only teacher responsible for instruction or was there shared responsibility? What proportion of each group were higher, middle, or lower achieving? Have I allocated the appropriate amount of time to all levels of learners in order to meet their needs for direct instruction, guided practice, and independent practice?

53. Interpreting Diagnostic Summary Reports Using the Diagnostic Summary Report 53 You choose: Use your own Diagnostic Summary Report One of the laminated demo reports Or log into the demo site using a demo accounts

54. What might account for what you see on the Diagnostic reports? 54 Handout You choose – Use the actual Diagnostic Summary Report OR one of the demo reports See the Digging Deeper questions in the Appendix

55. Reflection 55 Did you start by observing patterns & trends of achievement groups? Did you consider a variety of questions across the Domains or CIAO categories? – How many questions did you generate? What questions do you still have about this part of the process?

56. Check for Understanding What are the key differences between the Value Added Summary Report and Diagnostic Summary Report? What is the difference in the purpose of the two reports? Turn, Talk, & Report 56

57. Questions 57

58. SINGLE SUBJECT VALUE ADDED OR DIAGNOSTIC REPORTS 58

59. Teacher Value Added Single Subject/Grade/Course 59 Which students were included in the analysis?

60. Students Included in Reporting To actually be included in PVAAS Teacher Specific reporting, a student MUST: Have a PSSA or Keystone score from the most recent year NOT be a foreign exchange student NOT be a first-year ELL student NOT be Proficient or Advanced on a PRIOR Keystone exam NOT be claimed at less than 10% instructional responsibility NOT have tested with the PASA (alternate assessment) 60

61. Teacher Value Added Single Subject/Grade/Course 61 A comparison to the state distribution for this grade, subject, or course

62. Teacher Value Added Single Subject/Grade/Course Multiple Years When multiple years are available – trends over time 62

63. Diagnostic Report Single Subject/Grade/Course 63 Who are the students that fell into each group?

64. Questions 64

65. BUSINESS RULES PVAAS Teacher Specific Reporting 65

66. Are there students to be removed (based on PA business rules)? Per Subject/Grade/Course 66 Did not take the test Took the PASA

67. Are there any students with < 10% Instructional Responsibility? 100% 5% 50% 100% 5% 67

68. Minimum N Counts? Must have at least 11 students Must have an “Active N Count” of at least 6 students/6 FTE Students 68

69. Would this teacher receive reporting? Actual N Count ≥ 11 100% 50% 100% YES Actual N=22 69

70. What is an “Active N Count”? Must ≥ 6 50% Actual N = 11 Active N = 5.5 students = 1 Active 70

71. Going Back: Roster Verification Summary 71

72. Going Back: Roster Verification Summary 72

73. The Next Step in the Process 73 Now on to Step 3 See Appendix for slides on the Custom Diagnostic Report

74. The Next Step in the Process 74 Step 4 – Digging Deeper Step 5 & 6– Custom Student Report

75. PLANNING FOR TODAY PVAAS: Growth, Not Gotcha! 75

76. So What, Now What? You have analyzed the growth of your students from last school year You have asked yourself questions to reflect on your professional practices BUT, it is a new school year with different students! So, now what? 76

77. Looking Ahead…Today’s Students! Who are the students you are teaching now and what are their projections to future assessments? – How does that match up with the growth similar students made in previous years? What kind of support do you need to influence the growth of students in your class(es) this year? 77

78. What is a Student Projection? 66 33

79. Growth to Projections: Planning for Today o Create a Custom Student Report (PVAAS Student Projections) Create a list of projections for just MY students By subject/grade/course By section/period By subgroups T 79

80. Custom Student Report: Projection Summaries to Proficiency and Beyond for a Teacher’s Current Students 80 Diagnostic Report for a Teacher’s Group of Students in Prior Year

81. Creating Custom Student Report 81

82. Options to Access PVAAS Student Projections **Access to Student Level Data is Required If you cannot create the Custom Student Report, contact your School Administrator. – Can they assign you access? – Can they create the report for your students and assign it to you? – Can they export the data into your student data information system where you already have access to data for your students? 82

83. The Next Step in the Process 83 Step 7 – Develop an action plan based upon your findings and root cause

84. Questions 84

85. Feedback Role – Teacher, Instructional Coach, District Administrator, or School Administrator IU # What are your thoughts on Teacher Specific Reporting? 85

86. Teacher Contact Request Form

87. Final Check for Understanding! Diagnostic Reports contain the information that will be used in the teacher evaluation. Value Added reports give you information about the achievement level of your students. The Composite score is a look at growth across all grades, courses, subjects, and years for which you provide instruction for up to three consecutive years. The Student History report provides information about the growth of an individual student. The Custom Student Reports along with the Diagnostic report can help a teacher reflect on instructional practices for their current students. 87

88. Questions? Concerns? Did you get the information, answers, and support for addressing your questions and concerns? What are your outstanding questions, concerns, and needs at this time? 88

89. ACCESSING SUPPORT & RESOURCES PVAAS: Growth, Not Gotcha! 89

90. Virtual Learning Modules Free, online professional development Access through PVAAS website Teacher Reports (45 min) 90

91. PVAAS Archived Webinars Recent webinar recordings can be accessed at http://tinyurl.com/PVAAS-Webinars-1415 http://tinyurl.com/PVAAS-Webinars-1415 – Can’t access tiny urls at your LEA? E-mail pdepvaas@iu13.org for the full hyperlink. The link will remain the same for the duration of the 2014-15 school year.pdepvaas@iu13.org Recordings currently available include: – PVAAS Export Function – Account Management: Access to District, School, Student, and/or Teacher Specific Reporting – Using & Understanding PVAAS series: A Focus on District/School Reporting A Focus on Teacher Specific Reporting A Focus on Student Projection Reporting

92. Misconceptions Addressed Contains common misunderstandings Available on PVAAS login page https://pvaas.sas.com https://pvaas.sas.com 92

93. PVAAS Help Menus Detailed content and explanations of: − Concept of Growth − Terminology − Use of Information 93

94. PVAAS Login Page 94

95. PDE PVAAS Webpage http://www.education.state.pa.us/pvaas 95

96. Questions? PVAAS Materials or Statewide Implementation pdepvaas@iu13.org 717-606-1911 PVAAS Report Web Site https://pvaas.sas.com 96

97. www.pde.state.pa.us PENNSYLVANIA Value Added Assessment System ______________________________________________________________________ PVAAS 333 Market Street Harrisburg, PA 17126 97

98. APPENDIX 98

99. CUSTOM DIAGNOSTIC REPORTS PVAAS: Growth, Not Gotcha! 99

100. Custom Diagnostic Report Purpose – to explore the growth of groups of students that YOU define Must have a minimum of 15 students to create a Custom Diagnostic Report 100

101. What Custom Diagnostic Reports Might YOU Create? Growth data by class section or period Students with similar educational opportunities – Interventions, Enrichment, Curricular experiences, Instructional delivery approaches Examples: – Elementary: students in different flexible reading groups – Middle School: 8 th grade Algebra I students who had Pre- Algebra in 7 th grade vs. those who had regular track 7 th grade Math – High School –Biology period 1 class vs. period 8 class 101

102. Custom Diagnostic Report 102