1. PVAAS Overview: Evaluating Growth, Projecting Performance PVAAS Statewide Core Team Fall 2011 August 2011 1

2. Partners & Teams for Session Activities The activities throughout the session will be completed in pairs or small groups, while others will be done independently. Take a few minutes now to divide yourselves into: Partners (groups of 2), AND Small groups (groups of 4-6). August 2011 2

3. Session Workbook Workbook contains: Key notes from PowerPoint Session activities Yellow boxes on PPT slides indicate the page number(s) you should be referencing in your workbook. August 2011 3

4. Goal of Session By the end of the session, participants will be able to answer the following questions: What is PVAAS? How does PVAAS measure growth? What does PVAAS tell us about growth in our district and school? What does PVAAS tell us about how students are projected to perform on a future PSSA? Which reports are most meaningful for school level planning, grade level planning and student level planning? WB P. 1 August 2011 4

5. Agenda Pre-test What is PVAAS? PVAAS Value-added Reports Going Deeper with Value-added PVAAS Projections Projection Summaries PVAAS Help Menus and Podcasts Post-test WB P. 1 August 2011 5

6. PVAAS Overview Pre-test! WB P. 2 August 2011 Complete Independently 6

7. What Is PVAAS? WB P. 3 August 20117

8. PVAAS IS : Analysis of existing assessment data Longitudinal PSSA scores in available content areas Math, Reading, Science, Writing Provides reliable estimates A measure of a school’s effectiveness and growth over time. Critical Relationship: Growth Data vs. Achievement Data WB P. 3 August 2011 8

9. PVAAS: Is not another test Critical concept: Is not a comparison of single test scores WB P. 3 August 2011 9

10. Review of Literature Is there a relationship between: Demographics vs. Achievement? Demographics vs. Growth? Demographics are inherently controlled for in PVAAS by the use of all longitudinal data on all students. WB P. 3 August 2011 10

11. Summary and Reflections Work with a partner to complete the activity WB P. 4 August 2011 11

12. Summary and Reflections Answers WB P. 4 August 2011 12

13. Achievement and Growth Together! August 201113

14. High Achievement Average Achievement Low Achievement Current Data Collection Approaches Achievement is based on each student’s ending point. WB P. 5 August 2011 14

15. PVAAS Adds Another Variable To Achievement…GROWTH! Below Standard for PA Academic Growth Progress is based on each student’s starting point. WB P. 5 August 201115 Above Standard for PA Academic Growth

16. Take notes on page 5 of your workbook using the information on the next slides WB P. 5 August 201116

17. The Scatter Plot Coordinate System Vertical Axis PSSA Percent Proficient or Advanced Horizontal Axis PVAAS Growth Index AYP Percent Proficient Target Standard for PA Academic Growth Boundary WB P. 5 August 2011 17

18. Scatter Plot Quadrants and Their Meanings Quad 1 - Excelling Adequate Achievement Positive Growth Quad 4 - Improving Below Adequate Achievement Positive Growth Quad 2 - Slipping Adequate Achievement Negative Growth Quad 3 - Underperforming Below Adequate Achievement Negative Growth WB P. 5 August 2011 18

19. Interpreting Scatter Plots Work with your partner to review either scatter plot A or scatter plot B. Fill-in strengths and weaknesses of the scatter plot (A or B) on page 8 of your workbook. Remember, each “dot” represents a school! We will discuss as a group…“What can you conclude by looking at both?” WB P. 6 August 2011 19

20. Example: District Scatter Plot WB P. 6 August 2011 Note: Each dot on the scatter plot represents a school from this district. 20 A

21. Example: District Scatter Plot: Same Schools, Different Subject WB P. 6 August 2011 Note: Each dot on the scatter plot represents a school from this district. 21 B

22. NEW! NEW! Web-Based Scatter Plots Page 7 in your workbook. WB P. 7 August 201122

23. Web-based Scatter Plots WB P. 7 August 2011 23

24. Web-based Scatter Plots! WB P. 7 August 2011 Users may choose to plot different variables with achievement and/or growth! This includes demographics such as: % Tested Economically Disadvantaged % Tested Limited English Proficiency % Tested Special Education % Tested Minority 24

25. Example WB P. 7 August 2011 25

26. Summary and Reflections WB P. 8 Work independently to complete the Summary and Reflections page in your workbook. August 2011 26

27. Two Types of PVAAS Information August 201127

28. TWO Types of PVAAS Information Today Looking Back/Evaluation… Value-added Growth Reports For Groups of Students Looking Forward/Planning… PVAAS Projection Reports For Individual Students and Groups of Students WB P. 9 August 201128

29. PVAAS Value-Added Reporting How is growth measured in PVAAS? WB P. 9 August 201129

30. Value-added Wouldn’t it be great to know if your district/school met the standard for PA Academic Growth for students tested in grades 4-8? And in grades 9-11? WB P. 9 August 2011 30

31. PVAAS Growth Methodologies TestingSubjects – GradesMethodologies PSSA in consecutive years Mathematics 4-8 Reading 4-8 Growth Standard PSSA not in consecutive years Writing – 5, 8, 11 Science – 4, 8, 11 Math & Reading – 11 Predictive WB P. 9 August 201131

32. Growth Charts WB P. 10 August 2011 32

33. Growth Charts WB P. 10 Percentile Bands 3 rd -15 th -50 th -85 th -97 th Horizontal Axis Age (months - years Vertical Axis Length (cm) August 201133

34. Growth Charts WB P. 10 August 2011 34

35. Concept of Growth Growth Standard Methodology Grades 4–8, Math and Reading Complete the activity on page 10 in your workbook. WB P. 10 August 201135

36. Growth Chart Activity WB PP. 10 & 11 August 2011 36

37. WB P. 11 One example of a growth spurt from 6 months to 18 months One example of less than typical growth from 6 months to 18 months August 2011 X X X X 37

38. A Challenge to Comparing Scores Across Years The means and standard deviations of PSSA exams change every year. A score of 1300 can be above the state mean one year and below it in another. Solutions – rescaling using a Normal Curve Equivalent Scale (NCE). Rescales all tests to have a state mean of 50 and standard deviation of 21.06. August 2011 38

39. Example of Rescaling Grades 4-8: Math & Reading 1000 1100 1200 1300 1400 1500 Scaled Score Units 0 10 20 30 40 50 60 70 80 90 100 NCE Units August 2011 39

40. Growth Standard Methodology Grades 4 through 8 – Reading & Math A group of students makes one year’s growth when … The group maintains their relative achievement level from one year to the next in relation to the statewide distribution (2006 baseline). August 2011 Group’s Position – This Year WB P. 12 4 th NCE Distribution 2006 Group’s Position – Last Year 5 th NCE Distribution 2006 Note: A dot represents a school. 40

41. Complete the calculation in your workbook using the information on the next slide. WB P. 12 August 201141

42. What Value-Added Growth Means! Growth Standard Methodology Grades 4-8, Reading and Math 0 10 20 30 40 50 60 70 80 90 100 NCE Scale State Mean = 50 Achievement 5 th grade 47.6 Achievement 6 th grade 52.7 PVAAS Growth ≈ Change in position from 5 th to 6 th grade ≈ 52.7 – 47.6 ≈ 5.1 NCE units WB P. 12 August 2011 42

43. Complete “Check for Understanding” WB P. 13 August 201143 Complete independently or with your partner Complete independently or with your partner

44. WB P. 13 August 2011 Example of Growth Measure=0 for a low-achieving district/school Example of Growth Measure=0 for a high-achieving district/school 44

45. WB P. 13 August 2011 Example of negative growth for a district/school 45

46. Answer the questions on page 14 with your partner. WB P. 14 August 2011 46

47. Value-added Measure of Growth Grades 4-8 Reading and Math Comparison to a State Growth Standard The Growth Standard specifies the minimal acceptable academic gain from grade to grade for a group of students. The use of a Growth Standard creates the possibility that ALL schools can demonstrate appropriate growth. WB P. 14 August 2011 47

48. NEW! NEW! Legends and Color Codes Alignment of colors and interpretative legends across all Value-Added Reports (all subjects and grade levels) New 5 color-coding scheme All colors now have the same interpretation and meaning Use of the data becomes easier as there is no confusion about what the colors indicate New 5 Color Codes Allow for more differentiation among schools making a year’s worth of academic growth or more Allow for more differentiation in Science and Writing reporting, as well as grades 9-11 Reading and Math reporting WB P. 15 August 2011 48

49. August 201149 WB P. 15

50. NEW! PVAAS Value-added Growth Descriptors Grade 4-8, Math and Reading WB P. 15 August 2011 50 DB Significant evidence that the district/school exceeded the standard for PA Academic Growth LB Moderate evidence that the district/school exceeded the standard for PA Academic Growth G Evidence that the district/school met the standard for PA Academic Growth Y Moderate evidence that the district/school did not meet the standard for PA Academic Growth R Significant evidence that the district/school did not meet the standard for PA Academic Growth

51. Complete the boxes on page 16. WB P. 16 August 2011 51

52. PVAAS School Report Grades 4 through 8 WB P. 16 August 2011 52

53. Your Turn! Interpret this report on page 16. WB P. 16 August 2011 53

54. Check for Understanding In Your Own Words Describe the growth of students tested in 5 th grade math in 2011. What does the 3-year growth history of 5 th grade math indicate? What does the 3-year achievement estimate history indicate for this most recent group of students in 5 th grade? WB P. 17 August 201154

55. Concept of Growth: Predictive Methodology Writing and Science Grade 11 - Math and Reading Complete the activity on workbook page 18. WB P. 18 August 201155

56. Predictive Methodology Writing, Science, and Grade 11 Reading & Math The mean observed score from the actual test is not significantly different from the mean predicted score for the group of students investigated. The mean predicted score is calculated based on all Reading and Math data in each student’s record in the group. August 2011 Mean Observed Score ≈ Mean Predicted Score Mean Predicted Score ± error Mean Observed Score WB P. 18 56

57. PVAAS Value-added Reports Writing, Science & Grade 11 Math, Reading Predictive Methodology A predicted score for each student; Mean of predicted scores for the group of students; Mean of observed scores for the group of students. Growth Measure ≈ Observed Mean - Predicted Mean WB P. 19 August 2011 57

58. WB P. 19 August 2011 58 NEW! PVAAS Value-added Growth Descriptors Grade 9-11, Math and Reading Science and Writing DB Significant evidence that the district/school exceeded the standard for PA Academic Growth LB Moderate evidence that the district/school exceeded the standard for PA Academic Growth G Evidence that the district/school met the standard for PA Academic Growth Y Moderate evidence that the district/school did not meet the standard for PA Academic Growth R Significant evidence that the district/school did not meet the standard for PA Academic Growth Note that these are now the same growth color and descriptors as Grades 4-8 Math and Reading!!

59. Answer the questions in the boxes on page 20 using the information on the next slide WB P. 20 August 201159

60. Science Value-added Report WB P. 20 August 201160

61. Independent Practice WB P. 21 August 2011 Report B Report A 61

62. Summary and Reflections Work independently to answer the questions on page 22. WB P. 22 August 2011 62

63. Definition: A group meets the standard for PA Academic Growth when they maintain their relative achievement level from one year to the next in relation to the statewide distribution (2006 baseline). WB P. 23

64. Predictive Methodology Math & Reading – Grades 9-11 Writing – Grades 5, 8, & 9-11 Science – Grades 4, 8, & 9-11 Definition: A group meets the standard for PA Academic Growth when their actual achievement meets their predicted achievement - based on the average schooling experience. WB P. 23

65. Going Deeper… Performance Diagnostic Report Fill-in the appropriate information on page 24 using the information on the next three slides. WB P. 24 August 201165

66. Performance Diagnostic Report WB P. 24 August 2011 Blue Bar – Current Year Gold Bar – Previous Years Missing Bar – Insufficient Number of Students Whisker – Margin of Error on Growth Value 66

67. Complete the balloons for the picture at the bottom of page 24. WB P. 24 August 201167

68. PVAAS Performance Diagnostic Growth Descriptors - Interpretation August 2011 68 Zero (0) Line Growth Met the standard for PA Academic Growth Green Did not meet the standard for PA Academic Growth Pink Exceeded the standard for PA Academic Growth Blue What the whiskers tell us… WB P. 24

69. Performance Diagnostic Report WB P. 24 August 2011 69

70. Performance Diagnostic Use the information on the next two slides to complete the boxes on page 25. WB P. 25 August 201170

71. WB P. 25 August 2011 Link to all students reported for this subject. Links to students in selected predicted PSSA performance category 71

72. Student Lists from Performance Diagnostic Reports WB P. 25 August 2011 Current PSSA Performance Category for each student 72

73. Use the information on the next slide to complete the boxes on page 26. WB P. 26 August 201173

74. Patterns of Growth WB P. 26 August 2011 A B C 74

75. Check for Understanding WB P. 26 August 2011 75

76. Performance Diagnostic Subgroup Reports WB P. 27 August 201176

77. WB P. 27 August 2011 Performance Diagnostic Subgroup Report 77

78. Best to Compare Subgroup to Entire Grade Level Same School – Same Grade – Same Subject WB P. 27 Entire grade levelSubgroup August 201178

79. Summary and Reflections Discuss with a partner. List conclusions you can make about the growth of different groups of students in 8 th grade Reading. WB P. 28 Entire Grade Level Economically DisadvantagedSpecial Education August 2011 79

80. PVAAS Projections Individual Students and Groups of Students August 201180

81. Student Projections Wouldn’t it be great to know the likelihood that a student will be proficient on a future PSSA? WB P. 29 August 2011 81

82. Student Projection Methodology Is this the same methodology as the one used to estimate growth? No, the projection methodology is a separate modeling process that focuses on individual students. What data are used in this methodology? ALL available longitudinal data in both Reading and Math are used in projection calculations! What students have projections? In what subjects? Projections are available for all students who have a minimum of 2 years of historical data. The next slide details the grade levels and subjects in which projections are reported. WB P. 29 August 2011 82

83. PVAAS Fall 2011 Reporting: Projections to Basic, Proficient, Advanced Grade PSSA Last Taken Math and Reading:  3  4 or 5  4  5 or 6  5  6 or 7  6  7 or 8  7  8  8  11 Writing:  3  5  4  5  5  8  6  8  7  8  8  11 Science:  3  4  4  NA  5  8  6  8  7  8  8  11 Grade PSSA Last Taken Grade PSSA Last Taken WB P. 29 August 2011 83

84. Information to complete boxes on pages 29 & 30 in your workbook will be discussed on the next two slides. WB PP. 29-30 August 2011 84

85. PVAAS Student Report WB P. 29 August 2011 85

86. PVAAS Student Projection Report WB P. 30 August 2011 86

87. Independent Practice WB PP. 31 & 32 August 2011 Report A Report B 87

88. PVAAS Projections for Groups of Students Student Search WB P. 33 August 2011 88

89. Identified Students Meeting Search Requirements WB P. 33 August 2011 89

90. Quality of Projections for Decision Making The projection is precise, and is created using a model that has been reviewed and approved by four different peer review panels and the GAO (US Government Accountability Office). The most recent 2008 growth model proposal to USDOE includes information regarding the statistical model and projection reliability study. (http://www.ed.gov/admins/lead/account/growthmodel/pa/ index.html)http://www.ed.gov/admins/lead/account/growthmodel/pa/ index.html Recent studies have confirmed that the PVAAS projections (even as far as 3 years into the future) are more reliable at looking at the future performance of a student than the most recent PSSA score. August 2011 90

91. Summary and Reflections Discuss with a partner and answer the questions for the reports displayed on page 34. WB P. 34 August 2011 91

92. Projection Summary Reports What are they? This is a report that summarizes the numbers and percentages of students in various likelihood ranges of performing at a proficient level on a future PSSA exam. How might a school use these reports? Intervention Planning Resource Allocation Strategic Planning School Improvement Planning Cautions This report provides ONE indicator about likelihood of future performance. Additional data should be used with the projection for better decision-making. WB P. 35 August 2011 92

93. School (Single Grade) Projection Summary Report WB P. 35 August 2011 93

94. District (Single Grade) Projection Summary Report WB P. 36 August 201194

95. Summary and Reflections Discuss with a partner and answer the questions on page 36 of your workbook. WB P. 36 August 2011 95

96. TWO Types of PVAAS Information August 2011 Today Looking Back/Evaluation… Value-added Growth Reports For Groups of Students Looking Forward/Planning… PVAAS Projection Reports For Individual Students and Groups of Students 96

97. PVAAS Help Menus WB P. 37 August 2011 97

98. PVAAS Help Menus WB P. 37 August 2011 98

99. PVAAS Podcasts on iTunes U August 2011 99 WB P. 38

100. August 2011100

101. Fall Professional Development Opportunities Check out what is being offered this school year by visiting the PVAAS login page!

102. PA’s Intent for PVAAS Another Tool for Continuous Improvement Measure of Growth to Add to Status/Achievement Measures Not to Be Used in Isolation WB P. 38 August 2011 102

103. PVAAS Overview Post-test! WB P. 39 August 2011103

104. Discussion of Post-Test As a small group, review questions 1-10 on the Post-test. Most importantly, report out and discuss question 11 as a small group – 3 ways in which you intend to use PVAAS. WB P. 39 August 2011 104

105. Questions: PVAAS Materials or Statewide Implementation pdepvaas@iu13.org 717-606-1911 PVAAS Report Web Site https://pvaas.sas.com WB P. 40 August 2011 105

106. www.pde.state.pa.us 333 Market Street Harrisburg, PA 17126 August 2011 106