PVAAS Overview: Evaluating Growth, Projecting Performance PVAAS Statewide Core Team for PDE Fall

Partners & Teams for Session Activities Some activities throughout the session will be completed in pairs or small groups, while others will be done independently. Take a few minutes now to divide yourselves into: Partners (groups of 2), AND Small groups (groups of 4-6). Fall

Session Workbook Workbook contains: Key notes from PowerPoint Session activities Yellow boxes on PPT slides indicate the page number(s) you should be referencing in your workbook. Fall

Goal of Session By the end of the session, participants will be able to answer the following questions: What is PVAAS? What does PVAAS tell us about how students are projected to perform on a future PSSA? How does PVAAS measure growth? What does PVAAS tell us about growth in our district and school(s)? Which reports are most meaningful for school level planning, grade level planning, and student level planning? WB P. 1 Fall

What Is PVAAS? Fall 20125

PVAAS: Is an analysis of existing assessment data Longitudinal PSSA scores in available content areas of Math, Reading, Science, and Writing Provides reliable projections and estimates of growth Is a measure of district’s and school’s effectiveness and growth over time Provides information on the critical relationship of… Growth vs. Achievement WB P. 2 Fall

PVAAS: Is not another test Is not a comparison of single test scores! WB P. 2 Fall

In the Perfect World… Calculating Growth Measures would be easy The 3 M’s would not exist Measurement Error Mobility Missing Data We live in the real world! Fall WB P. 3

PVAAS Analyses Address the Issues – the 3 M’s! How? Research-based Statistical Modeling Minimizes the effects of measurement error associated with any one test score Measurement Error Accounts for mobility of students Mobility Compensates for missing data Missing Data Fall WB P. 3

PVAAS uses advanced statistical methodologies   Multivariate Response Modeling (also known as the Growth Standard Methodology)   Univariate Response Modeling (also known as the Predictive Methodology) Research-based Statistical Modeling Regularly used in other fields, such as psychology, pharmaceutical sciences, health care, and medicine Proven to be precise and reliable Fall WB P. 3

Achievement vs. Progress Student Achievement: Where are we? Highly correlated with demographic factors Student Progress: How far have we come? Highly dependent on what happens as a result of schooling rather than on demographic factors By concentrating on the progress students make, PVAAS puts the emphasis on something educators can influence! Fall WB P. 4

Demographics and Growth As long as you can sufficiently account for the prior achievement of the student,… WB P. 4 Fall …there is NO significant relationship between demographics and student progress, or growth!

Prior Achievement How can we adequately account for prior achievement? What about measurement error in test scores? The SAS, Inc. EVAAS team has found, the only way to adequately account for prior achievement is to use all available test data for each student to dampen the effects of measurement error. WB P. 4 Fall

In Summary… Is there a relationship between: Demographics vs. Achievement? YES Demographics vs. Growth? NO WB P. 4 Fall PVAAS does not need to adjust for the demographic status of students directly since it is accounting for the students’ prior achievement. In other words, demographics are inherently controlled for in PVAAS by the use of all longitudinal data on all students.

Summary and Reflections Work with a partner to complete the activity on page 5 of your workbook. WB P. 5 Fall

Summary and Reflections Answers WB P. 5 Fall

Two Types of PVAAS Information Fall

PVAAS Analyses PSSA Writing PSSA Science PSSA Reading PSSA Math LOOKING AHEAD Planning for Students’ Needs: Student Projections to Future Tests LOOKING BACK Evaluating Schooling Effectiveness: Value Added & Diagnostic Reports 18 Keystones WB P. 6 TWO Types of PVAAS Information Fall 2012

PVAAS Projections Individual Students and Groups of Students Fall

How Are Projections Used? 20 Weather Forecasts Population Growth Demand for Electricity Inventory Management Investment Strategies Fall 2012

What Are Projections? Given a specific set of circumstances… …what’s the most likely outcome? WB P. 6 Fall

Setting Expectations How do teachers set expectations for their students? Information o o What do they know about this student? Experience o o How have students like this one performed in the past? 22 Fall 2012 WB P. 6

PVAAS Projections What are they based on? Expectations based on what we know: o o About this student » » Prior test scores (PSSA), across subjects o o About other students who have already taken this test (PSSA) » » Their prior test scores (PSSA), across subjects » » Their scores on the test (PSSA) to which we are projecting o o About how much progress students at this school have made in the past in this subject, on average » » Growth Measure from School Value-Added Report 23 Fall 2012 WB P. 6

Given this student’s testing history, across subjects… …what is the student likely to score on an upcoming test, assuming the student has the kind of schooling experience this school has provided in the past? What Are PVAAS Projections? WB P. 6 Fall

Student Projection Methodology What data are used in this methodology? ALL available longitudinal data in both Reading and Math are used in projection calculations for all subject areas. Science data are also used in projection calculations for Science. Likewise, Writing data are also used in projection calculations for Writing. What students have projections? In what subjects? Projections are available for all students who have a minimum of 2 years of historical data. The next slide details the grade levels and subjects in which projections are reported. Fall WB P. 7

PVAAS Reporting: Projections to PSSA (Basic, Proficient, Advanced) Grade PSSA Last Taken Math and Reading:  3  4 or 5  4  5 or 6  5  6 or 7  6  7 or 8  7  8  8  11 Writing:  3  5  4  5  5  8  6  8  7  8  8  11 Science:  3  4  4  NA  5  8  6  8  7  8  8  11 Grade PSSA Last Taken Grade PSSA Last Taken WB P. 7 Fall Projections to Keystone Exams available in fall 2013

Information to complete boxes on page 8 in your workbook will be discussed on the next slide. WB P. 8 Fall

PVAAS Student Projection Report WB P. 8 Fall

How would you interpret? WB PP Fall 2012 Report A Report B 29

PVAAS Projections for Groups of Students Student Search WB P. 11 Fall

Identified Students Meeting Search Requirements WB P. 11 Fall

Quality of Projections for Decision Making PVAAS projections have been reviewed and approved by four different peer review panels and the US Government Accountability Office prior to being used as one of Pennsylvania’s Provisions for making AYP. Recent studies have confirmed that the PVAAS projections (even as far as 3 years into the future) are more reliable at looking at the future performance of a student than the most recent PSSA score. Fall WB P. 12

How often did students with PVAAS probabilities of 70% or higher score at proficient or advanced? 33 WB P. 12 Fall 2012 A document containing more information can be found on the PVAAS Login Page.

Setting Goals and Raising Expectations 34 Using PVAAS projections (along with other data) to set the appropriate goals for all students Fall 2012

Projection Summary Reports What are they? This is a report that summarizes the numbers and percentages of students in various likelihood ranges of performing at a proficient level or higher on a future PSSA exam. Cautions This report provides ONE indicator about likelihood of future performance. Additional data should be used with the projection for better decision-making. WB P. 13 Fall

District/School (Single Grade) Projection Summary Report WB P. 13 Fall Use the Projection tab in the blue menu bar to select projections to different grade levels and subjects.

How would you interpret? WB PP Fall 2012 Report B 37 Report A

Using PVAAS Projection Reporting Fall

How Might Educators Use Projection Summary Reports? Take 3 minutes with your small group to discuss how you, as educators, might use PVAAS Projection Summary Reports. Capture your ideas on page 16 of your workbook Forecasting the future needs of groups of students for: Instructional Decisions Intervention Enrichment Course Placement Decisions 2. 2.Anticipating staffing/support needs for students WB P. 16 Fall

Purpose of Projection Summary Reports Administrators can use for program planning: Intervention Planning Resource Allocation Course Availability & Placement Decisions School Improvement Planning Strategic Planning Teachers can use for instructional planning: Proportion of students on a path to success (proficiency or higher) Proportion of students at risk for reaching proficient or advanced levels of performance Plan for instructional strategies and classroom practices based on students’ needs WB P. 16 Fall

PVAAS Analyses PSSA Writing PSSA Science PSSA Reading PSSA Math LOOKING AHEAD Planning for Students’ Needs: Student Projections to Future Tests LOOKING BACK Evaluating Schooling Effectiveness: Value-Added & Diagnostic Reports 41 Keystones WB P. 17 TWO Types of PVAAS Information Fall 2012

PVAAS Value-Added Reporting How is growth measured in PVAAS? Fall

PVAAS Value-Added Reporting TestingSubjects – GradesMethodologies PSSA in consecutive years Mathematics, 4-8 Reading, 4-8 Growth Standard PSSA not in consecutive years Writing – 5, 8, 11 Science – 4, 8, 11 Math & Reading – 11 Predictive Keystone Exams (beginning 2013) Algebra I English Lit Biology Predictive Fall WB P. 17

Measuring Growth or Progress Students don’t all start the year at the same place academically Fall

Measuring Growth or Progress Standard for PA Academic Growth Regardless of their entering achievement level, students should not lose ground academically Is it not a reasonable goal to maintain the achievement level of a group of students? Not necessarily increase their achievement level, but maintain their achievement level Fall WB P. 17

PVAAS Growth Measure & Color Descriptors The Growth Measure represents the amount of growth a group of students has made. The Growth Color Descriptors represent the level of evidence we have in the data to say whether or not the group of students increased, decreased, or maintained their achievement levels on average. PVAAS yields a Growth Measure AND a Growth Color Descriptor. Fall WB P. 18

PVAAS Value-Added Growth Descriptors Fall WB P. 18 DB 2 SEs or More Above 0 Significant evidence that the district/school exceeded the standard for PA Academic Growth LB 1 to 2 SEs Above 0 Moderate evidence that the district/school exceeded the standard for PA Academic Growth G Between 1 SE Above 0 & 1 SE Below 0 Evidence that the district/school met the standard for PA Academic Growth Y 1 to 2 SEs Below 0 Moderate evidence that the district/school did not meet the standard for PA Academic Growth R 2 SEs or More Below 0 Significant evidence that the district/school did not meet the standard for PA Academic Growth

An Analogy for Measuring Growth – Grades 4-8 Math & Reading Fall

Growth Charts Doctors plot a child’s length, or height, over time. How would we describe Destiny’s growth or progress? Destiny’s Growth Fall

Prior Achievement for Group Most Recent School Year for Group Higher Achievement Concept of Measuring Growth in PVAAS – Grades 4-8 Math & Reading - Represents the mean achievement level of the group of students Lower Achievement Higher Achievement Lower Achievement Fall WB P. 19

Value-Added Reports Fall WB P. 19

Measuring Growth at a Deeper Level PVAAS does NOT use the percentages of students at various PSSA performance levels to measure growth. Instead, the use of NCE units are used to estimate growth measures for groups of students. Fall WB P. 20

Example of Rescaling Grades 4-8: Math & Reading Scaled Score Points Fall NCE Units WB P. 20

Value-Added Growth Measure Grades 4-8, Reading and Math Fall NCE Scale State Mean = 50 Achievement 5 th grade 47.6 Achievement 6 th grade 52.7 PVAAS Growth ≈ Change in position from 5 th to 6 th grade ≈ 52.7 – 47.6 ≈ 5.1 NCE units WB P. 20

Complete the boxes on page 21 of your workbook. Fall WB P. 21

PVAAS District Value-Added Report Grades 4 through 8 Fall WB P. 21

How would you interpret? Fall WB P. 22

PVAAS Growth Methodologies TestingSubjects – GradesMethodologies PSSA in consecutive years Mathematics, 4-8 Reading, 4-8 Growth Standard PSSA not in consecutive years Writing – 5, 8, 11 Science – 4, 8, 11 Math & Reading – 11 Predictive Keystone Exams (beginning 2013) Algebra I English Lit Biology Predictive Fall

An Analogy for Measuring Growth – Science, Writing, Grades 9-11 Math & Reading Using past performance to predict future performance Fall

Prior Achievement Higher Achievement Concept of Measuring Growth in PVAAS Science, Writing, Grades 9-11 Math & Reading Lower Achievement Predicted Achievement Mean Predicted Achievement - Represents the mean achievement level of the group of students - Represents the mean predicted achievement level of the group of students Fall WB P. 23

Value-Added Reports Fall WB P. 23

Measuring Growth at a Deeper Level PVAAS does NOT use the percentages of students at various PSSA performance levels to measure growth. Instead, the use of scaled score points are used to estimate growth measures for groups of students. Fall WB P. 24

Answer the questions in the boxes on page 24 in your workbook using the information on the next slide. Fall WB P. 24

Science Value-Added Report WB P. 24 Fall

How would you interpret? Fall WB P. 25

Can both high achieving and low achieving schools make growth? Fall

Schools with Lower Achievement - Represents the mean achievement level of the group of students Is a “green” in PVAAS sufficient for moving students to higher levels of achievement? Higher Achievement Lower Achievement Prior Achievement for Group Most Recent School Year for Group Higher Achievement Lower Achievement Fall WB P. 26

Is a “green” in PVAAS sufficient for moving students to higher levels of achievement? Fall

Schools with Lower Achievement Is a “green” in PVAAS sufficient for moving students to higher levels of achievement? Higher Achievement Lower Achievement Prior Achievement - Represents the mean achievement level of the group of students - Represents the mean predicted achievement level of the group of students Predicted Achievement 8 th Grade Science Mean Predicted Achievement for the Group Fall WB P. 26

Is a “green” in PVAAS sufficient for moving students to higher levels of achievement? B A Fall

(100% Advanced) Schools with High Achievement - Represents the mean achievement level of the group of students Can this group of high achieving students still meet the standard for PA Academic Growth? Higher Achievement Lower Achievement Prior Achievement for Group Most Recent School Year for Group (100% Advanced) Fall WB P. 27

Value-Added Reports Fall

(100% Advanced) Schools with High Achievement Can this group of high achieving students still meet the standard for PA Academic Growth? Higher Achievement Lower Achievement Prior Achievement 11 th Grade Reading (100% Advanced) - Represents the mean achievement level of the group of students - Represents the mean predicted achievement level of the group of students Mean Predicted Achievement for the Group Predicted Achievement 11 th Grade Reading Fall WB P. 27

Value Added Reports Fall

Regardless of the entering achievement level of the group… Students should not lose ground academically A reasonable goal would be to at least maintain the achievement level of the group All groups of students can make progress or growth Fall WB P. 28

Practice! Turn to a partner Explain the concept of growth 76 Previous School Year Current School Year Higher Achievement Lower Achievement Higher Achievement Lower Achievement Fall 2012 WB P. 28

Performance Diagnostic Report Looking Deeper into Growth Fall

Performance Diagnostic Report Math, Reading, Science, & Writing Grades 4-8 & Grades 9-11 Fall WB P. 29 Fill-in the appropriate information on page 29 using the information on the next two slides.

Performance Diagnostic Report WB P. 29 Fall 2012 Blue Bar – Current Year Gold Bar – Previous Years Missing Bar – Insufficient Number of Students Whisker – Margin of Error on Growth Value 79

PVAAS Performance Diagnostic Growth Descriptors - Interpretation Fall 2012 Zero (0) Line Growth Met the standard for PA Academic Growth Green Did not meet the standard for PA Academic Growth Pink Exceeded the standard for PA Academic Growth Blue What the whiskers tell us… WB P

Performance Diagnostic Report Fall

Check for Understanding WB P. 30 Fall

Performance Diagnostic Report Patterns of Growth Fall

WB P. 31 Fall Patterns of Growth The patterns of growth serve as an indicator of where you are headed in terms of growing students, and where you may have areas of improvement. The meaning of the patterns of growth should be viewed through the context of your achievement goals. What are your achievement goals? As a result, where do you need to make more than a year’s worth of growth?

Performance Diagnostic Patterns 85 System benefits lower-achieving students Fall 2012 WB P. 31 Shed Pattern Reverse Shed Pattern System benefits higher-achieving students

Performance Diagnostic Patterns 86 Fall 2012 WB P. 31 Tent PatternV Pattern System benefits middle level learners System benefits higher and lower achieving students

Performance Diagnostic Patterns 87 System benefits all types of learners except those right below proficiency Fall 2012 WB P. 32 Opportunity Gap Pattern

Performance Diagnostic Patterns 88 System benefits all types of learners with more progress occurring with the lower achieving students Fall 2012 WB P. 32 Desirable Pattern

Performance Diagnostic Report Growth of Subgroups Fall

Performance Diagnostic Subgroup Reports WB P. 33 Fall

WB P. 33 Fall 2012 Performance Diagnostic Subgroup Report 91

Best to Compare Subgroup to Entire Grade Level Same District/School – Same Grade – Same Subject WB P. 33 Entire grade levelSubgroup Fall

Let’s discuss these reports. What conclusions can you make about the growth of different groups of students in 8 th grade Reading? WB P. 34 Entire Grade Level Economically DisadvantagedSpecial Education Fall

Using PVAAS Growth Reporting Fall

How Might Educators Use Value-Added & Diagnostic Reports? Take 5 minutes with your small group to discuss how you, as educators, might use the PVAAS Value-Added and Performance Diagnostic Reports. Fall WB P. 35

Purpose of Value-Added & Performance Diagnostic Reports   Assessing curricular needs across grades and subjects   Evaluating instructional needs of students at various achievement levels and in different subgroups   Ensuring instructional coherence/alignment Curriculum Instruction Assessment Supports, including scheduling and staffing needs   Monitoring implementation and effectiveness of grade-level or course-specific programs Fall WB P. 35

Achievement and Growth Together! Fall

High Achievement Average Achievement Low Achievement Traditional Data Collection Approaches Achievement is based on each student’s ending point. WB P. 36 Fall

PVAAS Adds Another Variable To Achievement…GROWTH! Below Standard for PA Academic Growth Growth, or progress, is based on each student’s starting point. WB P. 36 Fall 2012 Above Standard for PA Academic Growth 99

Take notes on page 36 of your workbook using the information on the next slides WB P. 36 Fall

The Scatterplot Coordinate System Vertical Axis PSSA Percent Proficient or Advanced Horizontal Axis PVAAS Growth Index AYP Percent Proficient Target Standard for PA Academic Growth Boundary WB P. 36 Fall

Scatterplot Quadrants and Their Meanings Quad 1 - Excelling Adequate Achievement Positive Growth Quad 4 - Improving Below Adequate Achievement Positive Growth Quad 2 - Slipping Adequate Achievement Negative Growth Quad 3 - Underperforming Below Adequate Achievement Negative Growth WB P. 36 Fall

Web-Based Scatterplots Fall

Web-based Scatterplots Fall

Web-based Scatterplots Fall 2012 Growth versus Entering Achievement Level Growth versus Percentage of Students Proficient or Advanced Entering Achievement Level versus Percentage of Students Proficient or Advanced Customized based on Demographic Population 105 Users may choose from several different scatterplots!

Web-based Scatterplots Fall 2012 Customized Axis Selections include: % Tested Student Population Economically Disadvantaged % Tested Student Population Limited English Proficiency % Tested Student Population Special Education % Tested Student Population Minority 106

Interpreting Scatterplots Work with your partner to review BOTH scatterplot A and scatterplot B. Fill-in strengths and weaknesses of both scatterplots (A or B) on page 37 of your workbook. Remember, each “dot” represents a school! We will discuss as a group…“What can you conclude by looking at both?” WB P. 37 Fall

Example: District Scatterplot WB P. 37 Fall 2012 Note: Each dot on the scatter plot represents a school from this district. 108 A Math

Example: District Scatterplot: Same Schools, Different Subject WB P. 37 Fall 2012 Note: Each dot on the scatterplot represents a school from this district. 109 B Reading

Let’s Look at the Live Site! Fall

PA’s Intent for PVAAS Measure of Growth to Add to Existing Achievement Measures Indicator of Likelihood of Future Student Performance to Add to Current Achievement Data Another Tool for Continuous Improvement Not to Be Used in Isolation WB P. 38 Fall

PVAAS Resources Fall

NEW! PVAAS Virtual Learning Modules Fall Various types of VLMs Concept & foundations of measuring growth Understanding and using reports Integrating reports for decision-making New online, interactive resource Provides real-time professional development Designed in sections – pick & choose based on your needs WB P. 38

PVAAS Help Menus WB P. 39 Fall

PVAAS Predefined Packets WB P. 40 Fall Packet choices include: comprehensive planning school improvement grade level planning district diagnostic school projection summaries student projections

PVAAS Podcasts on iTunesU Fall 2012 WB P

Additional PVAAS Supports Fall 2012 WB P

Questions? PVAAS Materials or Statewide Implementation PVAAS Report Web Site Fall

333 Market Street Harrisburg, PA Fall