1. PVAAS: One of the Data Tools in a Standards-Aligned System to Support Data-Informed Decision-Making in K-12 Schools
PVAAS Core Team
June, 2007

2. Addressing the Priorities: A Standards-Aligned System
We approached this from the perspective of Pennsylvania’s Framework for a Standards-Aligned System –
A Standards-Aligned System consists of the six components revolving around student learning as displayed in this graphic – our goal was to address each component for mathematics to provide comprehensive support to districts;
Each component of the system connects to the other components - as all interface to support achievement and progress for all of our students.
These are voluntary resources for districts – some districts will need/use some sections more than others – we really wanted to differentiate supports through our resources.

3. Pennsylvania’s Design

4. PDE Provides The Data Tools…
PSSA Data Interactive by e-Metric
Interactive tool for analyzing PSSA data by district, school, grade, subgroup and student – Reporting category information is also available
PVAAS
Value-added tool that reports growth of cohorts of students and projections for individual students for performance on future PSSAs
PAAYP
Public site that reports all AYP related data on district, school, grade and subgroup - Reporting category information is also available
4Sight Member Center
Reporting site for benchmark data by grade, class, subgroup and student – intended to be formative and used often throughout the school year

5. PDE Provides The Framework…
“Getting Results!”
A Framework for Continuous School Improvement Planning
The framework provides guiding questions for root cause analyses in the following data environments:
PSSA and AYP inquiry
PVAAS
4Sight Benchmark Data
Local Assessment Data Sets

6. Today’s Focus - PVAAS
The Power of 2
PROGRESS
&
ACHIEVEMENT

7. What Is PVAAS?
In this section, we will describe what PVAAS is and clarify what PVAAS is not.
[CLICK]

8. Student Learning Measures
Status Measures
PSSA
% proficient or not proficient
Other local assessment data
Improvement Measures
Safe Harbor
PPI – Pennsylvania Improvement Index
Progress/Growth Measures
PVAAS
Before we talk specifically about value-added, let’s look at what types of measures of student learning that we have now in schools-- [CLICK]
Currently there are STATUS measures such as PSSA or local assessments that indicate if students are proficient or not.
[CLICK]
There are IMPROVEMENT measures, like Safe Harbor, that tell us if we have moved more students to proficiency. Improvement measures compare different cohorts of students at the same place in their sequence of grades.
Now, in addition to these types of learning measures, there is a GROWTH measure. A growth measure compares the performance of the same cohort of students in consecutive years of schooling.
Pennsylvania uses PVAAS as our statistical measure of progress. PVAAS asks questions such as: Are we headed in the right direction with all students? Even if a cohort of students is not proficient, are they headed in the right direction? If a student is proficient, are we at least making the expected growth?
Looking specifically at progress adds a new and important lens to looking at student learning.

9. Stair-Step Expectations
Grade Three Grade Four Grade Five
Grade 3 Test Grade 4 Test Grade 5 Test
In the perfect classroom, in the perfect school, in the perfect district - all students would come to us with the same skill levels.
We would know where to begin instruction and be able to provide the same instruction for all students.
This slide is used with permission from JHussey, Battelle for Kids. Copyright © 2003
In a perfect world students would start school at about the same academic level. Proficiency levels could simply be achieved by students gaining age appropriate knowledge and skills each year.

10. Differentiated Reality
Grade Three Grade Four Grade Five
Grade 3 Test Grade 4 Test Grade 5 Test
We know that all students are unique.
We know that students come to us as different starting points.
We also have a minimum standard for all students to reach proficiency at the end of each grade as measured by state assessments in grades 3-8, and 11 (SY 05-06).
This slide is used with permission from JHussey, Battelle for Kids. Copyright © 2003
Students begin school at different places, they often progress at different rates. Yet all students are expected to achieve the same level of academic proficiency each year.

11. The Need for Progress Measures
Grade Three Grade Four Grade Five
Grade 3 Test Grade 4 Test Grade 5 Test
Because of these student differences, we need to pay attention not only to end of year proficiency levels, but also to how much progress students make in a given year, even if they haven’t reached the year-end proficiency goal.
We need to be sure we are headed in the right direction.
Our role in schools is to meet students where they are when they enter our schools, assess their skills and match our instruction to their instructional needs.
This slide is used with permission from JHussey, Battelle for Kids. Copyright © 2003
To measure school effectiveness in this “differentiated” world, we need to pay attention not only to proficiency levels but also to how much progress students make in a given year.

12. PVAAS IS: A statistical analysis of existing assessment data.
A realistic measurement of a school’s progress/growth over time.
Progress data to add to achievement data as schools are analyzing their work with students and making improvement plans.
In this high-stakes assessment and accountability climate, we need accurate information to guide us in making appropriate decisions for students…decisions about instruction, curriculum, and assessment.
PVAAS provides us with additional diagnostic data for decision-making at the district, school, and grade or subject level.
[CLICK]
PVAAS uses a statistical analysis of existing assessment data to provide a precise measurement of student progress/growth over time.
PVAAS is another tool to assist educators in making instructional data-informed decisions.
It provides progress data to add to achievement data as schools are analyzing their work with students.
PVAAS offers powerful diagnostic information about school practice by leveraging student achievement data with complex statistical tools. It provides a tool to identify strengths and weaknesses, allowing educators to determine the unique causes of either.

13. PVAAS is NOT:
Another Test
A measure for teacher accountability

14. PVAAS has Two Methodologies
Value-added Analysis for growth in cohorts of students by school, grade, subgroup.
Projection capability to calculate the likelihood of a student achieving a specified target performance level on a future PSSA test.
It is important to keep these two concepts and methodologies separate.
[CLICK]
Value-added  cohorts of students
Projections  individual students

15. The PVAAS Methodologies
Looking Forward…
PVAAS Projection Reports For Individual Students and Cohorts of Students
Looking Back…
Value-added Growth Reports For Cohorts of Students
Today

16. Student Projections Wouldn’t it be great to know the likelihood
that a student will
be proficient on
a future PSSA?

17. Student Projection Methodology
A separate longitudinal modeling process focused on the performance scores of individual students.
The model uses all of the available data (both math and reading) to project future PSSA performances in Math and Reading.
Projections can be calculated for individual student or groups of students.

18. PVAAS Student Projection Methodology

19. PVAAS Projection Reports for Groups of Students
If schools are interested in targeting a student or particular types of students, PVAAS will allow schools to create lists based on filtering for specific subgroups or projected proficiency levels.
Participants may identify specific demographic information or proficiency levels, and PVAAS will search for students meeting that profile.
The Search for Students also allows teachers and administrators to generate a listing of students whose probabilities of achieving a selected proficiency level fall within a specified range of probabilities. This very useful feature assists in identifying students who may need special programming of some type. For example, this search will return all of the female students for whom the probability of scoring Advanced on the 6th grade PSSA Math Test is between 70% and 100%.
This is particularly helpful when looking for patterns of performance across a school or district and when attempting to identify students with a small probability of achieving a proficient rating. For example, restricting the search to students with between a 0 and 50% chance of being Proficient on the next PSSA provides an “at risk” list.
If you specify the probability range of 0% to 100%, you will generate a list of all students and their likelihood to be in the selected performance category on the next PSSA exam on the specified subject.
[CLICK]
by Subgroup and/or Projected Proficiency Level

20. Identified Students Meeting Search Requirements
When the parameters of the search have been submitted, a list is generated that lists the students that meet the particular criteria outlined in the student search.
In this example, the report lists the female students whose probability of an Advanced rating on the 6th grade PSSA math exam is between 70% and 100%.
Note that all demographic information on the listed students is also provided.
Note also that each of the columns is again “sortable.” By [CLICK]ing on the column name, the report is sorted by the contents of that column. This allows the user to group cluster of students based on demographic information.
[CLICK]

21. Your Turn!!!! Go to: Login: Demo Site Password: https://pvaas.sas.com
PVAAS.Training
PVAAS
(or use your own account)
Reports:
Restrict by grade:
Restrict by Proficiency :
Student Search 8
PSSA
11th Math Proficient
0% to 40%

22. Let’s do a multiple search…

23. …and paste the results into a new EXCEL tool…

24. The result: PVAAS Projection Summary

25. PVAAS Projection Summary Tool One Application
Strategic Planning:
What percent of students are likely to be proficient on a future PSSA examination?
Likelihood Range
Number
Percent
Greater than or equal to 70%
169
58.9
Between 30% and 70% 54
18.8
Less than or equal to 30% 61
22.3
Pie Chart Legend
Greater than or equal to 70%
Between 30% and 70%
Less than or equal to 30%

26. PVAAS Projection Summary
Intervention Planning:
What percent of students are likely to need extra support in preparation for a future PSSA examination?
Likelihood Range
Number
Percent
Greater than or equal to 70%
169
58.9
Between 30% and 70% 54
18.8
Less than or equal to 30% 61
22.3
Pie Chart Legend
Greater than or equal to 70%
Between 30% and 70%
Less than or equal to 30%

27. Your Turn: The PVAAS Projection Summary
An example:
On the next slide, you will see a Projection Summary of 8th graders to the Proficient level on their 11th grade PSSA math and reading exams.
What do you see?
What do you recommend?

28. 8th to 11th Projection to Proficiency

29. Value-added Measure of Cohort Growth
Comparison to a State Growth Standard
The Growth Standard specifies the minimal designated academic gain from grade to grade for a cohort of students.
The use of a Growth Standard creates the possibility that ALL schools can demonstrate appropriate growth.
In the three pilot phases from , students were compared to the average school in their testing pool. Comparison to averages always guarantees that there must be some schools below average, and, therefore, all schools could not be at or above the expected growth target.
Now that Pennsylvania is implementing PVAAS statewide, the data set allows Pennsylvania to yield a more defined and accurate picture of growth in schools.
The introduction of a Growth Standard sets a target of growth that ALL schools can achieve since growth only depends on the performances of that school and not the average performance of schools in a testing pool.
[CLICK]

30. A Conceptual View of the PVAAS Value-added Growth Methodology
In this section we will discuss important considerations that provide a foundation for the methodology that produces the growth measures of PVAAS. This view will be very conceptual since the details of the particular statistical methodology are very complex.
The methodology detailed in this presentation represents the methodology for statewide implementation. This methodology will be used to yield PVAAS reports for SY05-06 reporting for Grades 4 and 6. Future PVAAS reporting will use this new methodology for all PVAAS reporting for Grades 4 through 8.
The 100 pilot districts received PVAAS reports from a different methodology due to the limited number of districts. A statewide pool of 501 districts creates the opportunity for the methodology detailed in this presentation.
[CLICK]

31. The Assessment Dilemma
True Student Achievement
Assessment is focused on measuring the understanding that a student or group of students possess at any one time. This is an extremely difficult quality to measure.
[Click]
One observation does give us an indication of student understanding. However, if we could use more than just the one observation, our confidence in the accuracy of the estimate of the students understanding should increase.
Any test is just a snapshot in time!

32. Four Important Issues Difference in scores in consecutive years.
1. Measuring Gain
2. Growth Standard
Convert all scores to same scale – NCE scale
3. Comparing Scores from year to year
Combine current observed scores will all previous scores in students’ histories.
4. Estimating Achievement
Student Name
Quiz 1
Test 1
Quiz 2
Project 1
Test 2
Test 3
Cum
Possible 25
100 30 75 70
Percent
Joe 20 97 23 72 65 55 83
Betty 14 76 27 56
73.5
Elizabeth 21 96 60 64 78 86

33. PVAAS Value-added Cohort Growth Descriptors
Favorable Indicator
Estimated gain at or above growth standard.
Students in this cohort have made at least one year growth.
All schools can achieve this rating.
Caution Indicator
Estimated gain below growth standard but by less than one standard error.
Students in this cohort have grown less than the standard.
Stronger Caution
Estimated gain below growth standard by more than one but less than two standard errors.
Students in this cohort have fallen behind their peers.
The Ratings on the PVAAS School Report and on the District Value-Added Report are color coded to assist with quick recognition of the rating.
[CLICK]
Green - Estimated gain at or above growth standard. Students In this cohort have made at least one year of growth. All schools can achieve this rating.
Yellow - Estimated gain below growth standard but by less than one standard error. Students in this cohort have grown less than the standard.
Light Red - Estimated gain below growth standard by more than one but less than two standard errors. Students in this cohort have fallen behind their peers.
Red - Estimated gain well below growth standard by more than two standard errors. Students have made little progress.
Emphasize that the Value-Added side of PVAAS is about the growth of cohorts of students – not about the growth of individual students.
Strongest Warning
Estimated gain well below growth standard by more than two standard errors.
Students in this cohort have made little progress.

34. Current Data Collection on each student’s ending point.
Approaches
High Achievement
Average Achievement
Achievement is based
on each student’s ending point.
This slide illustrates how we have traditionally looked at school achievement.
We give a test, usually at the end of the year, and see whether or not students are measuring up to end of year expectations, without taking into account their starting point.
By looking at end of year performance, we conclude that some students or schools or districts are “low,” “average,” or “high” in terms of student achievement. However, this does not provide the entire picture as we are trying to make data-informed decisions in schools/districts.
This does not tell us about student growth/progress during that school year.
[CLICK]
Low Achievement

35. PVAAS Adds Another Variable To Achievement…Progress
Below
Growth
Standard
At or Above
Growth
Standard
PVAAS is a different way of looking at data that adds the critical dimension of progress and growth.
It is not another test but a comprehensive analysis of achievement trends and progress rates.
With PVAAS we are able to look at BOTH rate of progress as well as level of achievement at a given point in time.
PVAAS is able to provide a measure that illustrates what value “schooling” has had on students - regardless of variables such as ethnicity, economic status, ESL and students with disabilities.
PVAAS indicates how much progress these students have made from the time they entered school until the time they leave, on an annual basis.
[CLICK]
Progress is based
on each student’s starting point.

36. So how can schools begin to look at achievement and progress?
This slide provides a two-dimensional representation of school performance – achievement and progress.
The vertical dotted lines indicate the boundaries of ratings of the estimated growth of a cohort of students relative to the Growth Standard.
The horizontal line indicates the boundary for the Proficient rating of performance for a cohort of students. School personnel should check their PSSA scores to determine the percent of students who are Proficient or Advanced, and check PVAAS for the measure of the growth of that cohort. Together, the status and growth data provide a much deeper view of the cohort’s level of understanding.
[CLICK]

37. Input PSSA and PVAAS data into the EXCEL data table
Data Entry
Input PSSA and PVAAS data into the EXCEL data table

38. The Scatterplot
Scatterplots for both math and reading are automatically generated…

39. Scatterplot District Report Template

40. Your Turn: Your recommendation?
This scatter plot displays measures of both achievement and growth of a 4th Grade Math for all the elementary schools in a district from the 2006 PVAAS Analysis. These data are from an actual Pennsylvania district.
It appears that only one school in the district did not reach the 45% target. However, three did not meet the Growth Standard.
It is likely that personnel from the three schools may want to have a conversation with the personnel from several of the other schools.
Available
Now!

41. Your Turn: Your recommendation?
This scatter plot displays the measures of achievement and growth in 4th Grade Reading from the same district as the math scatter plot on the previous slide.
This display suggests that there is perhaps a more systemic problem in reading throughout the district.

42. IU1 Scatterplots
Grade 4 – Reading and Math
Grade 6 – Reading and Math

43. PVAAS School Report
Did every student cohort at each grade level reported make at least one year’s growth in Math and demonstrate expected rates of progress?
(Getting Results!, p.17)

44. Your Turn! Interpret this report!

45. Your Turn!!!! Go to: Login: Password: https://pvaas.sas.com
PVAAS.Training
PVAAS
(or use your own account)
Reports:
Select:
District/School Value Added
District
School – if School Value Added is chosen

46. PVAAS Diagnostic Report
Did every student cohort at each performance level
(below basic/basic/
proficient/advanced) meet or exceed the expected growth in Math?
(Getting Results!, p.17)

47. Your Turn! Interpret this report!
In the district/school diagnostic report, students’ gains can be disaggregated by quintiles or by PSSA performance levels based on their prior achievement levels. This allows the school personnel to assess the growth of these subgroups quickly and easily. Since this part of the report is to be used only for diagnostic purposes, one standard error is used in interpreting the significance of the results.
The chart provides a visual representation of the district/school’s diagnostic report. It shows the most recent information as well as the previous year’s information.
The power of this report is in looking at patterns to reflect on their local practices.
This report does not tell us WHY progress is being made or not made. The report tells us about the amount of progress made for different groups of students.
This particular report suggests:
The students who placed in the highest quintile met the Growth Standard – their confidence band contains the green line with value 0 0 and therefore we can conclude that these students maintained their position in the performance distributions between 7th and 8th grades.
The students who placed in quintiles 2 through 4 did not meet the Growth Standard – their confidence bands were below the green line. This indicates that these students slipped in the performance distribution of all students from their position at the end of the 7th grade to the end of 8th grade.
Diagnostic reports will be generated for subgroups that consist of 5 or more students. Since there are only 2 students in the lowest quintile, analysis for this quintile is not provided.
The goal of the district or school is to have significantly positive bars for all students disaggregated by prior performance categories.
[CLICK]
This type of diagnostic report is also available for subgroups based on the demographic data submitted by districts.

48. Looking at a subgroup…
Diagnostic Reports from PSSA Subgroups are also available.
As schools begin to analyze their PVAAS data, teachers and administrators need to look at their effectiveness across the entire continuum of students. To assist with this analysis, schools may view diagnostic reports for specified subgroups of students. This can be achieved by [CLICK]ing on “yes” in response to subgroup reporting on the Diagnostic Report and selecting the desired subgroups. Each category must contain at least five students for it to be represented on the graph.
For a more targeted Line of Inquiry, schools may wish to use the “Student Search” report. Participants can use the protocol of the “Student Search” report to compare the reports on the targeted subgroups with the general student population.
This may elicit a rich discussion about curriculum, assessment and instruction.
[CLICK]

49. Your Turn!!!! Go to: Login: Password: https://pvaas.sas.com
PVAAS.Training
PVAAS
(or use your own account)
Reports:
Select Subgroups:
District/School Performance Diagnostic
Optional filter to view diagnostic report for specified subgroup(s)

50. PVAAS School Search Function
Wouldn’t it be nice to be able to connect with schools making high growth that are similar to your school?

51. Responsibilities of District School Search User
Access to this feature is for continuous improvement purposes
This new features removes the manual process for searching for high growth in a district
This is not a tool to rank or rate schools
Do not print or copy lists of schools that could be misinterpreted as a ranking or rating scale

52. A NEW Feature
In addition to all of the powerful reports provided by SAS in the PVAAS data tool, SAS has made available to Pennsylvania the School Search feature.
This feature allows you to investigate the achievement and growth of schools with similar demographic profiles in your district.

53. Select A School
The demographic information and current PVAAS ratings for the selected school are displayed…

54. Rating Scheme
Observed achievement (MEAN) values and Estimated growth values (GAIN) are rated with a quintile value.
PVAAS breaks ALL scores in a particular grade into quintiles: 1 = lowest 20%, 2 = next 20%, …, 5 = highest 20% in the state.
PVAAS then reports the quintile in which the school’s score resides.
Achievement Mean (1-5)
Growth Score/Gain (1-5)
1=Lowest Quintile; 5= Highest Quintile

55. Comparable School Search
This feature allows you to investigate the PVAAS gains for schools within districts or across IU regions
for which you have access,
that have at least one grade in common with the selected school, and
that fit demographic characteristics specified.

56. An Example
Adams Elementary has a 92% population of minority students. You can search for schools within your district that have a minority subgroup that comprises any percent range of their population. Here we chose between 80% and 100% of their population.

57. The Results
The list is in descending order by Cumulative Gain Index. The list can be sorted by any column by clicking on its heading.
Notice: Adams Elementary is the Selected School and is highlighted in brown in the listing

58. Multiple Criteria
This search produced a report for all schools that fit the demographic criteria in all categories as specified.

59. A Listing of All Schools
A search with no demographic information selected will produce a list of all schools for which you have access and that share at least one tested grade with the selected school.

60. Why Perform A Search…
To identify schools with similar demographic profiles that might act as a resource to the selected school
To determine what schools within a district are exhibiting significant growth
To encourage schools to network based on growth successes

61. Your Turn!!!! Go to: Login: Password: https://pvaas.sas.com
PVAAS.Training
PVAAS
(or use your own account)
Reports:
School Search
Select a school.
Choose different ranges on demographics.

62. How Can PVAAS Help Schools?
In this section, we will discuss the value of value-added assessment and how it can help educators improve instructional programs.
[CLICK]

63. Key Learning from PVAAS
Value-added information will tell you:
How much progress cohorts of students have made in each subject area and grade level.
How much progress cohorts of students at different prior achievement levels have made.
How a cohort of students progress in one curricular area or program compares to their progress in others.
Value added information is all about progress:
How much progress students have made in each subject area and grade level,
How much progress students at different prior achievement levels have made, and
How students’ progress in one curricular area or program compares to their progress in others
[CLICK]
(This slide is used with permission from J. Hussey, Battelle for Kids. Copyright © 2003)

64. PSSA and PVAAS The Power of 2!
PSSA tells us where a student is.
PVAAS tells us where that student is headed.
Location and direction are critical partners in the quest for having all students achieve proficiency
The data from both achievement and progress measures gives schools valuable information to make data-informed decisions.
“The Power of 2!” is founded on the fact that educators need information about where their students ARE and where they ARE HEADING!
It is not sufficient to know only position or direction. To chart the best path to proficiency, educators need to know both the position and the direction of their students’ progress.
And most importantly, without data, all we have is opinion!
[CLICK]
Without data, all we have is an opinion!

65. PA’s Intent for PVAAS Another Tool for Continuous Improvement
Measure of Growth to Add to Achievement/Status Measures
Not to Be Used in Isolation
No Teacher-Level Analyses
PVAAS is another tool as part of the cadre of analysis tools provided by PDE for districts—such as the eMetric’s PSSA Data Interactive and pa.ayp.com.
PVAAS is a measure of growth/progress to add to achievement/status measures.
Not to be used in isolation in decision making; part of collaborative dialogue in districts/schools.
There is no teacher-level Value-Added analysis in PVAAS. The data that PDE is sending to SAS, Inc. (the PVAAS vendor) does not include the data that would be needed to do teacher-level analyses.
[CLICK]

66. Questions:

67. 333 Market Street Harrisburg, PA 17126
For PVAAS information,
-- Select Assessment, then PVAAS on left menu bar
333 Market Street Harrisburg, PA 17126
Gerald L. Zahorchak, D.Ed Secretary of Education Commonwealth of Pennsylvania