1. Using NAPLAN summative data to identify instructional successes and challenges. Presented by Philip Holmes-Smith School Research Evaluation and Measurement Services

2. Overview of Session Using NAPLAN summative data to identify instructional successes and challenges Understanding the National Scale, School Summary analysis, Five-Year Trend analysis, Individual student-level data analysis Value added (Growth) analysis

3. Understanding the National Scale

4. National Curriculum The National Scale is an arbitrary scale – at this stage it is not related to points along a developmental curriculum. But, it is highly likely that it will be mapped onto the National Curriculum at some time in the future.

5. fixed in 2008 The National Scale was fixed in 2008 as follows: – – Range: 0-1000 – – Mean: 500 – – Standard Deviation: 100 (i.e. 68% students between 400-600) +1 STD 2008 National Mean -1 STD 900 800 700 600 500 400 300 200 100 Understanding the National Scale

6. National Averages (2008) Only 20.1% of Australian school communities are more advantaged My School Website (2010) School’s Average National Average Average for school’s with similar ICSEA

7. National Curriculum The National Scale is an arbitrary scale – at this stage it is not related to points along a developmental curriculum. But, it is highly likely that it will be mapped onto the National Curriculum at some time in the future. fixed in 2008 The National Scale was fixed in 2008 as follows: – – Range: 0-1000 – – Mean: 500 – – Standard Deviation: 100 (i.e. 68% students between 400-600) Bands The Scale has been divided into ten Bands which are used for reporting to parents. – – Band 1 covers all scores equal to or less than 270. – – Bands 2 – 9 increment by 52 score points each Band – – Band 10 covers all scores above 686. Understanding the National Scale

8. National Curriculum The National Scale is an arbitrary scale – at this stage it is not related to points along a developmental curriculum. But, it is highly likely that it will be mapped onto the National Curriculum at some time in the future. fixed in 2008 The National Scale was fixed in 2008 as follows: – – Range: 0-1000 – – Mean: 500 – – Standard Deviation: 100 (i.e. 68% students between 400-600) Bands The Scale has been divided into ten Bands which are used for reporting to parents. – – Band 1 covers all scores equal to or less than 270. – – Bands 2 – 9 increment by 52 score points each Band – – Band 10 covers all scores above 686. Victorian students in 2011: At this stage Bands have no explicit curriculum meaning but results show that for Victorian students in 2011: – – A typical Yr3 level of performance is at the bottom of Band 5 – – A typical Yr5 level of performance is almost halfway into Band 6 – – A typical Yr7 level of performance is a third into Band 7 – – A typical Yr9 level of performance is at the bottom of Band 8 Yr 3 Yr 5 Yr 7 Yr 9 Understanding the National Scale

9. Parent Reports (NB. Bands do not equate to Victorian Year Level means) Year 3 Year 5 Year 7 Year 9

10. Victorian State Averages - 2011 (By Year Level and Dimension)

11. Cautionary Note #1 Equal scores amongst different dimensions (on the National Scale) do not equate to equal levels of performance in terms of expected VELS levels. e.g. – A National Yr9 Reading score of 586.6 is equivalent to a VELS score of 5.25, but – A higher National Yr9 Numeracy score of 592.5 is equivalent to a lower VELS score of 4.82.

12. Comparing National Scale Scores to Estimated VELS Equivalent scores Comparing National Scale Scores to Estimated VELS Equivalent scores The Victorian means for Year 3 and Year 9 Reading and Numeracy on the National scale are compared to the estimated VELS equivalent scores below: Compared to our expected curriculum outcomes for Year 3 students (2.175), the State Reading mean is about 1½ term ahead of where we expect a typical Year 3 student to be. However, the State Numeracy mean is about 2½ terms below where we expect a typical Year 3 student to be. Compared to our expected curriculum outcomes for Year 9 students (5.175), the State Reading mean is just above where we expect a typical Year 9 student to be. However, the State Numeracy mean is about 2½ terms below where we expect a typical Year 9 student to be. Year 3 Victoria MeansYear 9 Victoria Means Dimension National Scale VELS Equivalent National Scale VELS Equivalent Reading434.52.39586.65.25 Numeracy414.11.85592.54.82

13. Comparing AIM scores (VELS May equivalent) to NAPLAN scores (Est. VELS equivalent)

14. Comparing AIM scores (VELS May equivalent) to NAPLAN scores (Est. VELS equivalent)

15. Comparing AIM scores (VELS May equivalent) to NAPLAN scores (Est. VELS equivalent)

16. Comparing AIM scores (VELS May equivalent) to NAPLAN scores (Est. VELS equivalent)

17. Comparing AIM scores (VELS May equivalent) to NAPLAN scores (Est. VELS equivalent)

18. Cautionary Note #2 “Estimated VELS Equivalent Scores” should be viewed with caution. Specifically: “Estimated VELS Equivalent Scores” should be viewed with caution. Specifically: – It is doubtful that the “Estimated VELS Equivalent Scores” for Writing, Spelling or Grammar & Punctuation are truly VELS scores. (The writing marking rubric does not use VELS criteria and there are no VELS criteria for Spelling or Grammar & Punctuation.) – “Estimated VELS Equivalent Scores” are based on an equating study that is now three years old.

19. School Summary Analysis

20. The School Summary Report Choose “School Summary Report” Click on “Preview Report” to view results Choose to look at All Students or Girls vs. Boys or LBOTE Students or ATSI Students Choose to look at National Scale Scores (Estimated VELS scores are problematic) 2010

21. The School Summary Report

22. Interpreting “box and whisker” graphs ? ? ?? There are 30 students in this Year Level. Therefore: – – 50% (or 15 students) are above the median. – – 50% (or 15 students are below the median. – – 50% (or 15 students) are inside the “box”. Half of these (7-8 students) are above the median and half (7-8 students) are below the median. – – 10% (or 3 students) are at or below the 10 th percentile “whisker”. – – 10% (or 3 students are at or above the 90 th percentile “whisker”. – – 15% (or 4-5 students) are spread between the 25 th down to the 10 th percentile. – – 15% (or 4-5 students) are spread between the 75 th up to the 90 th percentile. 15 students above median 15 students below median 7-8 spread above the median. 15 students inside the box. 7-8 bunched below the median. 3 students on or below the 10 th percentile. (Note, all we know is at least one student scored on the 10 th percentile.) 3 students on or above the 90 th percentile. (Note, all we know is at least one student scored on the 90 th percentile.) 4-5 students between the 75 th up to the 90 th percentile. 4-5 students between the 25 th down to the 10 th percentile.

23. Interpreting the School Summary Report 1. 1.Is the school’s median above, at or below the State median? School’s median is about half a band below the State median

24. Interpreting the School Summary Report 1. 1.Is the school’s median above, at or below the State median? 2. 2.For VELS only, is the Reading median above, at or below the Numeracy median? School’s median is about half a band below the State median Reading is higher than Numeracy

25. Interpreting the School Summary Report 1. 1.Is the school’s median above, at or below the State median? 2. 2.For VELS only, is the Reading median above, at or below the Numeracy median? 3. 3.How does the school’s spread compare to the State spread? School’s median is about half a band below the State median School has far fewer high performers than the State School has far more low performers than the State School’s median is at the State’s 25 th percentile Reading is higher than Numeracy

26. The School Summary Report Review your own School Summary Report Go to: https://naplands.vcaa.vic.edu.au/https://naplands.vcaa.vic.edu.au/ (Use your school’s NAPLAN Data Service logon and password to access the report)

27. Five-Year Trend Analysis

28. Difference scores show how much better or worse you are compared to the State average This year’s score appears to be part of an improving trend compared to the State average

29. Five-Year Trend Analysis Look at your own SPA data Go to: http://www.sreams.com.au/http://www.sreams.com.au/ (Use your SPA logon and password to access you SPA account or click on “Purchase SPA” to purchase and set up a new account for your school)

30. Individual Student-level Analysis: Grouping Students for Teaching

31. Individual Student-level Analysis Score that place a student one or more years below expected are coloured red Score that place a student at or about expected level are coloured yellow Score that place a student one or more years above expected are coloured green

32. Individual Student-level Analysis Group requiring more basic level material Group requiring grade level material Group requiring more advanced material Click on a dimension heading to sort the scores

33. Individual Student-level Analysis Look at your own SPA data Go to: http://www.sreams.com.au/http://www.sreams.com.au/ (Use your SPA logon and password to access you SPA account or click on “Purchase SPA” to purchase and set up a new account for your school)

34. Value-Added (Growth) Analysis

35. Growth Analysis (NAPLAN scale) Letitia appears to have grown 163.6 units from 436.8 in Yr3 to 600.4 in Yr5. Javier appears to have gone backwards 40.7 units from 506.2 in Yr3 to 465.5 in Yr5.

36. Growth Analysis (NAPLAN scale) On Average, the school has grown its students 66.79 units compared to the State’s 83.2 units. This is 16.41 units less than the state.

37. Growth Analysis (VELS scale) Letitia appears to have grown 1.64 VELS units from 2.52 in Yr3 to 4.16 in Yr5. Javier appears to have gone backwards 0.43 VELS units from 3.11 in Yr3 to 2.68 in Yr5.

38. Growth Analysis (VELS scale) On Average, the school has grown its students 0.68 VELS units compared to the State’s 0.84 VELS units. This is 0.16 VELS units less than the state.

39. Growth Analysis (VELS scale) Difference scores show how much more or less growth you show compared to the State growth This year’s growth appears to be part of an worsening trend compared to the State growth

40. Value-Added (Growth) Analysis Look at your own SPA data Go to: http://www.sreams.com.au/http://www.sreams.com.au/ (Use your SPA logon and password to access you SPA account or click on “Purchase SPA” to purchase and set up a new account for your school)