Using Summative Data to Monitor Student Performance: Choosing appropriate summative tests. Presented by Philip Holmes-Smith School Research Evaluation.

Slides:



Advertisements
Similar presentations
Assessing Student Performance
Advertisements

Fluency Assessment Bryan Karazia.
Using NAPLAN summative data to identify instructional successes and challenges. Presented by Philip Holmes-Smith School Research Evaluation and Measurement.
Overview of Formative Assessment (Diagnostic & Feedback) vs. Summative Assessment (Mastery & Standardised) Presented by Philip Holmes-Smith School Research.
Choosing appropriate summative tests.
Standards, data and assessment. Links to Tfel 1.6 Design, plan and organise for teaching and learning 2.4 Support and challenge students to achieve high.
Assessment in 2010 and Beyond: An Australian Perspective A/P Heather Fehring 20 th January 2010.
Another addition to the ACER assessment suite….
Universal Screening: Answers to District Leaders Questions Are you uncertain about the practical matters of Response to Intervention?
STAR Basics.
About the tests PATMaths Fourth Edition:
Principal Performance Evaluation System
PAT - MATHS Progressive Achievement Tests in Mathematics 3 rd Edition.
Chapter 9 Fluency Assessment Tina Jensen. What? Fluency Assessment Consists of listening to students read aloud for a given time to collect information.
September 13, 2014 The Whole School Success Partnership Saturday, September 13th, 2014.
Using Formative (Diagnostic) Data to Plan for Instruction: NAPLAN Reading, Numeracy, Spelling & Grammar and Punctuation Presented by Philip Holmes-Smith.
Using Data to Improve Student Achievement Summer 2006 Preschool CSDC.
ASSESSMENT FOR BETTER LEARNING USING NAPLAN DATA Presented by Philip Holmes-Smith School Research Evaluation and Measurement Services.
Richards Middle School Columbus, Georgia
Office of Institutional Research, Planning and Assessment January 24, 2011 UNDERSTANDING THE DIAGNOSTIC GUIDE.
Module 3: Unit 1, Session 2 MODULE 3: ASSESSMENT Adolescent Literacy – Professional Development Unit 1, Session 2.
Assessment & Evaluation of Learning Guest Lecturer: Diane Powell Dip Teaching (Primary), BEd, MEd (Ed Leadership) (Assistant Principal - Kismet Park PS)
EDU 385 EDUCATION ASSESSMENT IN THE CLASSROOM
Overview of Summative (Mastery & Standardised) Testing vs. Formative (Diagnostic) Testing Presented by Philip Holmes-Smith School Research Evaluation and.
Overview to Common Formative Assessments (CFAs) Adapted from The Leadership and Learning Center Presented by Jane Cook & Madeline Negron For Windham Public.
Port Phillip/Bayside Network: Using student achievement data to inform improved teaching programs Presented by Philip Holmes-Smith (School Research Evaluation.
1 Curriculum Based Measures Improving Student Outcomes through Progress Monitoring.
Standardized Testing (1) EDU 330: Educational Psychology Daniel Moos.
ELA & Math Scale Scores Steven Katz, Director of State Assessment Dr. Zach Warner, State Psychometrician.
Using Data to Improve Student Achievement Summer 2006 Preschool CSDC.
Using Data to Improve Student Achievement Summer 2006 Preschool CSDC.
Using Data to Improve Student Achievement Summer 2006 Preschool CSDC.
Introduction to CEM Secondary Pre-16 Information Systems Nicola Forster & Neil Defty Secondary Systems Programme Managers London, June 2011.
Georgia will lead the nation in improving student achievement. 1 Georgia Performance Standards Day 3: Assessment FOR Learning.
CEM (NZ) Centre for Evaluation & Monitoring College of Education Dr John Boereboom Director Centre for Evaluation & Monitoring (CEM) University of Canterbury.
Life without Levels Assessing children without levels.
Anthony Speranza St Mark’s Primary School Dingley, Victoria ICT / T&L Leader Debbie Darvell St Mark’s Primary School, Dingley, Victoria Principal Philip.
Broomfields Junior School Y6 ASSESSMENT INFORMATION EVENING THURSDAY 28 TH JANUARY 2016.
KS2 Parent Workshop Assessment without levels End of KS2 tests
SAT’s Information Parent’s Meeting 10 th February February 2016.
Assessment at CPS A new way of working. Background - No more levels New National Curriculum to be taught in all schools from September 2014 (apart from.
Helmingham Community Primary School Assessment Information Evening 10 February 2016.
Lostock Gralam CE Primary School Parent Information Meeting January 2016.
Agenda  What is On Demand Testing?  Types of Tests Available  Uses and Benefits of On Demand Testing  Progress Tests  Linear Test Reports (Progress.
Using Data for Coaching Presented By Outer Sub Region Coaches.
Assessment at Castle Batch Assessment is at the heart of teaching and learning at our school: It provides evidence to guide teaching and learning.
 The introduction of the new assessment framework in line with the new curriculum now that levels have gone.  Help parents understand how their children.
Assessment Assessment is the collection, recording and analysis of data about students as they work over a period of time. This should include, teacher,
CEM (NZ) Centre for Evaluation & Monitoring College of Education Dr John Boereboom Director Centre for Evaluation & Monitoring (CEM) University of Canterbury.
Information for Parents on Key Stage 2 SATs. When do these tests happen? Key Stage 2 SATs take place nationally in the week commencing 9th May Children.
Year 2 and Year 6 SATs Parents Evening Thursday 5 th May 2016.
KS1 SATS Guidance for Parents
Blackshaw Primary School.  DfE – statutory assessments:  Reception – Baseline, EYFS profile  Year 1 (and 2) - Phonics Check  Year 2 and 6 - end of.
Curriculum Night Elementary. What do I as a parent need to know to support student assessments at CCAS? Essential Question.
Setting NAPLAN DATA Targets Presented by Philip Holmes-Smith School Research Evaluation and Measurement Services.
Interpretation. Rationale Interpretation “… gathering, recording, interpreting, using and reporting information about a child’s progress and achievement.
Using Data to Improve Student Achievement Summer 2006 Preschool CSDC.
PARENTS’ INFORMATION SESSION -YEAR 6 SATS 2017
Create a Strong Start ACT® Aspire ®.
Understanding your PreACT scores
UPMINSTER INFANT SCHOOL
PARENTS’ INFORMATION SESSION -YEAR 6 SATS 2017
Key Stage 1 SATs Meeting.
Understanding your PreACT scores
Responding to Recent Debates in Education: Review of KS2 Testing, Assessment & Accountability
Key Stage 1 SATs Meeting.
JACKSON SCHOOL DISTRICT Middle School Math Informational Night
Responding to Recent Debates in Education: Review of KS2 Testing, Assessment & Accountability
What is this PAT information telling me about my own practice and my students? Leah Saunders.
Presentation transcript:

Using Summative Data to Monitor Student Performance: Choosing appropriate summative tests. Presented by Philip Holmes-Smith School Research Evaluation and Measurement Services

Overview of the session 1.Diagnostic vs. Summative Testing 2.Choosing Appropriate Summative Tests The reliability of summative (standardised) tests. Choosing appropriate summative tests. When should you administer summative tests?

1.Overview of Diagnostic vs. Summative Testing

Examples of Diagnostic Testing Assessment tools such as: Assessment tools such as: –Marie Clay Inventory, –English Online Assessment (Government schools), –Maths Online Assessment (Government schools), –SINE (CEO schools), –On-Demand Linear Tests, –Probe. Teacher assessments such as: Teacher assessments such as: –teacher questioning in class, –teacher observations, –student work (including portfolios).

Diagnostic Testing Research shows that our most effective teachers constantly use feedback (including the use of diagnostic information to inform their teaching). Research shows that our most effective teachers (in terms of improving the learning outcomes of students) constantly use feedback (including the use of diagnostic information to inform their teaching). Hattie (2003, 2009)* shows that using feedback (including using diagnostic information about what each student can and can’t do to inform teaching) has one of the biggest impacts on improving student learning outcomes. Hattie (2003, 2009)* shows that using feedback (including using diagnostic information about what each student can and can’t do to inform teaching) has one of the biggest impacts on improving student learning outcomes. 2003: : Hattie, John. (2009). Visible Learning: A synthesis of over 800 meta-analyses relating to achievement. NY: Routledge. *

Examples of Summative (Standardised) Testing Government sponsored assessment tools such as: Government sponsored assessment tools such as: –NAPLAN, –English Online Assessment (Government schools), –On-Demand Adaptive Tests. Other commercial tests such as: Other commercial tests such as: – TORCH, – PAT-R, – PAT-Math I Can Do Maths). – PAT-Math (together with I Can Do Maths).

Summative (Standardised) Testing Summative testing is essential to monitor the effectiveness of your teaching. Summative testing is essential to monitor the effectiveness of your teaching. But, research shows that summative tests do not lead to improved learning outcomes. As the saying goes: But, research shows that summative tests do not lead to improved learning outcomes. As the saying goes: “You don’t fatten a pig by weighing it” So, although it is essential, keep summative testing to a minimum. So, although it is essential, keep summative testing to a minimum.

2. Summative Tests

Summative (Standardised) Testing Summative testing is essential to monitor the effectiveness of our teaching, but: Summative testing is essential to monitor the effectiveness of our teaching, but: – Is NAPLAN reliable for all students? – Are the other summative tests you administer reliable for all students? We need to maximise the reliability of the tests we use to monitor the effectiveness of our teaching. We need to maximise the reliability of the tests we use to monitor the effectiveness of our teaching.

Summative (Standardised) Testing Summative testing is essential to monitor the effectiveness of our teaching, but: Summative testing is essential to monitor the effectiveness of our teaching, but: – Do we currently gather enough information to monitor the effectiveness of our teaching of ALL students? e.g. Year 3 NAPLAN reflects the effectiveness of your Prep-Yr2 teaching but what about the Prep teaching vs. Yr1 teaching vs. the Yr2 teaching? Year 3 NAPLAN reflects the effectiveness of your Prep-Yr2 teaching but what about the Prep teaching vs. Yr1 teaching vs. the Yr2 teaching? Year 9 NAPLAN reflects the effectiveness of your Yr7-Yr8 teaching but what about the Yr 7 teaching vs. Yr 8 teaching? Year 9 NAPLAN reflects the effectiveness of your Yr7-Yr8 teaching but what about the Yr 7 teaching vs. Yr 8 teaching? We need choose appropriate summative tests to monitor the effectiveness of our teaching at all year levels from Prep – Yr10! We need choose appropriate summative tests to monitor the effectiveness of our teaching at all year levels from Prep – Yr10!

The Reliability of Summative Tests

Three Questions 1.Do you believe that your students’ NAPLAN and/or On-Demand results accurately reflect their level of performance?

Three Questions 1.Do you believe that your students’ NAPLAN and/or On-Demand results accurately reflect their level of performance? 2.If we acknowledge that the odd student will have a lucky guessing day or a horror day, what about the majority? – Have your weakest students received a low score? – Have your average students received a score at about expected level? – Have your best students received a high score?

Three Questions 1.Do you believe that your students’ NAPLAN and/or On-Demand results accurately reflect their level of performance? 2.If we acknowledge that the odd student will have a lucky guessing day or a horror day, what about the majority? – Have your weakest students received a low score? – Have your average students received a score at about expected level? – Have your best students received a high score? 3.Think about your students who received high and low scores: – Are your low scores too low? – Are your high scores too high?

Is this reading score reliable? High highs and Low lows

Item difficulties for a typical test

Summary Statements about Scores Low scores (i.e. more than 0.5 VELS levels below expected) indicate poor performance but the actual values should be considered as indicative only Low scores (i.e. more than 0.5 VELS levels below expected) indicate poor performance but the actual values should be considered as indicative only (i.e. such scores are associated with high levels of measurement error). (i.e. such scores are associated with high levels of measurement error). High scores (i.e. more than 0.5 VELS levels above expected) indicate good performance but the actual values should be considered as indicative only. High scores (i.e. more than 0.5 VELS levels above expected) indicate good performance but the actual values should be considered as indicative only. (i.e. such scores are associated with high levels of measurement error). (i.e. such scores are associated with high levels of measurement error). Average scores indicate roughly expected levels of performance and the actual values are more reliable Average scores indicate roughly expected levels of performance and the actual values are more reliable (i.e. such scores are associated with lower levels of measurement error). (i.e. such scores are associated with lower levels of measurement error).

Choosing appropriate summative tests

Item Difficulties for Booklet 6 on the PAT-R (Comprehension) scale score scale Average Item Difficulty

Converting Raw test Scores Booklet 6 ToPAT-R(Comprehension) scale score

PAR-R (Comprehension) Tests Test difficulties of the PAR-R (Comprehension) Tests on the TORCH score scale together with Year Level mean scores

Different norm tables for different tests

Source: Source: ACER, 2006 ACER, 2006 Test difficulties of the PAT-Maths Tests on the PATM scale score scale together with Year Level mean scores Which is the best test for an average Year 4 student? Year 1 Year 2 Year 3 Year 4 Year 5 Year 6&7 Year 8&9 Year 10

Source: Source: ACER, 2006 ACER, 2006 Year 1 Year 2 Year 3 Year 4 Year 5 Year 6&7 Year 8&9 Year 10 Test difficulties of the PAT-Maths Tests on the PATM scale score scale together with Year Level mean scores The best test for an average Year 4 student is probably Test 4 or 5

Things to look for in a summative test Needs to have a single developmental scale that shows increasing levels of achievement over all the year levels at your school. Needs to have a single developmental scale that shows increasing levels of achievement over all the year levels at your school. Needs to have “norms” or expected levels for each year level (e.g. The National “norm” for Yr 3 students on TORCH is an average of 34.7). Needs to have “norms” or expected levels for each year level (e.g. The National “norm” for Yr 3 students on TORCH is an average of 34.7). Needs to be able to demonstrate growth from one year to the next (e.g. during Yr 4, the average student grows from a score of 34.7 in Yr 3 to an expected score of 41.4 in Yr 4 – that is 6.7 score points). Needs to be able to demonstrate growth from one year to the next (e.g. during Yr 4, the average student grows from a score of 34.7 in Yr 3 to an expected score of 41.4 in Yr 4 – that is 6.7 score points). As a bonus, the test could also provides diagnostic information. As a bonus, the test could also provides diagnostic information.

Norms for Year 3 to Year 10 on the TORCH scale TORCH scale TORCH NORMS 50 th Percentile 10 th Percentile 90 th Percentile

My Recommended Summative Tests (Pen & Paper) Reading Comprehension Reading Comprehension – Progressive Achievement Test - Reading (Comprehension) (PAT-R, 4 th Edition) – TORCH and TORCH plus Mathematics Mathematics – Progressive Achievement Test - Mathematics (PAT-Maths, 3 rd Edition) combined with the I Can Do Maths

Selecting the correct PAT-C Test

Selecting the correct TORCH Test

Selecting the correct PAT-Math/ICDM Test

My Recommended Summative Tests (On-Line) On-Demand - Reading Comprehension On-Demand - Reading Comprehension – The 30-item “On-Demand” Adaptive Reading test On-Demand - Spelling On-Demand - Spelling – The 30-item “On-Demand” Adaptive Spelling test On-Demand - Writing Conventions On-Demand - Writing Conventions – The 30-item “On-Demand” Adaptive Writing test On-Demand – General English (Comprehension, Spelling & Writing Conventions) On-Demand – General English (Comprehension, Spelling & Writing Conventions) – The 60-item “On-Demand” Adaptive General English test On-Demand - Mathematics (Number, Measurement, Chance & Data and Space) On-Demand - Mathematics (Number, Measurement, Chance & Data and Space) – The 60-item “On-Demand” Adaptive General Mathematics test On-Demand - Number On-Demand - Number – The 30-item “On-Demand” Adaptive Number test On-Demand – Measurement, Chance & Data On-Demand – Measurement, Chance & Data – The 30-item “On-Demand” Adaptive Measurement, Chance & Data test On-Demand - Space On-Demand - Space – The 30-item “On-Demand” Adaptive Space test

Choosing the right starting point is still important (even for “Adaptive” Tests)

Summative Testing and Triangulation Even if you give the right test to the right student, sometimes, the test score does not reflect the true ability of the student – every measurement is associated with some error. Even if you give the right test to the right student, sometimes, the test score does not reflect the true ability of the student – every measurement is associated with some error. To overcome this we should aim to get at least three independent measures – what researchers call TRIANGULATION. To overcome this we should aim to get at least three independent measures – what researchers call TRIANGULATION. This may include: This may include: – Teacher judgment – NAPLAN results – Other pen & paper summative tests (e.g. TORCH, PAT-R, PAT- Maths, I Can Do Maths) – On-line summative tests (e.g. On-Demand ‘Adaptive’ testing, English Online)

Summative Testing and Triangulation BUT remember, more summative testing does not lead to improved learning outcomes so keep the summative testing to a minimum BUT remember, more summative testing does not lead to improved learning outcomes so keep the summative testing to a minimum

When should you administer summative tests?

Timing for Summative Testing Should be done at a time when teachers are trying to triangulate on each student’s level of performance. (i.e. mid-year and end-of-year reporting time.) Should be done at a time when teachers are trying to triangulate on each student’s level of performance. (i.e. mid-year and end-of-year reporting time.) Should be done at a time that enables teachers to monitor growth – say, every six months. (i.e. From the beginning of the year to the middle of the year and from the middle of the year to the end of the year.) Should be done at a time that enables teachers to monitor growth – say, every six months. (i.e. From the beginning of the year to the middle of the year and from the middle of the year to the end of the year.)

Suggested timing For Year 1 – Year 6 and Year 8 – Year 10 For Year 1 – Year 6 and Year 8 – Year 10 – Early June (for mid-year reporting and six-monthly growth*) – Early November (for end-of-year reporting and six-monthly growth) For Prep and Year 7 and new students at other levels For Prep and Year 7 and new students at other levels – Beginning of the year (for base-line data) – Early June (for mid-year reporting and six-monthly growth) – Early November (for end-of-year reporting and six-monthly growth) * November results from the year before form the base-line data for the current year. (i.e. February testing is not required for Year 1 – Year 6 or for Year 8 – Year 10)