EXPLORING THE VALIDITY OF AUSTRALIA’S NATIONAL ASSESSMENT SYSTEM Assoc Prof Greg Thompson (QUT) Dr Lenore Adie (ACU) Prof Val Klenowski (QUT) BERA, Leeds, 2016

Validity of NAPLAN Validity: the most fundamental consideration in developing and evaluating tests (AERA, APA & NCME, 2014, p. 11) Lack of agreement about the best way to use the term ‘validity’ (Newton & Shaw, 2015) Validity is not the property of a test, but of the use of that test Each purpose of an assessment needs to be assessed for its own validity (Stobart, 2009) Lack of systematic analysis of the validity of NAPLAN

History of validity

Definition of validity The extent to which the proposed interpretations and uses of test scores are justified. The justification requires conceptual analysis of the coherence and completeness of the claims and empirical analyses of the inferences and assumptions inherent in the claims (Kane, 2015, p.1).

Argument-based approach to validity Two-step argument-based approach: Specifying intended interpretation and use of the test as an interpretation/use argument (IUA) which includes “inferences and assumptions leading from test performances to conclusions and decisions based on the test scores” (Kane, 2015, p. 4).   Develop the argument based on the whether the interpretations and uses of the test are supported by appropriate evidence.

Establishing an argument for the valid use of NAPLAN data “actual interpretations and uses of test scores in context are far more varied and nuanced than an intended interpretation might imply” (Moss, 2015: 1) NAPLAN Aims 1. to help drive improvements in student outcomes 2. provide increased accountability for the community (ACARA, 2011). In practice, these aims have become reinterpreted and reinscribed by jurisdictions, reports and policymakers

Two Illustrative Examples In these two contexts the proposed uses and interpretations of the NAPLAN scores have a negative impact on validity. 2014 writing test “Choose a rule or a law that you think needs to change and give your reasons” The ambiguity and lack of clarity of the task compromised the interpretations and uses of the results For what is being tested is not the students’ knowledge, skills and understanding pertinent to writing “but their ability to interpret the questions” (Baird, Ahmed, Hopfenbeck, Brown, & Elliott, 2013, p. 15) and “how well candidates cope with those surprises….” Ofqual, 2008, as cited by Baird et al., 2013, p. 15) The task was changed so that in 2015 Years 3 and 5 completed a different task to Years 7 and 9.

2. The uneven nature of participation across the population, and therefore the reasonableness of comparisons like MySchool A comparison between the average scores of the states must take into account the likelihood that what is being compared is as much different rates of participation as it is different levels of student achievement.

Implications for the validity of NAPLAN The use of test scores should be matched with accountability for those decisions. Arguments about the validity of the use of test data is mediated by the context and the stakes associated with the consequences. Disentangle the accountability and learning aims of NAPLAN Create a set of arguments for use and score interpretation targeted at specific groups such as teachers, principals, media, system authorities and policymakers as a means to assist them in understanding the limits of data - what data can and cannot be used for - to improve the assessment literacy of the population.

References ACARA. (2011). Why NAP. Retrieved March 11, 2014, from ACARA: Australian Curriculum and Reporting Authority: http://www.nap.edu.au/about/why-nap.html AERA, APA & NCME (2014). Standards for educational and psychological testing. Washington DC: AERA. Australian Government. (2014). Australian Government response to the Senate Education and Employment References Committee Report: Effectiveness of the National Assessment Program – Literacy and Numeracy. Accessed 6 November, 2015 https://docs.education.gov.au/system/files/doc/other/response_to_naplan_senate_report_0.docx Coburn, C.E. & Turner, E.O. (2011). Research on data use: A framework and analysis. Measurement: Interdisciplinary Research and Perspectives, 9(4), 173-206, DOI: 10.1080/15366367.2011.626729 Department of Education, Training and Employment. (2013). Great teachers = Great results. A direct action plan for Queensland schools. Queensland Government. Accessed 25 November, 2015 http://deta.qld.gov.au/about/government-responses/pdf/great- teachers-great-results-action-plan.pdf Kane, M.T. (2015). Explicating validity. Assessment in Education: Principles, Policy & Practice, DOI: 10.1080/0969594X.2015.1060192 Madaus, G., Russell, M., & Higgins, J. (2009). The Paradoxes of High Staks Testing: How they affect students, their parents, teachers, principals, schools. and society. Charlotte, NC: Information Age Publishing. Newton, P., & Shaw, S. (2015). Disagreement over the best way to use the word 'validity' and optios for reaching consensus. Assessment in Education: Principles, Policy and Practice, DOI: 10.1080/0969594X.2015.1037241 Nichols, S., & Berliner, D. (2007). Collateral Damage: How high-stakes testing corrupts America's schools. Harvard Education Press. Stobart, G. (2009). Determining validity in national curriculum assessments. Educational Research, 51(2), 161-179. Storey, A. (2000). A leap of faith? Performance pay for teachers. Journal of Education Policy, 15:5, 509-523, DOI: 10.1080/026809300750001667 Wu, M. (2010). The inappropriate use of NAPLAN data. Professional Voice, 8(1), 21-25.