1. EXPLORING THE VALIDITY OF AUSTRALIA’S NATIONAL ASSESSMENT SYSTEM
Assoc Prof Greg Thompson (QUT)
Dr Lenore Adie (ACU)
Prof Val Klenowski (QUT)
BERA, Leeds, 2016

2. Validity of NAPLAN
Validity: the most fundamental consideration in developing and evaluating tests (AERA, APA & NCME, 2014, p. 11)
Lack of agreement about the best way to use the term ‘validity’ (Newton & Shaw, 2015)
Validity is not the property of a test, but of the use of that test
Each purpose of an assessment needs to be assessed for its own validity (Stobart, 2009)
Lack of systematic analysis of the validity of NAPLAN

3. History of validity

4. Definition of validity
The extent to which the proposed interpretations and uses of test scores are justified. The justification requires conceptual analysis of the coherence and completeness of the claims and empirical analyses of the inferences and assumptions inherent in the claims (Kane, 2015, p.1).

5. Argument-based approach to validity
Two-step argument-based approach:
Specifying intended interpretation and use of the test as an interpretation/use argument (IUA) which includes “inferences and assumptions leading from test performances to conclusions and decisions based on the test scores” (Kane, 2015, p. 4).  
Develop the argument based on the whether the interpretations and uses of the test are supported by appropriate evidence.

6. Establishing an argument for the valid use of NAPLAN data
“actual interpretations and uses of test scores in context are far more varied and nuanced than an intended interpretation might imply” (Moss, 2015: 1)
NAPLAN Aims
1. to help drive improvements in student outcomes
2. provide increased accountability for the community (ACARA, 2011).
In practice, these aims have become reinterpreted and reinscribed by jurisdictions, reports and policymakers

7. Two Illustrative Examples
In these two contexts the proposed uses and interpretations of the NAPLAN scores have a negative impact on validity.
2014 writing test
“Choose a rule or a law that you think needs to change and give your reasons”
The ambiguity and lack of clarity of the task compromised the interpretations and uses of the results
For what is being tested is not the students’ knowledge, skills and understanding pertinent to writing “but their ability to interpret the questions” (Baird, Ahmed, Hopfenbeck, Brown, & Elliott, 2013, p. 15) and “how well candidates cope with those surprises….” Ofqual, 2008, as cited by Baird et al., 2013, p. 15)
The task was changed so that in 2015 Years 3 and 5 completed a different task to Years 7 and 9.

8. 2. The uneven nature of participation across the population, and therefore the reasonableness of comparisons like MySchool
A comparison between the average scores of the states must take into account the likelihood that what is being compared is as much different rates of participation as it is different levels of student achievement.

11. Implications for the validity of NAPLAN
The use of test scores should be matched with accountability for those decisions.
Arguments about the validity of the use of test data is mediated by the context and the stakes associated with the consequences. Disentangle the accountability and learning aims of NAPLAN
Create a set of arguments for use and score interpretation targeted at specific groups such as teachers, principals, media, system authorities and policymakers as a means to assist them in understanding the limits of data - what data can and cannot be used for - to improve the assessment literacy of the population.

12. References
ACARA. (2011). Why NAP. Retrieved March 11, 2014, from ACARA: Australian Curriculum and Reporting Authority:
AERA, APA & NCME (2014). Standards for educational and psychological testing. Washington DC: AERA.
Australian Government. (2014). Australian Government response to the Senate Education and Employment References Committee Report: Effectiveness of the National Assessment Program – Literacy and Numeracy. Accessed 6 November,
Coburn, C.E. & Turner, E.O. (2011). Research on data use: A framework and analysis. Measurement: Interdisciplinary Research and Perspectives, 9(4), , DOI: /
Department of Education, Training and Employment. (2013). Great teachers = Great results. A direct action plan for Queensland schools. Queensland Government. Accessed 25 November, teachers-great-results-action-plan.pdf
Kane, M.T. (2015). Explicating validity. Assessment in Education: Principles, Policy & Practice, DOI: / X
Madaus, G., Russell, M., & Higgins, J. (2009). The Paradoxes of High Staks Testing: How they affect students, their parents, teachers, principals, schools. and society. Charlotte, NC: Information Age Publishing.
Newton, P., & Shaw, S. (2015). Disagreement over the best way to use the word 'validity' and optios for reaching consensus. Assessment in Education: Principles, Policy and Practice, DOI: / X
Nichols, S., & Berliner, D. (2007). Collateral Damage: How high-stakes testing corrupts America's schools. Harvard Education Press.
Stobart, G. (2009). Determining validity in national curriculum assessments. Educational Research, 51(2),
Storey, A. (2000). A leap of faith? Performance pay for teachers. Journal of Education Policy, 15:5, , DOI: /
Wu, M. (2010). The inappropriate use of NAPLAN data. Professional Voice, 8(1),