1. Examining Consequential Validity Latvian experience
Inga Fārte-Zālīte Head, STANAG 6001 Testing Team
NAF Language School, Latvia
Brno, Sept 2016

2. Background
Consequential validity and what it implies
Guidelines for determining whether a given test should be used for a particular purpose in a particular way
Another way of looking at validity
Consequences of current test use in Latvia
Summary

3. Background STANAG tests in Latvia since 2003
Job/ rank requirements since 2008
Job/rank requirements since 2013
Plus levels since 2014

4. STANAG test in Latvian NAF
Qualification/ promotion a) for captains, senior & higher officers (SLP at least ; valid 5 years); b) for positions at NATO HQs (2 years); c) for studies at BALTDEFCOL (2 years)
At the end of an intensive course check the effectiveness of the training programme

5. Training and STANAG test
Intensive English Language Course 16 weeks; 3 locations;
divided in 7 different levels depending on entrance ALCPT score + short writing test for placement
One cannot retake the STANAG test while having a valid certificate, unless s/he has taken a language course

6. Consequential Validity
The idea of validity is connected to the idea of “inference-making”. Those inferences depend on how a test is used.
Tests are not used in a vacuum; they are always intended to serve the needs of an educational system or of society at large. (L.Bachman)
Cronbach., L. J. Tests have always been used as an impartial way to perform a political function – that of determining who gets what

7. Consequential Validity
We as testers must recognize that considerations about the rights and interests of test takers, institutions responsible for testing and making decisions based on those tests, and the public interests are essentially political; that they change over time; that they have implications a) for the practice of our professions; b) for the kinds of tests we develop and c) the ways in which we justify their use.
Hulin

8. Consequential Validity
We as testers must go beyond the scientific demonstration of empirical and logical evidence to the consideration of the potential consequences of testing;
From applied linguistics and testing → public policy.
Hulin

9. Guidelines for determining whether a given test should be used for a particular purpose in a particular way are not well defined.

10. Areas to be considered in the ethical use and interpretation of test results
the construct validity/ evidence that supports the particular interpretation we wish to make;
the value systems of test takers, developers and users and also those of the educational program and society at large that inform the particular test use;
the practical usefulness of the test;
the consequences of the education system or society of using test results for a particular purpose
Lyle Bachman, Fundamental Considerations in Language Testing, p
For example, if we feel it is necessary for our officers to be competent in xxx, we need to determine what evidence there is that ratings from the xxx test we intend to use for officer qualification are valid indicators of this ability.
If the test user feels that the ability to negotiate meaning by whatever linguistic means is more important than grammatical accuracy, how will this affect the way they interpret and use scores from tests of these abilities? To what extent does the type of test used reflect the personal bias of the test user, and to what extent does this differ from that of the test taker? If test takers and users feel that conversational use of the language if of greatest importance, to what extent is this value reflected in an all 4 skills test? Should our students be given passing marks and diplomas on the basis of their willingness to persevere in attending a tough 16week long intensive English course, or should these be awarded solely on the basis of merit, as determined by a criterion-referenced competency test?
If STANAG language tests are to be used for deciding whether a captain can become a major, what evidence is there to support this test use? What evidence is there that scores on ALCPT are relevant to training activities in Mali?

11. Consequences: continued
a) List the intended uses of the test;
b) List the potential consequences, both positive and negative, of using a particular test;
c) Rank them in terms of (un)desirability of their occurring + assign values;
d) Gather information that will help you determine the likelihood of each consequence (prior experience, empirical studies of relationships between similar tests and their consequences, historical documents)
In Lyle Bachman but Nitko, A.J. in Educational Tests and Measurement: an Introduction.

12. OR
Consider alternatives to testing as a means of achieving the same purposes
Lyle Bachman

13. What undermines valid use of test results
different stakeholders have competing agendas in education;
highly technical aspects of testing often are poorly understood within the general public;
test security can promote a sense of secrecy;
policy-makers and practitioners often can use testing results in ways that were not intended by test developers.
All of it feeds back into public mistrust.
Hitchcock, J.H., A.J. Onwuegbuzie and H.B. Khoshaim. Examining the consequential validity of standardized examinations via public perceptions: a review of mixed methods survey design considerations.

14. Validity = Overall Quality
allows one to express one’s opinion that some tests are better than others in a given situation, without having the connotation of precision and objectivity that the jargon of degrees of validity suggests.
Norman, G. The negative consequences of consequential validity.

15. Validity = a subjective summary of evidence
There is no coefficient like a reliability coefficient, saying this has a validity of 0.80 and that has a validity of 0.40.
There is good evidence of the validity of x.
There is good evidence of the lack of validity of y.
Norman, G. The negative consequences of consequential validity.
Rather it is inevitably, which of course means that others may have a different view. More often than not, it represents an informed and subjective aggregation of the available evidence.

16. Consequences for NAFL and testing
Senior officers who have problems achieving SLP choose to leave NAFL
Suggestions to change language testing policy – lower requirements, set up national standards/ tests
Organize STANAG test trips to Lithuania/ SHAPE/ elsewhere
Allow retaking the STANAG test until the required SLP is achieved
Establish appeals procedure involving independent experts (from other NATO countries)

17. Consequences for teaching
Job (self)evaluation for teachers
Assigning students to groups based on their desired end-of course STANAG test results, not entrance test results.
Evaluating a course based on their desired end-of course STANAG test results.

18. Summary
The questions involved in the appropriate use of the tests are not easy, but everyone has faced them and they will persist.
Appropriate answers will vary across cultures, specific contexts, and time.
Tests should be labeled just like dangerous drugs: «Use with care!» (Spolsky)

19. books and articles
Bachman, L. F. Fundamental considerations in language testing. OUP, pp
Hitchcock, J.H., A.J. Onwuegbuzie and H.B. Khoshaim. Examining the consequential validity of standardized examinations via public perceptions: a review of mixed methods survey design considerations. International Journal of Multiple Research Approches, Taylor & Francis, 2015.
Norman, G. The negative consequences of consequential validity. Springer Science+Business Media Dordrecht, 2015.
Welner, K. G. Consequential validity and the transformation of tests from measurement tools to policy tools. Teacher College Record Volume 115, Sept 2013.