Roadmap Towards a Validity Argument Peggy Garza Associate BILC Secretary

Views of Validity Traditional Current Adapted from Chapelle (1999) Validity was considered a characteristic of a test: the extent to which a test measures what it is supposed to measure. Validity is considered an argument concerning test interpretation and use: the extent to which test interpretations and uses can be justified. Reliability was seen as distinct from and a necessary condition for validity. Reliability can be seen as on type of validity evidence. Validity was often established through correlations of a test with other tests. Validity is argued on the basis of a number of types of evidence, including the consequences of testing. Adapted from Chapelle (1999)

Validity Argument Glossary definition: Interpretive argument that presents evidence to make a case justifying the score-based inferences and the intended uses of the test. Validity Argument

Testing Purpose Develop a Validity Argument for a particular test use NATO Interoperability Capability Targets Job Descriptions Training or exercise requirements From STANAG 6001: Participating nations adopt the appended table of language proficiency levels for the purpose of: Communicating language requirements for international staff appointments. Recording and reporting, in international correspondence, measures of language proficiency. Comparing national standards through a standardized table while preserving each nation’s right to maintain its own internal proficiency standards.

How are SLPs Officially Used in NATO? E1101 outlines English Language Capability Targets For NATO Command Structure staff / participation: As described by job description, OR Officers SLP 3 3 3 3 NCOs SLP 2+ 2+ 2+ 2+ For NATO operations, exercises or training Officers in command positions and principal staff officers SLP 2+ 2+ 2+ 2+ All other officers SLP 2 2 2 2 NCOs OR-5 and above likely to have contact with personnel from other nations SLP 2 2 2 2 Enlisted personnel planned to operate tactical comms, or on NATO comms networks or are members of tactical air control elements SLP 2 2 1 1

Job Description Requirement for English: SLP 2222 What does this really mean? SLP 2 1 2 1 Not Qualified  SLP 2 2 3 2 Qualified  Test validation is the process of making a case for the interpretation and uses of test scores.

Audience for Validity Argument Determine the audience for your validity argument “An argument is made and judged by an audience” Chapelle (2011) National stakeholders BILC: IAW sound practices in STANAG 6001 testing

Tasks, criteria, instruction Score need Test design & development Administration/ performance Rating/ evaluation Score use Purpose of assessment Tasks, criteria, instruction Performance Criteria Scores Validity evidence is created and collected throughout the test development cycle. Starting at the planning phase, tasks, criteria and instructions must correspond to STANAG 6001 descriptors, CTA statements, and established test development guidelines. Diagram adapted from Luoma 2004

Getting started : collecting & organizing evidence to support a validity argument for National STANAG 6001 Tests Table of Contents: Test Information for Stakeholders Testing Personnel Qualifications and Training Records Outline or Summary of Training Session Content Test Specifications Test Moderation Checklists and Documents Test Validation Documents and Statistical Data Test Administration Procedures Test Security Handbook Records of Norming Sessions for Raters Statistical Data on Rater Reliability Records and Summaries of a Priori Research Records and Summaries of a Posteriori Research _________________

Roadmap Activity Tasks Identify your audience(s) List evidence or documentation in as many Building Blocks as possible Share/exchange ideas and examples of evidence

Bibliography Chapelle, C. A. (2012). Validity argument for language assessment: The framework is simple. Language Testing, 29(1), 19–27. Chapelle, C. A. (1999). Validity in language assessment. Annual Review of Applied Linguistics, 19(1), 254–272. Luoma, S. (2004). Assessing Speaking. Cambridge: Cambridge University Press.