Introduction to the Workshop Peggy Garza Associate BILC Secretary

Testing the Test – How Valid is Our Validity Argument? Topics Validating criterion-referenced tests Qualitative versus quantitative analysis Piloting challenges Test validation partnerships Inter- and intra-rater reliability Data analysis tools Consequential validity Standard setting Modified Angoff procedure

23 Nations + 1 NATO HQ Armenia Austria Bosnia and Herzegovina Bulgaria Canada Czech Republic Denmark Estonia France Georgia Hungary Latvia Lithuania Macedonia Norway Poland Portugal Serbia SHAPE Slovenia Spain Sweden Tunisia United States of America

Facilitators Ray Clifford Mary Jo DiBiase Peggy Garza Roxane Harrison David Oglesby Branka Petek Nancy Powers Ulle Turk Jana Vasilj-Begovic

Workshop Objectives Share ideas and best practices on test validation Get answers to questions on test validation Develop an understanding of how to build a validity argument for national STANAG 6001 tests Learn about methods and tools for collecting evidence to support a validity argument

Workshop Format Plenary sessions Workshop activities Keynote speech Presentations Panel of national testing organizations Workshop activities

Workshop Activities Workshops 1&2: 1 ½ hours Workshops 3&4: 2 hours Workshop 5: 1 hour Small groups and practical hands-on work Topics #1 Reading Item Moderation: Alignment, led by Mary Jo DiBiase, Peggy Garza, Gerd Brendel, and David Oglesby #2 Piloting: Important Part of Test Validation, led by Branka Petek #3 Modified Angoff, led by Jana Vasilj-Begovic and Nancy Powers #4 Statistical Analysis Using jMetrik, led by Ulle Turk #5 Roadmap toward a Validity Argument led by Mary Jo DiBiase and Peggy Garza

What is a Validity Argument? Glossary definition Interpretive argument that presents evidence to make a case justifying the score-based inferences and the intended uses of the test. Is there a better definition?

Evidence What do we mean by evidence? Quantitative Qualitative

My Objective Obtain input for making improvements to the testing seminars on the topic of test validation