What does exam validity really mean? Andrew Martin Purdue Pesticide Programs

What we’re going to do  Take a brief look at the evolution of validity theory from 1950 to the present.  Review modern validity theory.  Develop an analogy between test validation and pesticide enforcement.  Examine modern validity theory and its practical implications for pesticide applicator certification tests.

That’s your opinion. Everybody knows that. Our expert says…!  Sources 1. :  Cureton, E. (1951)  Cronbach, L. (1971)  Messick, S. (1989)  Shepard, L. (1993)  Kane, M. (2004)  AERA, APA, & NCME (1966, 1985, & 1999) 1. Full citations available upon request

Validity theory ca (does this sound familiar?)  It [validity] indicates how well the test serves the purpose for which it is used. Cureton (1951)

Validity defined ca  The validity of a test is an estimate of the correlation between raw test scores and the “true” (that is, perfectly reliable) criterion scores. Cureton (1951)

Problems with the criterion model  The criterion model raises two questions:  What counts as a good criterion measure?  How can the criterion measure be validated?

Validity theory ca  Validity information indicates the degree to which the test is capable of achieving certain aims…we may distinguish three of the numerous aims of testing:  Content validity  Criterion-related validity  Construct validity. APA, AERA, & NCME (1966)

How was the trinitarian model supposed to work?  Validation of an instrument calls for an integration of many types of evidence. The varieties of investigation are not alternatives any one of which would be adequate.The investigations supplement one another. Cronbach, 1971

Problems with the trinitarian model  The trinitarian model encouraged test developers to cherry-pick validity evidence.

Validity theory today  Validity is a unified concept… Messick, 1989 It is a judgment of the extent to which theory and evidence support the way test scores are interpreted and used. See also APA, AERA, & NCME, 1985 & 1999

Are there problems with a unified approach to validity?  [Current validity standards] do not help answer the question “How much evidence is enough?” Shepard, 1993

Validity and validation  Theory is supposed to guide practice.  Modern validity theory assumes a guiding role when it is coupled with a conception of validation as a practical argument.

Steps in an argument-based approach to validation  1. Specify the the proposed interpretation and use of the test scores.  2. Assemble available evidence to support the plausibility of test score interpretation and use.  3. Evaluate doubtful assumptions in detail.  4. Repeat steps 2 and 3 until the argument is plausible (or rejected) Kane, 2004

How is a validation argument like an enforcement case?  The test developer is supposed to:  gather appropriate evidence,  organize it in a compelling fashion,  and make a sound (plausible) case for how test scores will get interpreted and used.

What does this mean for me, the test developer (or how much evidence is enough)?  An argument-based approach to validation offers answers to two vexing questions:  1. How do I get started?  2. Where do I focus my resources?