Validity and Item Analysis Chapter 4

 Concerns what instrument measures and how well it does so  Not something instrument “has” or “does not have”  Informs counselor when it is appropriate to use instrument and what can be inferred from results  Reliability is a prerequisite for validity Validity

 Content-Related  Criterion-Related  Construct Traditional Categories of Validity

 Evidence Based on Test Content  Evidence Based on Response Processes  Evidence Based on Internal Structure  Evidence Based on Relations to Other Variables  Evidence Based on Consequences of Testing Validation Evidence: Standards for Educational and Psychological Testing

 Degree to which the evidence indicates that items, questions, or tasks adequately represent intended behavior domain  Central focus is typically on how the instrument’s content was determined  Content-related validation evidence should not be confused with face validity Evidence Based on Test Content

 Concerns whether individuals respond in a manner consistent with construct measured  May also examine information processing differences by subgroup  Computer simulated performances Evidence Based on Response Processes

 Examining internal structure using factor analysis  Can also examine internal structure of instrument for different subgroups (differential item functioning) Evidence Based on Internal Structure

 Correlational method  Convergent and discriminant evidence  Multitrait-multimethod matrix  Prediction or instrument-criterion relationship  Concurrent and predictive validity  Regression  Decision Theory  Validity Generalization Evidence Based on Relations to Other Variables

Correlational Method  Correlational method  examines relationship between instrument and criterion or other pertinent variables  Validity coefficient  Multitrait-multimethod matrix  examines relationships between instrument and traits that are theoretically related and unrelated to it

 Concurrent validity  Predictive validity Prediction/Instrument-Criterion Relationship

Regression a = y-intercept b = slope X = score

Standard Error of Estimate  The margin of expected error in the individual’s predicted criterion score as a result of imperfect validity

Decision Theory: Expectancy Table

Expectancy Table (cont.)

 Method of combining validation studies to determine if validity evidence can be generalized  Must have a substantial number of studies  Must use meta-analytic procedures Validity Generalization

 Examples of social consequences:  Group differences on tests used for employment selection  Group differences in placement in special education  Counselors should consider both validation evidence and social implications Evidence-Based Consequences of Testing

 Gradual accumulation of evidence  Counselor must evaluate the information to determine if it appropriate for which client under what circumstance  Validation evidence should also be considered in informal assessments Conclusion on Validation Evidence

 Examining and evaluating each item in instrument  Item difficulty: p = # who answered item correctly total # of individuals  Item discrimination:  Extreme group method: d = upper % – lower %  Correlational method Item Analysis

 Focus is on each item and establishing items that measure ability or level of a latent trait  Involves examining item characteristic curve and calibration of each individual item  Parameters:  Difficulty  Slope (discrimination)  Probability of guessing Item Response Theory

Item Characteristic Curve