1. Validity and Item Analysis Chapter 4

2.  Concerns what instrument measures and how well it does so  Not something instrument “has” or “does not have”  Informs counselor when it is appropriate to use instrument and what can be inferred from results  Reliability is a prerequisite for validity Validity

3.  Content-Related  Criterion-Related  Construct Traditional Categories of Validity

4.  Evidence Based on Test Content  Evidence Based on Response Processes  Evidence Based on Internal Structure  Evidence Based on Relations to Other Variables  Evidence Based on Consequences of Testing Validation Evidence: Standards for Educational and Psychological Testing

5.  Degree to which the evidence indicates that items, questions, or tasks adequately represent intended behavior domain  Central focus is typically on how the instrument’s content was determined  Content-related validation evidence should not be confused with face validity Evidence Based on Test Content

6.  Concerns whether individuals respond in a manner consistent with construct measured  May also examine information processing differences by subgroup  Computer simulated performances Evidence Based on Response Processes

7.  Examining internal structure using factor analysis  Can also examine internal structure of instrument for different subgroups (differential item functioning) Evidence Based on Internal Structure

8.  Correlational method  Convergent and discriminant evidence  Multitrait-multimethod matrix  Prediction or instrument-criterion relationship  Concurrent and predictive validity  Regression  Decision Theory  Validity Generalization Evidence Based on Relations to Other Variables

9. Correlational Method  Correlational method  examines relationship between instrument and criterion or other pertinent variables  Validity coefficient  Multitrait-multimethod matrix  examines relationships between instrument and traits that are theoretically related and unrelated to it

10.  Concurrent validity  Predictive validity Prediction/Instrument-Criterion Relationship

11. Regression a = y-intercept b = slope X = score

12. Standard Error of Estimate  The margin of expected error in the individual’s predicted criterion score as a result of imperfect validity

13. Decision Theory: Expectancy Table

14. Expectancy Table (cont.)

16.  Method of combining validation studies to determine if validity evidence can be generalized  Must have a substantial number of studies  Must use meta-analytic procedures Validity Generalization

17.  Examples of social consequences:  Group differences on tests used for employment selection  Group differences in placement in special education  Counselors should consider both validation evidence and social implications Evidence-Based Consequences of Testing

18.  Gradual accumulation of evidence  Counselor must evaluate the information to determine if it appropriate for which client under what circumstance  Validation evidence should also be considered in informal assessments Conclusion on Validation Evidence

19.  Examining and evaluating each item in instrument  Item difficulty: p = # who answered item correctly total # of individuals  Item discrimination:  Extreme group method: d = upper % – lower %  Correlational method Item Analysis

20.  Focus is on each item and establishing items that measure ability or level of a latent trait  Involves examining item characteristic curve and calibration of each individual item  Parameters:  Difficulty  Slope (discrimination)  Probability of guessing Item Response Theory

21. Item Characteristic Curve