1. Chapter 6 Validity §1 Basic Concepts of Validity
What is the Validity?
Interpretation
The validity of a test concerns what the test measure and how well it does so.
—Anne Anastasi

2. It tell us what can be inferred from test scores
—Anne Anastasi

3. Figure6.1 One Funny Picture

4. Validity can be defined as the agreement between a test score or measure and the quality it is believed to measure.
— Robert M. Kaplan
Dennis P. Saccuzzo
Does the test measure what it is supposed to measure?

5. Validity is the evidence for inferences made about a test score.
—AERA, APA, NCME
STANDARS FOR EDUCATIONAL AND PSYCHOLOGICAL TESTING

6. Validity effected by random and systematic errors.
Random errors and systematic errors both reduce the accuracy of the test.

7. Mathematic Definition of Validity
Validity coefficient is the ratio of
The variance concerned to the trait measured to observed score variance.
(6.1)

8. Comparing Validity with Reliability
The reliability of test is low, usually, the validity is low too;
The reliability of test is high, the validity isn’t necessarily high.
Figure Components of the Variance of Observed Scores
Reliability is a necessary premise for validity and validity represents the ultimate purpose of the test.

9. Content-Related validity Construct –Related Validity
Three Types of Validity
Criterion-Related Validity
Content-Related validity
Construct –Related Validity
Note: The most recent standards emphasize that validity is a unitary concept. The use of categories does not imply that there are distinct forms of validity

10. Effect Factors for Validity
Test Itself
Test Administration and Scoring
Examinees
The Criterion Chosen for Criterion Validity

11. Effect from test itself
The statement of the items is clear or not
The items represent the trait measured or not
The length of the test is adequate or not
The test difficulty is proper or not. …

12. Test administration and scoring
Whether the sample is representative, heterogeneous.
Whether the testing conditions are appropriate and unexpected disturbances occur.
Whether the tester administers the test according to the manual.
Whether the test guides for examinees are clear.
Whether the Scoring system is object and standard.

13. Interests and Motivation on the Test
Examinees
Interests and Motivation on the Test
Emotional State and Attitude During the Testing
State of Physical Health
Experiences on Test

14. The criterion chosen for criterion validity

15. §2 Content Validity and Construct Validity
Interpretation
Content validity involves the careful definition of the domain of behaviors to be measured by the test and the logical design of items to cover all the important areas of the domain.

16. The purpose of a content validity is to assess whether the items adequately represents a performance domain or construct of specific interest
It is established through a rational analysis of the content of a test.

17. Steps for Content Validation Using Experts Judgment
Defining the performance domain of interest
Selection a panel of qualified experts in the content domain
Providing a structured framework for the process of matching items to the performance domain
Collecting and summarizing the data from the matching process

18. Application
Content validity is most often employed with achievement test, so the performance domain is often defined by a list of instructional objectives.
Content validity is also applicable to certain occupational test designed for employee selection and classification.

19. Table6.1 Table of Instructional objectives
knowledge Comprehension application analysis evaluation synthesis
Sum
Chapter1
Chapter2
Chapter3
Chapter4 10 28 22 40
100

20. Distinction form Face Validity
The face validity refers to what it appears superficially to measure, not to what the test actually measures.

21. Construct Validity Interpretation
The construct validity of a test is the extent to which the test may be said to measure a theoretical construct or trait.

22. What is Construct?
Each construct is developed to explain and organize observed response consistencies. It
derives from established interrelationships among behavioral measures.
Examples: scholastic aptitude, intelligence, verbal fluency, anxiety, depression, self-esteem, etc..

23. Construct validation has focused attention on the role of psychological theory in test construction and on the need for formulate hypotheses that can be proved or disproved in validation process.
Anne Anastasi

24. Procedures for Construct Validation
Correlations between a measure of the construct and designated
Internal Consistency
Differentiation between Groups
Development Changes
Factor Analysis
Multitrait –multimethod matrix

25. Method Method Method 3
Trait A B C A B C A B C
1.True-False
A. Sex-Guilt (.95)
B. Hostility-Guilt (.86)
C. Morality-Conscience (.92)
2.Force Choice
A. Sex-Guilt (.95)
B. Hostility-Guilt (.76)
C. Morality-Conscience (.84)
3.Incomplete Sentences
A. Sex-Guilt (.48)
B. Hostility-Guilt (.41)
C. Morality-Conscience (.58)

26. Example How to Search the Evidences for a
Supposed Intelligence Test?
State the theory hypotheses of test:
1. Intelligence grows with the age growing
2. IQ is relatively stable
3. Intelligence is substantially related to school achievement
4.Intelligence is affected by inheritance
Administer the test to population and analyze the data. Judge: whether the test scores increase with the ages increasing; whether IQ and school achievements is correlated; IQs keep stably cross a time interval; whether the correlation between MZ is higher than the correlation between DZ.

27. §3 Criterion-Related Validity
Concepts
1.interpretation of Criterion-related Validity
It is the degree on which the test scores can be related to a criterion.
It indicate the effectiveness of a test in predicting an individual performance in specified activities.

28. Two Types
Predictive Validity refers to the degree to which test scores predict criterion measurement that will be made at some point in the future.
Concurrent Validity refers to the relationship between test scores and criterion measurements made at the time the test was given.

29. 2.What is criterion?
The Criterion is some behavior that the test scores are used to predicted.
For example, use the grade-point averages as the criterion of
a school admissions test .

30. The problems About Criterion The reliability of criterion
The validity of criterion
Whether it can be measured
Criterion contamination

31. Usually Used Criterion
academic achievement ( for intelligence test)
performance in specialized training (for special aptitude test)
job performance
contrasted group (for personality, domain-referenced test)
psychiatric diagnosis ( for personality test )
ratings by schoolteachers, job supervisor
previously available tests

32. Discrimination Between Two Groups
Procedures of Criterion-Related Validation
Validity Coefficient
Discrimination Between Two Groups

33. Estimate Validity Coefficient
Pearson Product Moment Correlation Coefficient

34. Exercise 1
Suppose that 10 male applicants were examined one job interests test and the admitted as salesman by one company. The job interest test scores (X) and the sale amount for the first year (Y, unit is “ten thousands $”) of each applicant are listed in the following table.
table Applicants’ Test Scores and Sale Amount
examinees X Y

35. Biserial Correlation Coefficient
(for correlation between a continuous variable and a dichotomous variable)
(6.2)
, is the percentage of examinees who get
point “1” on dichotomous variable
, is equal to 1-p
, is the mean of the test scores on the continuous variable
of the examinees who get point “1” on dichotomous variable
,is the mean of the test scores on the continuous variable
of the examinees who get point “0” on dichotomous variable
, is the Y oirdinate of the standard normal curvve at the z-score associated with the p value.
, is the standard deviation of test scores
for all examinees on continusous variable

36. Research Case
Use rb to estimate the validity of the fist application for WISC-R in Shanghai.
Data concerned:
the number of first level middle school students is 66
the number of second level middle school students is 286
the mean of IQs of the first level students is 114
the mean of IQs of the second level students is 96
the standard deviation of all students’ IQs is 14.53
if p=.1875, then Y is .2685

37. p=.1875,
then Y is .2685

38. Exercise 2
The middle school students attended a math test. The mean scores of students who have been instructed with higher math program is , and their number is 382. The mean of the students who have accepted normal program is , and their number is 618. The standard deviation for all students is Please estimate the validity coefficient of the math test.

39. 2. Discrimination Between Two Groups
Compare the means of two groups (t Test)
Degree of freedom

40. Compute the overlap amount of the two groups Method 1
Compute the number of the examinees from one group (usually contrasted )whose test scores is higher than the mean of the other group;
Compute the rate of the number of those test scores is higher than the mean for the other group;
Then calculate the rate of the two numbers.

41. Method 2
Compute the overlap percentage of the score distribution for each group

42. §4 Application of Validity Coefficient
Predict the Criterion Score
Establish Regression Equation
, is the predicted criterion
score for a examinee
, the test score of a examinee
, is the regression coefficient, and
, is the intercept, and

43. Example Figure 6.3 100 Examinees ’ Scores on Job Aptitude Test and Real Performance Scores

45. If one applicant get 6 on the test, then we can use the regression equation to predict his job performance in the future.

46. Exercise 3
Suppose a group of students from high school were examined a job interests test. Researcher obtained these statistics:
The validity coefficient is 0.6. If John got 54 points on the job interest test, then what his criterion scores (job performance) would be?

47. 2. Estimate Error
Standard Error of Estimate ( )
The error of estimate shows the margin of error to be expected in the individual’s predicted criterion score, as a result of the imperfect validity of the rest.

48. X X

49. Coefficient of Determination,
indicating the proportion of the variance of criterion test scores which is related to the variance of the predictor test scores.

50. 3. Establish the approximate interval for an actual criterion Y

51. Validity Coefficient and Classification DecisionXc X
Figure 6.4 Scatter Plots of the Predictor and Criterion Scores

52. Basic Concepts Cut-off Scores Valid Acceptance Valid Rejection
False Acceptence
False Rejection

53. Four rates
Base Rate
the proportion of successful applicants selected without the use of a test.
Selection Ratio
the proportion of applicants who must be accepted
Hit Rate
the percentage of predictions that are correct.
Success Ratio
the proportion of selected applicants who succeed

54. Table 6.3 Taylor-Russell Table foe a Base Rate of .60