1. Reliability and Validity what is measured and how well

2. Reliability Consistency Consistency – Does the test agree with itself? Stability Stability – Does the test agree with itself over time? Agreement Agreement – Do different raters agree with each other?

3. Consistency Reliability Equivalent forms reliability Equivalent forms reliability – Correlation between the scores on two parallel forms of a test Internal consistency reliability Internal consistency reliability – Correlation between half sections of the test (Split Half), or between all of the items (Internal Consistency)

4. Stability Reliability Test – Retest Reliability Test – Retest Reliability The same test is given at two different administrations to the same group of respondents. The same test is given at two different administrations to the same group of respondents. Correlation between time 1 and time 2. Correlation between time 1 and time 2.

5. Agreement Reliability Inter-Rater Reliability Inter-Rater Reliability – Correlation between raters – Correlation between rater and expert – % agreement between raters – % agreement between rater and expert – Chance corrected methods (Kappa) – Variance partitioning methods

6. The Radio Signal Analogy Signal to noise ratio Signal to noise ratio Total Signal Received = True signal + Noise Total Signal Received = True signal + Noise Signal / (Signal + Noise) Signal / (Signal + Noise)

7. A Little Math About Reliability X = T + E X = T + E Observed Score = True Score + Error Observed Score = True Score + Error σ 2 X = σ 2 T + σ 2 E σ 2 X = σ 2 T + σ 2 E The spread of Observed scores = The spread is True scores + The spread in Error scores. The spread of Observed scores = The spread is True scores + The spread in Error scores.

8. A Little Math About Reliability r xx’ = σ 2 T / σ 2 X r xx’ = σ 2 T / σ 2 X r xx’ = 1 – (σ 2 E / σ 2 X ) r xx’ = 1 – (σ 2 E / σ 2 X )

9. Reliability and PRI Scores

10. Validity Validity is the degree to which a test measures what it is intended to measure. Validity is the degree to which a test measures what it is intended to measure. Validity is the meaningfulness, appropriateness, and usefulness of the inferences made from the information a test provides. Validity is the meaningfulness, appropriateness, and usefulness of the inferences made from the information a test provides.

11. Validity “Truth” and “Use” “Truth” and “Use” What is the test really measuring? What is the test really measuring? For whom is the test appropriate? For whom is the test appropriate? How should the information the test provides be used? How should the information the test provides be used?

12. Constructs Assumptions we make when we use a test: Assumptions we make when we use a test: The subject possesses some true amount of the latent theoretical construct that the test is designed to measure. The subject possesses some true amount of the latent theoretical construct that the test is designed to measure. Depression, Coping, Math Aptitude, etc. Depression, Coping, Math Aptitude, etc.

13. Constructs The amount of the construct the subject possesses is not directly measurable. The amount of the construct the subject possesses is not directly measurable. Observable behaviors can represent the latent construct (ability, trait, etc.) and can be measured. Observable behaviors can represent the latent construct (ability, trait, etc.) and can be measured. The goal is to measure as many of these observable behaviors as we can and to measure them accurately. The goal is to measure as many of these observable behaviors as we can and to measure them accurately.

14. Types of Validity Content Validity Content Validity Does the test cover all of the intended content? Does the test cover all of the intended content? Measured by expert opinion. Measured by expert opinion.

15. Types of Validity Concurrent Validity Concurrent Validity Does the test agree with other existing measures of the same construct? Does the test agree with other existing measures of the same construct? Correlations between the test scores and scores from other measures. Correlations between the test scores and scores from other measures.

16. Types of Validity Types of Concurrent Validity Evidence Types of Concurrent Validity Evidence Convergent Validity Convergent Validity Discriminant Validity Discriminant Validity

17. Types of Validity Known Groups Validity Known Groups Validity Does the test distinguish between groups of subjects with known differences on the construct or related constructs? Does the test distinguish between groups of subjects with known differences on the construct or related constructs?

18. Known Groups Validity and the PRI

21. Consequential Validity Is the test information useful for decision making? Is the test information useful for decision making? Does it have any unintended consequences? Does it have any unintended consequences? Can the information be misused? Can the information be misused?

22. Predictive Validity Can the test be used to predict future behavior? Can the test be used to predict future behavior? Like Concurrent Validity (both are Criterion Validity), but some time passes between the test and the criterion. Like Concurrent Validity (both are Criterion Validity), but some time passes between the test and the criterion. SAT and GPA. SAT and GPA.

23. Construct Validity All validity is really construct validity. All validity is really construct validity. Does it measure what it is intended to measure? Does it measure what it is intended to measure? Does the test agree with the theory in the field? Does the test agree with the theory in the field? Does it reveal the true amount of the construct that a subject possesses? Does it reveal the true amount of the construct that a subject possesses?

24. Other Related Issues Tests should have Face Validity. Tests should have Face Validity. Does the subject believe the test is measuring the intended construct? Does the subject believe the test is measuring the intended construct? Some tests do not directly reveal what is being measured. Some tests do not directly reveal what is being measured.

25. Other Related Issues Reliability and validity are properties of the information that a test provides, NOT of the test itself. Reliability and validity are properties of the information that a test provides, NOT of the test itself. The farther away you get from the original purpose for which a test was developed and validated, the weaker the inferences that can be made. The farther away you get from the original purpose for which a test was developed and validated, the weaker the inferences that can be made.

26. Other Related Issues No single indicator is sufficient for decision making. A battery of indicators, or sources of information, is always better. No single indicator is sufficient for decision making. A battery of indicators, or sources of information, is always better. Reliability is a necessary condition for the correct use of a test, but not a sufficient one. Reliability is a necessary condition for the correct use of a test, but not a sufficient one.

27. Other Related Issues Validity is the most important property of the information a test provides. Validity is the most important property of the information a test provides. Consistent information. Consistent information. Truthful information. Truthful information. Useful information. Useful information.

28. The Credibility of a Witness

29. The Usefulness of a Car

30. Finding Lost Keys on a Dark Street