Validity and Reliability Margaret Wu

Validity According APA Standards for educational and psychological testing: The degree to which evidence and theory support the interpretations of test scores entailed by proposed uses of tests. To establish validity: We need to accumulate evidence for appropriate interpretations of scores. That is, to evaluate the use of test scores, and not the test itself. A test may be valid for one kind of use, but not valid for another kind of use.

Sources of Validity Evidence Evidence based on Test Content Evidence based on Response Process Evidence based on Internal Structure Evidence based on Relations to Other Variables Evidence based on Consequences of Testing

Evidence based on Test Content Content coverage – representation of the domain Check assessment framework and test blueprints Assessment framework has definition of the test domain Test Blueprint has the coverage of subdomains. Check against intended use of the test: Subdomain scores reported? Inferences on individual student performance? Inferences on group performance? Inferences on trends over time? Test blueprint and curriculum match Test blueprint and test items match

Evidence based on Response Processes Response processes of test takers, For example: Questionnaire: social desirability answers or students’ “real” responses Items testing reasoning: Is reasoning needed, or perhaps memorised algorithm? Evidence can be gathered through: Theoretical evidence: test blueprints include students’ required cognitive demand Empirical evidence: cognitive labs (think-aloud procedure); pre-test; response time Students with disability Judges/raters’ judging processes Test administration procedures: testing time; test security, testing environment

Evidence based on Internal Structure Good/poor discriminating items Expected item difficulty order matches empirical order Item inter-relationship; unidimensionality Test reliability Presence of differential item functioning Size of measurement error; sampling error; in relation to the validity of using the test scores

Evidence based on relations to other variables Predictive validity Concurrent validity Relationship to other tests Relationship to group variables, e.g., gender, demographic Are the relationships consistent with what’s expected given the construct of the test?

Evidence based on consequences of testing Intended and unintended consequences of test use. Effect of group differences in test scores on employment selection Narrowing of school curriculum to exclude learning objectives not assessed. What are the benefits of using the test scores? Are there any detrimental effects from administering the test?

Reliability Reliability refers to how consistent would the test scores be should “similar” tests be administered.

(a) The temperature was 7° . It fell by 4° . The temperature was then David can answer 60% of the items (if we have the opportunity to administer all items) (c) (-11) + (+3) = (a) The temperature was 7° . It fell by 4° . The temperature was then A class starts at 10:30. The class is 40 minutes long. What time does the class finish? Possible Grade 5 Mathematics Item Pool – Many questions can be asked (a) 16 × 10 = A class starts at 10:30. The class is 40 minutes long. What time does the class finish? The correlation between students’ scores on two “parallel” tests is called reliability (j) 139.2 ÷ 1000 = Each apple weighs around 160 grams. How many apples together will weigh close to half a kilogram? 40 questions are sampled from the large item pool David’s test scores on similar NAPLAN tests will have a range of 10 score points (e.g., between 20/40 to 30/40). The difference between David’s test score and his “TRUE” score is called measurement error/. NAPLAN 2008 Test NAPLAN 2009 Test David’s score: 25/40 NAPLAN 2010 Test David’s score: 28/40 David’s score: 20/40

Margin of error in measuring student performance One test collects only a small sample of performance. Possible variation in scores is called Measurement Error. Note that Measurement Error does not refer to mistakes made in the assessment (e.g., wrong scoring; incorrect question) . Is the measurement error too large in NAPLAN? 11

How big an error size is acceptable? The answer is It depends. An example Effectiveness of a weight loss program Expect a loss of 0.5 kg after one week. Measurement scale is accurate to 1kg. Not good enough for measuring individual change OK for a group change, if group size is ‘large’. The key is to assess whether the measurement error is too large in comparison to the magnitude that we want to measure. 12 12

Magnitudes of measurement error As a rough guide, a lower bound for measurement error is given by sqrt(4/N) where N is the number of items in the test.

On the NAPLAN scale… 14

Summary of measuring individuals The main message is that a one-off test does not provide very accurate information at the individual student level other than an indicative level of whether a student is below average, at average or above average. If ever one single test of 30 items is used for high-stakes purposes such as selection into colleges or awarding certificates, we should be very wary of the results.

Reliability and Measurement Error Reliability = Var(T)/Var(X) variance of true score divided by variance of observed score Reliability = (Var(X) – Var(E)) / Var(X) Variance of observed scores minus square of measurement error, then divide by variance of observed scores. Sqrt(Var(E)) is Measurement Error.