Evaluation of measuring tools: reliability

After we have evaluated the quality of test items and eliminated those that are not considered adequate, we must evaluate the overall quality of the test. In this chapter we discuss the problem of the reliability and accuracy of the measure, trying to find an answer to the question “to what extent the scores obtained by subjects in the test are affected by measurement errors and how much”.

The problem of measurement error

Measurement error is the difference between the empirical score obtained by a subject on a test and his/her true score. Objective: Elaborate tests that lead to the minimum possible measurement error. That the obtained score gives the greatest degree of real information on the characteristics under study. There are other errors, random ones (which ones are studied trough analysis of reliability).

Types of measurement errors

Measurement error: the difference between the empirical score of a subject and their true score. We obtain an individual measure of the accuracy of the test. The standard error of measurement: standard deviation of measurement errors. It’s a measurement of the group because it is calculated for all subjects of the sample. Estimation error of the true score: the difference between the true score of the subject and the true score predicted by the regression model. The standard error of estimation of the true score: standard deviation of estimation errors.

Substitution error: the difference between the score obtained by a subject in a test and that one obtained in another parallel test. It would be committed to replacing the test scores on the X1 by those from a parallel test X2. The standard error of substitution: standard deviation of substitution errors. Prediction error: the difference between the scores obtained by a subject in a test (X1) and predicted scores in the same test (X1') from a parallel test X2. The standard error of prediction = standard deviation of prediction errors.

The linear model of Spearman

X (empirical score)= V (true level)+ E (measurement error) He's going to help us estimate the amount of error that are affecting to the empirical scores and the true level of subjects in the characteristic of study. X (empirical score)= V (true level)+ E (measurement error)

A) E = X – V B) E (e) = 0 C) D) Cov (V, E) = 0 E) F) Cov (X, V) = G) H)

Interpretation of the reliability coefficient

The correlation between the empirical scores obtained by a sample of subjects in two parallel forms of the test. The ratio between the variance of true scores and the variance of empirical scores. As this ratio increases, the measurement error decreases. Reliability index:

Factors that affect reliability

TEST LENGTH If we increase the length of the test (if we add parallel items): More information about the attribute under study. Lower error when estimating the true score of a subject. So, reliability will increase.

SAMPLING VARIABILITY The reliability coefficient can vary depending on the homogeneity of the group. The lower the reliability coefficient the more homogeneous the group. * We assume that the standard error of measurement of a test remains constant independently of the variability of the group in which it is applied.

Reliability as equivalence and stability of measures Coefficient of reliability or equivalence

A test must meet two requirements : It should measure the characteristic that really needs to be measured (be valid). Empirical scores obtained by applying the test should be: Accurate (free of error), and Stable (when we evaluate a trait or characteristic with the same test at different times and under conditions as similar as possible, if the studied trait has not changed, you must obtain similar results: reliability of the test).

a) Parallel forms method 1. Elaborate two parallel forms of one test X and X’. 2. Apply the two tests on a sample of subjects representative of the population targeted by the test. 3. Calculate Pearson’s correlation. X1 and X2:scores obtained by subjects in each form of the test. If applications are made at the same time there is greater control over the conditions of application. Difficulty to elaborate two parallel forms.

b) Test-retest method 1. Apply the same test on two separate occasions to the same sample of subjects. 2. Calculate the correlation X1 and X2: scores obtained by subjects in each of the test applications. It does not require different forms of the same test. Possible influence of memory, the time interval between one application and another, and the attitude of the subject.

Reliability as internal consistency

Methods to estimate the reliability of a test that only require one application: A) Based on the division of the test in two parts: Spearman-Brown Rulon Guttman-Flanagan B) Based on the covariation of items: Cronbach's alpha coefficient

a) Methods Based on the division of the test in two parts The estimation of reliability is not affected by the factors discussed. Save time and effort. 1. Apply the test to a sample of subjects. 2. Once obtained the scores, divide the test in two parts, calculate the correlation between the scores obtained by subjects in both parts and apply a correction formula. The parts should be similar in difficulty and content.

Spearman-Brown The two parts must be parallel, so we should check the assumptions of parallelism (true scores of the subjects are the same in both tests, the variance of measurement errors is the same in both tests).

Spearman-Brown. Example We have applied a numerical aptitude test of 20 items to a sample of 6 subjects. The table results are the scores obtained on even items (X1) and odd ones (X2). Calculate the reliability coefficient assuming that the two parts of the test are parallel. Subjects X1 X2 X12 X22 X1X2 1 8 4 64 16 32 2 7 49 3 6 36 48 5 25 20 56 Total 42 34 302 202 241

Rulon and Guttman-Flanagan They are applied when, despite not being strictly parallel parts, we can consider tau-equivalent (test in which the true scores of subjects of the sample are the same in both forms but the error variances are not necessarily equal) or essentially tau-equivalent (test in which the true score for each subject in one of the test is equal to the other plus a constant).

Rulon y Guttman-Flanagan

b) Method based on the covariation of items It requires the analysis of variance and covariance of the subjects' responses to the items. It is an estimation of the internal consistency of test’s items. Cronbach's alpha coefficient. It is based on the mean correlation among all test’s items.

Cronbach's alpha. Example We have applied a test of visual perception to 6 subjects. The results of the table show the scores of subjects in each of the five test items. Calculate the value of the coefficient of reliability of the test. Subjects 1 2 3 4 5 A B C D E F

Estimation of the true score of the subjects in the attribute of interest

Estimations about the value of a subject's true score on a test and the error that affects the empirical scores obtained in the same test. We can’t calculate the exact value of the subject’s true score, but we can establish a confidence interval within which we will find the score with a given confidence level.

a) Method of Chebychev’s inequality It is applied if there is not any assumption about the empirical scores distribution or the errors distribution. The true score will be between two values​​, the upper limit and the lower limit.

Example We have administered a numerical reasoning test to 200 subjects. We have obtained: Mean=52, SX=7, rXX=0.73. Estimate the true score of a subject who obtained an empirical score of 65 points on the test. Confidence level of 95%. Too wide interval which involves a vague estimation. It may be due to a low reliability coefficient or that this method does not consider the type of distribution of empirical scores.

b) Estimation based on the normal distribution of errors It assumes a normal distribution both of measurement errors and empirical scores. 1. Calculate Zs in the normal distribution table for the desired confidence level. 2. Calculate the standard error of measurement. 3. Calculate the maximum measurement error that we are willing to admit. 4. Confidence interval in which we will find the true score.

Example The same data than previously. The confidence interval has been reduced significantly.

c) Estimation based on the regression method It is more convenient to make the confidence interval not from empirical scores (which are biased due to measurement errors), but from the estimated true score. 1. Make the regression equation of V on X. To determine the regression equation is useful for: Describe concisely the relationship between variables. Predict the values ​​of a variable depending on the other.

2. Calculate Zs in the normal distribution table for a given level of confidence. 3. Calculate the standard error of estimation Svx. 4. Calculate the maximum error of estimation. 5. Calculate the confidence interval in which we will find the true score.

Example Same data than previouly. Estimate the true score (in raw, differential and typical scores) of a subject who scored 65 points.