Reliability

Basics of test score theory Each person has a true score that would be obtained if there were no errors in measurement. However, measuring instruments are imperfect, the score observed for each person almost always differs from the person’s true ability or characteristic.

We never no the true score, only estimate it. If E is very small or zero, you can that the measurement is very reliable. If E is very big, then we can not trust the observed score is close to the true score – not reliable….

What is reliability? The extent to which a score or measure is free of measurement error. Theoretically, reliability is the ratio of true score variance to observed score variance. The ratio can be estimated using a variety of correlation methods, including coefficient alpha, spilt-half, test-retest and parallel forms.

Observed scores and reliabilities. Which one is the most reliable? A B C T (true score, 진점수 )E (error, 오차 ) X (observed score, 관찰점수 ) T (true score, 진점수 )E (error, 오차 ) X (observed score, 관찰점수 ) T (true score, 진점수 )E (error, 오차 ) X (observed score, 관찰점수 )

If you repeat testing the same person over and over, you may get close to the true score.

Which one uses the most reliable measurement?

Can you estimate their true scores? 송이민준휘경 아동학 적성 검사 (12 회 반복 ) 3, 4,4,4, 5,5,5,5,6,6,7 9 1,4,5,5,6,6,7,7,7,7,8,9 2,5,5,5,6,6,6,6,6, 7,7,9

Source of Error and Reliability There are many sources of errors that make observed scores deviated from your true scores!!! You may spoil your exam because of loud noises or hot temperatures in the room. You may spoil your exam because the test items are not representative of what you have studied. We can fix it (but still not perfect!!)

Test-retest method Source of error- time sampling Same test given at two points in time Correlation between scores obtained on the two occasions.

Which test is more reliable? Test ATest B Test 1Test 2Test1Test2 민준 7788 휘경 7678 송이 4756 수정 5479 인욱 4323 철수 7756 영수 5957 지영 6856 옥경 8754 효진 9789

Test A r=.33Test B r=.90

Alternate forms or parallel forms Source of error-item sampling errors, selected items are not representative of what to be measured. Two parallel forms of tests are administered to the same group of people on the same day. Correlation between equivalent forms of the test that have different items. A large pool of items, two parallel forms

Results of the two parallel types of tests Type AType B 민준 76 휘경 66 송이 76 수정 44 인욱 34 철수 75 영수 97 지영 87 옥경 77 A pool of items Type A Type B

r=.879

Split-half method Source of error-internal consistency. Some items in the test are measuring something else. Consistency of items in within the same test. Corrected correlation between two halves of the test.

Split-half method 민준 76 휘경 65 송이 75 수정 43 인욱 34 철수 75 영수 98 지영 87 옥경 77 A 20 item test

r=.869

Coefficient alpha Source of error-internal consistency. Some items in the test are measuring something else. Corrected correlation between items and total scores. The most popular method.

Which item is measuring something else? A nine item test and four people took the test….

SPSS Coefficient Alpha results

How reliable is reliable? In basic psychological research it has been suggested that reliability estimates in the range of.70 and.80 are good enough for most purposes Reliabilities greater than.95 are not very useful because all of the items are testing essentially the same thing and that the measure could easily be shortened. In clinical settings, high reliability is extremely important. When tests are used to make important decisions about someone’s future, evaluators must be certain to minimize any error in classification. Thus, a test with a reliability of.90 might not be good enough.

Questions Do we ever know the true score in psychological testing? Explain the reliability using the terms, true score, observed score, and errors. Can you explain the source of errors that make test scores unreliable and the possible ways to fix them? You have a serious medical problem and you have to make a decisions based on a test result. How reliable do you want the test to be?