1. Reliability Lesson Six
What is reliability? When we talk about the reiliable, we talk about the result or scores. The test result is consistent . I give you the quiz today to this class, and I give you the quiz again with the same group. I will give the similar result. You rank no. 1 and you will rank no. 1 another time. With the test. We are talking about the test itself. We did not concern the test content.

2. Case
Imagine that a hundred students take a 100-item test at three o’clock one Thursday afternoon. The test is not impossible difficult or ridiculously easy for these students, so they do not all get zero or a perfect score of 100. Now what if in fact they had not taken the test on the Thursday but had taken it at three o’clock the previous afternoon? Would we expect each student to have got exactly the same score on the Wednesday as they actually did on the Thursday?
The answer to this question must be no. Even if we assume that the test is excellent, that the conditions of administration are almost identical, that the scoring calls for no judgment on the part of the scorers and is carried out with perfect care, and that no learning or forgetting has taken place during the one-day interval.
Human beings are not like that ; they simply do not behave in exactly the same way on every occasion, even when the circumstances seem identical. What we have to do is construct, administer and score tests in such a way that the scores actually obtained on a test on a particular occasion are likely to be very similar to those which would have been obtained if it had been administered to the same students with the same ability, but at a different time. The more similar the scores would have been, the more reliable the test is said to be.

3. Contents Definition of reliability
Factors contributing to unreliability
Types of reliability
Indication of reliability: Reliability coefficient
Ways of obtaining reliability coefficient:
Alternate/Parallel forms
Test-retest
Split-half & KR-21/KR-20
Two ways of testing reliability
How to make test more reliable
Online video

4. Definition of Reliability (1)
“The consistency of measures across different times, test forms, raters, and other characteristics of the measurement context” (Bachman, 1990, p. 24).
If you give the same test to the same testees on two different occasions, the test should yield similar results.
In other words, the different times the result should be the same. Test forms are equivalent/ parallel form. You might get the similar result. Test gives to different test taker and get the similar score. It concern the rater, how the test is given, administration will affect the reliability.

5. Definition of Reliability (2)
A reliable test is consistent and dependable.
Scores are consistent and reproducible.
The accuracy or precision with which a test measures something; that is, consistency, dependability, or stability of test results.
We are talking about the test results. Again, we did not concern about the content.

6. Factors Contributing to Unreliability
X=T+ E (observed score = true score + error score)
Concerned with freedom from nonsystematic fluctuation.
Fluctuations in
the student
scoring
test administration
the test itself
What kinds of factor will influence the test reliability. The test takers- student; or scoring means test raters. How the test is given (test administration), the problem with test (ambiguous item). In reality, we have acknowledge the testing exists errors. X is the test results or scores. These scores consist of two parts. One is his true ability, and the error score. For, example, multiple choice. 35% of get right. If your eyes closed, you pick up a as the correct answer. You guess right. It is not your own ability, it is called error score. In that case, we have to know in the testing situation, there is nonsystematic flunctuation. The really error from the test taker themselves. You expect you get the right answers. However, you did not perform well (careless, it is not my day, stay up late), in your normal

7. Types of Reliability Student- (or Person-) related reliability
Rater- (or Scorer-) related reliability
Intra-rater reliability
Inter-rater reliability
Test administration reliability
Test (or instrument-related) reliability
To do with test taker itself
Or some people call scorer. Intra-rater: we are talking about the one rater. the same people; inter-raters – more than 2 raters.
The way you give the test; the classroom situation, times. (too early before the ring.
To do with the test itself.

8. Student-Related Reliability (1)
The source of the error score comes from the test takers.
Temporary illness
Fatigue
Anxiety
Other physical or psychological factors
Test-wiseness (i.e., strategies for efficient test taking)
We are going to talk each of them one by one.
The students refer to test takers. Fatigue (too tired ; stay up late). Anxiety (too nervous; oral interview. You did not say the word.)
Test-wiseness. I give you some example. For example.. Test taking strategies. How to distribute your time? You have to finish your test in one hour. You might use your time evenly to finish four times.
Multiple choice. How to get the correct answers? “four choice” eliminate the unlikely answers. Those strategies call test-wiseness. To stick to the passage, don’t think too much. Otherwise, it will lead to the different direction. It is the test-role.

9. Student-Related Reliability (2)
Principles:
Assess on several occasions
Assess when person is prepared and best able to perform well
Ensure that person understands what is expected (e.g., instructions are clear)
Some principles to follow; teacher cannot control students’ behavior. Give more tests and get average. Give more chances. Those functions will given out. Only final exams is mainly score for the whole grades.
It is caud the control to fit into this
3, make sure that student understand what is expected. “oral instructions.

10. Rater (or Scorer) Reliability (1)
Fluctuations: including human error, subjectivity, and bias
Principles:
Use experienced trained raters.
Use more than one rater.
Raters should carry out their assessments independently.
Errors – human error as the rater. Carelessness and give the wrong grading. The composition and oral test is hard to be objectives.
Iraters need to be independent not be influced by other raters.

11. Rater Reliability (2) Two kinds of rater reliability:
Intra-rater reliability
Inter-rater reliability
One rater have to use the

12. Intra-Rater Reliability
Fluctuations including:
Unclear scoring criteria
Fatigue
Bias toward particular good and bad students
Simple carelessness
One rater has clear scoring criteria. No one is totally objective. We are human beings, and making mistakes.

13. Inter-Rater Reliability (1)
Fluctuations including:
Lack of attention to scoring criteria
Inexperience
Inattention
Preconceived biases
You did not attention to scoring. Two or more raters involved in this situation, we need to calculate the inter-rater reliability.

14. Inter-Rater Reliability (2)
Used with subjective tests when two or more independent raters are involved in scoring
Train the raters before scoring (e.g., TWE, dept. oral and composition tests for recommended students).

15. Inter-Rater Reliability (3)
Compare the scores of the same testee given by different raters.
If r= high, there’s inter-rater reliability.
The evidence will be called. Statistical formula is to help you get the r. if we want to know if there is inter-rater reliability you need to do the calculation and get r. try to convince the people.
They did not train the raters and did not have inter-rater reliability.

16. Test Administration Reliability
Street noise
Listening comprehension test
Photocopying variations
Lighting
Variations in temperature
Condition of desks and chairs
Monitors
When I give the test, some factors might affect your performance. (noisy street.)
2. The lousy copying machine, you cannot read clear and take more time to
3. Light broken, too hot/ sweating. 4.

17. Test Reliability Measurement errors come from the test itself:
Test is too long
Test with a time limit
Test format allows for guessing
Ambiguous test items
Test with more than one correct answer
The long test will be more reliable than the short term. It includes more samples of test taker’s ability. One question, and 100 questions. If you miss one, you still have 90% to get right. But it is too long, 200 items, you will be tired. There is more chances that I will make the mistakes in the test.
Test with a time limit will have cause some problems. Take 100 items, 70% answers are known, 30% is guessing. He got 25% correct. He got 95. It is likely his test-wiseness good. Multiple-choice is good for inter-rater reliability but increases the guessing factors.

18. Ways of Enhancing Reliability
General strategies:
Consider possible sources of unreliability
Reduce or average out nonsystematic fluctuations in
raters
persons
test administration
instruments

19. How to Make Tests More Reliable? (1)
Take enough samples of behavior
Try to avoid ambiguous items
Provide clear and explicit instructions
Ensure tests are well layout & perfectly legible
Provide uniform and undistracted condition of administration
Try to use objective tests

20. How to Make Tests More Reliable? (2)
Try to use direct tests
Have independent, trained raters
Provide a detailed scoring key
Try to identify the test takers by number, not by names
Try to have more multiple independent scoring in subjective tests
(Hughes, 1989, pp ).

21. Reliability Coefficient (r)
To quantify the reliability of a test  allow us to compare the reliability of different tests.
0 ≤ r ≤ 1 (ideal r= 1, which means the test gives precisely the same results for particular testees regardless of when it happened to be administered).
If r = 1: 100% reliable
A good achievement test: r>= .90
R<.70  shouldn’t use the test
Let’s talk about the r. usually, the ideal r between 1 to 0 . When we have r = o , what does that mean? The result is not reliable .You don’t use the test results in ay ways, something is terrible wrong in the test. In other words, the r = is bigger is better. is more reliable. The perfect r is 1. the average .7 to 1.
When you have r < .7, you should not use the test. It refers to test items itself own. We did not take into account of test taker and raters.

22. How to Get Reliability Coefficient
Type of Reliability
How to Measure
Stability or Test-Retest
Give the same assessment twice, separated by days, weeks, or months. Reliability is stated as the correlation between scores at Time 1 and Time 2.
Alternate Form
Create two forms of the same test (vary the items slightly).  Reliability is stated as correlation between scores of Test 1 and Test 2.
Internal  Consistency
(Alpha, a)
Compare one half of the test to the other half.  Or, use methods such as Kuder-Richardson Formula 20 (KR20) or Cronbach's Alpha.   
We are talking about the theory. If we give the test to same people several times. The results will be consist.
Two parallel forms to the same test takers on different occasion. This time, next time. If the tests have good reliable, it means the results are similar.
One form – you use the exact same test. Pre-test and post test.
One administration. One time. And divided into two part. Or two sets of scores. Individual test takers have two set of scores. And we can correlate those two score. We called it internal consistency. There are three types.

23. How to Get Reliability Coefficient
Two forms, two administrations: alternate/parallel forms
One form, two administrations: test-retest
One form, one administration (internal consistency):
split-half (Spearman-Brown procedure)
KR-21
KR-20
We are talking about the theory. If we give the test to same people several times. The results will be consist.
Two parallel forms to the same test takers on different occasion. This time, next time. If the tests have good reliable, it means the results are similar.
One form – you use the exact same test. Pre-test and post test.
One administration. One time. And divided into two part. Or two sets of scores. Individual test takers have two set of scores. And we can correlate those two score. We called it internal consistency. There are three types.

24. Alternate/Parallel Forms
Two forms, two administrations:
Equivalent forms (i.e., different items testing the same topic) taken by the same test taker on different days
If r is high, this test is said to have good reliability.
the most stringent form
Two forms, or called parallel form.. Are measuring the same ability. If you give two test and give to test takers at the same time same day, the result is correlated. How do you like this method as the test takers? It is boring to take the test twice. It is very ideal. If you are not highly motive, you will perform differently.
The teacher is not easy to prepare two parallel forms. It takes time and limit time. T likes to take test-retest.

25. Test-Retest One form, two administrations
The same test is administered to the same testees with a short time lag, and then calculate r.
Appropriate for highly speeded test
Learner effect. The first one, you did not answer items correct. The second one, after four month, you may answer items correct. However, it cannot be a too short. Tests takers might memorized the item. One or two weeks are better time lag. The problem is test takers. They have to take the test twice. Therefore, we have the third method. Called split-half.

26. Split-half (Spearman-Brown Procedure)
One test, one administration
Split the test into halves (i.e., odd questions vs even questions) to form two sets of scores.
Also called internal consistency Q1 Q2 Q3 Q4 Q5 Q6
Was designed by spearman and Brown.
First Half
Second Half

27. Split-half (2) Note that the r isn’t the reliability of the test
A math relationship between test length and reliability: the longer the test, the more reliable it is.
Rel.total = nr/1+ (n-1)r  Spearman & Brown Prophecy Formula
E.g., correlation between 2 parts of test; r= .6  rel. of full test = .75
If lengthen the test items into 3 times: r= .82
The r is not the real reliability of the test. You know why. Because the test is cut into half. The real test should be twice long. 100 items cut into half. 50 items. The reliably become the half of the reliablgigiy of the all test. We need some adjustment. The real = number times r / 1
150 items , we will increase the reliability of the test. The theory is the longer the test is, the more reliable the test is. I did not want you to remember the formula.

28. Kuder-Ridchardson formula 21
KR-21 = k/(k-1){1-[x (1- x/k)]/s2}
k= number of items; x= mean
s= standard deviation (formula see Bailey 100)
description of the spread outness in a set of scores (or score deviations from the mean)
o<=s  the larger s, the more spread out
E.g., 2 sets of scores: (5, 4,3) and (7,4,1); which group in general behaves more similarly?
The important concept I am going to talk about is the standard deviation. We use S to represent it. For example, you have 20 people in the group. Get 20 scores. Add up all those score up. Use the mean as the center, your score can distance from the mean. Deviate from the mean.
How can we get sd.:
3 test takers. Average of the mean is four, another group take the same test. The mean is 4 too. You cannot say they have a similar general behaves. Sd. In the second group is bigger than the first one. Because the SDs are different , they don’t have same behaves.
50 scores. The first group is scattered in the middle crowded. Not spread out like the second group.
p the first one SD 1; the second SD 3. we have small no. it is easy to tell. However, you have the large no. you need calculate them.

29. Kuder-Ridchardson formula 20
KR-20= [k/(k-1)][1-(∑pq/s2)
p= item difficulty (percent of people who got an item right)
q= 1-p (i.e., percent of people who got an item wrong)
It is another way to calculate the reliability. The first formula is conservative. You might higher. We can come out the ways of reliability.

30. Ways of Testing Reliability
Examine the amount of variation
Standard Error of Measurement (SEM)
The smaller the better
Calculate “reliability coefficient”
“r”
The bigger the better
What the r you are expected? The bigger is better. It is the one way. Another way is we examine the amought of variation can calculate the SEM. Is smaller is the better.
SEM has to do with individual