2. the extent to which an experiment, test or any measuring procedure shows the same result on repeated trials.

4. THURSDAY
FRIDAY

5. TEST A TEST B Student Score obtained Score which
would have been obtained on the following day
Bill
Mary Ann
Harry
Don Colin
Sue
Kate
Sam 68 46 19 89 43 56 27 76 82 28 34 67 63 59 35 23 62
Student
Score obtained
Score which
would have been obtained on the following day
Bill
Mary Ann
Harry
Don Colin
Sue
Kate
Sam 65 48 23 85 44 56 38 19 67 69 52 21 90 39 59 35 16 62

6. Scores on an interview using a five-point scale
Student
Score obtained
Score which would have been obtained on the following day
Bill
Mary
Ann
Harry
Don
Colin
Sue
Kate
Sam 5 4 2 3 1

8. Types of Reliability Student- (or Person-) related reliability
Rater- (or Scorer-) related reliability
Intra-rater reliability
Inter-rater reliability
Test administration reliability
Test (or instrument-related) reliability

9. 1-Student-Related Reliability
The source of the error score comes from the test takers.
Temporary illness
Fatigue
Anxiety
Other physical or psychological factors
Test-wiseness (i.e., strategies for efficient test taking)

10. 2-Rater (or Scorer) Reliability
Fluctuations: including human error, subjectivity, and bias
Principles:
Use experienced trained raters.
Use more than one rater.
Raters should carry out their assessments independently.
Two kinds of rater reliability:
Intra-rater reliability
Inter-rater reliability

11. A-Intra-Rater Reliability
Unclear scoring criteria
Fatigue
Bias toward particular good and bad students
Simple carelessness

12. B-Inter-Rater Reliability
Fluctuations including:
Lack of attention to scoring criteria
Inexperience
Inattention
Preconceived biases

13. 3-Test Administration Reliability
Street noise
Listening comprehension test
Photocopying variations
Lighting
Variations in temperature
Condition of desks and chairs
Monitors

14. 4-Test Reliability Measurement errors come from the test itself:
Test is too long
Test with a time limit
Test format allows for guessing
Ambiguous test items
Test with more than one correct answer

26. THE RELIABILITY COEFFICIENT

28. Reliability Coefficient (r)
To quantify the reliability of a test  allow us to compare the reliability of different tests.
0 ≤ r ≤ 1 (ideal r= 1, which means the test gives precisely the same results for particular testees regardless of when it happened to be administered).
If r = 1: 100% reliable
A good achievement test: r>= .90
R<.70  shouldn’t use the test

30. How to make a test MORE reliable?
Question: How many components do we have to test reliability?
Answer: 2
Question: What are they?
Answer: 1- Performance of candidates from occasion to occasion
2- the reliability of scoring

31. How to make a test MORE reliable?
To achieve consistent performances from candidates
Set up a score reliability

32. How can performance of candidates be made reliable? -7
1-Take enough samples of behavior 2- Do not allow candidates too much freedom 3- Write unambiguous items 4- Provide clear and explicit instructions

33. How can performance of candidates be made reliable? -7
5- Ensure that test are well laid out and perfectly legible 6-Candidates should be familiar with format and testing techniques 7- Provide uniform and non-distracting conditions of administration

34. How can performance of candidates be made reliable?
1- Take enough samples of behavior ???
*Other things being equal, the more items that one has on a test, the more reliable the test will be.
THE MORE THE MERRIER = DO NOT CAUSE boredom with LONG lists of questions DECREADING the RELIABILITY LEVEL
*how many extra items are similar to the ones already in the test will be needed to increase the reliability coefficient (factor) to a required level
E.g.: Visual materials are very useful to learn something.
E.g.: I love to see teachers use power- points, posters and visual materials.
Every additional item represent a fresh start for the candidate

35. How can performance of candidates be made reliable?
2-Do not allow candidates too much freedom *to offer candidates to choose the questions – no choice *to offer candidates to choose how to answer – restricted in terms of possible answers

36. How can performance of candidates be made reliable?
2-Do not allow candidates too much freedom
Example: Which one is more reliable?
Write a composition on tourism.
Write a composition on tourism in this country.
Write a composition on how we might develop the tourists industry in this country.
Discuss the following measures intended to increase the number of foreign tourists coming to this country:
1-Better advertising and more information (where &what form)
2-Improve facilities (hotels, transportation, communication…)
3-Training of responsible people(guides, hotel managers...)

37. How can performance of candidates be made reliable?
3- Write an unambiguous items * the meaning of items should be clear * acceptable answer should also be clear both for the test writer and test taker * No place for a test taker to make any interpretations for the question in different ways on different occasions

38. How can performance of candidates be made reliable?
3- Write an unambiguous items
How can we prepare such clear items
Write a draft for items
Give them to a peer / colleague to check it
Pilot it among non-participant students
If no chance for piloting, scorers will take the responsibility.

39. How can performance of candidates be made reliable?
4- Provide clear and explicit instructions
*Peer / colleague correction and criticism
TO AVOID complaining about students’ being unintelligent, stupid ;)
5- Ensure that the tests are well laid out (edited) and perfectly legible (easy to read) NO badly typed items or no legible handwriting NO too much text in too small a space NO poorly re-produced copies WHY??? LOWER THE RELIABILITY

40. How can performance of candidates be made reliable?
6- Candidates should be familiar with format and testing techniques *Ask familiar questions and format and structures that they have been familiar with 7- Provide uniform and non-distracting conditions of administration *administrating the difference in uniformity, timing (the same strictness)- acoustic conditions – a quiet setting with no distracting sounds or movements

41. How can score reliability be achieved?
1- Use items that permit scoring which is as objective as possible 2- Make comparison between candidates as direct as possible 3-Provide a detailed scoring key 4-Train scorer

42. How can score reliability be achieved?
5-Agree acceptable responses and appropriate scores at outset of scoring 6- Identify candidates by number not name 7- Employ multiple, independent scoring

43. How can score reliability be achieved?
1- Use items that permit scoring which is as objective as possible
*How can objectiveness be accomplished?
-Multiple choice (not easy to prepare)
Open-ended questions with has a unique, possibly one-word correct response candidates produce (spelling may be problematic)
Example: In a listening test: Q: what was different about the results?
A)_________________________________________.
B)_______was more closely associated with_______.

44. How can score reliability be achieved?
2- Make comparisons between candidates as direct as possible *How can this be done? *Asking the same questions and expecting the same answers *Limit the students’ way of answering and choosing the quesitons NO MUCH FREEDOM ;)

45. How can score reliability be achieved?
3- Provide a detailed scoring key -A clear explanations for possible acceptable answers -A clear scoring for the correct answers -Prepare as much detailed answer key as possible: partially or totally accepted and how grading will be done for them half or full point?

46. How can score reliability be achieved?
4- Train scorers *If the scoring will be subjective , take writing courses, SCORERS should be familiar with the material and questions because it needs special training. *Double marking should be done.

47. How can score reliability be achieved?
5- Agree acceptable responses and appropriate scores at outset of scoring *Every scorer should all agree on the scores for the test or else scoring can’t start *open to new possible answers and be flexible *Remind the possible answers and make everybody agree on the scoring *If so, all scorer should be given knowledge about it and those should be added to the answer key

48. How can score reliability be achieved?
6- Identify candidates by number not name Knowing students’ name, gender, nationality, physical appearance affect scoring process Blind reading = to be objective
7- Employ multiple / independent scoring
Scoring should be done by two different independent scorers =Double markers
And the difference should be observed and a final decision is made by a third person

49. Gürşen SARIALTIN & Tuba DEMİR
THANK YOU
Gürşen SARIALTIN & Tuba DEMİR