the extent to which an experiment, test or any measuring procedure shows the same result on repeated trials.

THURSDAY FRIDAY

TEST A TEST B Student Score obtained Score which would have been obtained on the following day Bill Mary Ann Harry Don Colin Sue Kate Sam 68 46 19 89 43 56 27 76 82 28 34 67 63 59 35 23 62 Student Score obtained Score which would have been obtained on the following day Bill Mary Ann Harry Don Colin Sue Kate Sam 65 48 23 85 44 56 38 19 67 69 52 21 90 39 59 35 16 62

Scores on an interview using a five-point scale Student Score obtained Score which would have been obtained on the following day Bill Mary Ann Harry Don Colin Sue Kate Sam 5 4 2 3 1

Types of Reliability Student- (or Person-) related reliability Rater- (or Scorer-) related reliability Intra-rater reliability Inter-rater reliability Test administration reliability Test (or instrument-related) reliability

1-Student-Related Reliability The source of the error score comes from the test takers. Temporary illness Fatigue Anxiety Other physical or psychological factors Test-wiseness (i.e., strategies for efficient test taking)

2-Rater (or Scorer) Reliability Fluctuations: including human error, subjectivity, and bias Principles: Use experienced trained raters. Use more than one rater. Raters should carry out their assessments independently. Two kinds of rater reliability: Intra-rater reliability Inter-rater reliability

A-Intra-Rater Reliability Unclear scoring criteria Fatigue Bias toward particular good and bad students Simple carelessness

B-Inter-Rater Reliability Fluctuations including: Lack of attention to scoring criteria Inexperience Inattention Preconceived biases

3-Test Administration Reliability Street noise Listening comprehension test Photocopying variations Lighting Variations in temperature Condition of desks and chairs Monitors

4-Test Reliability Measurement errors come from the test itself: Test is too long Test with a time limit Test format allows for guessing Ambiguous test items Test with more than one correct answer

THE RELIABILITY COEFFICIENT

Reliability Coefficient (r) To quantify the reliability of a test  allow us to compare the reliability of different tests. 0 ≤ r ≤ 1 (ideal r= 1, which means the test gives precisely the same results for particular testees regardless of when it happened to be administered). If r = 1: 100% reliable A good achievement test: r>= .90 R<.70  shouldn’t use the test

How to make a test MORE reliable? Question: How many components do we have to test reliability? Answer: 2 Question: What are they? Answer: 1- Performance of candidates from occasion to occasion 2- the reliability of scoring

How to make a test MORE reliable? To achieve consistent performances from candidates Set up a score reliability

How can performance of candidates be made reliable? -7 1-Take enough samples of behavior 2- Do not allow candidates too much freedom 3- Write unambiguous items 4- Provide clear and explicit instructions

How can performance of candidates be made reliable? -7 5- Ensure that test are well laid out and perfectly legible 6-Candidates should be familiar with format and testing techniques 7- Provide uniform and non-distracting conditions of administration

How can performance of candidates be made reliable? 1- Take enough samples of behavior ??? *Other things being equal, the more items that one has on a test, the more reliable the test will be. THE MORE THE MERRIER = DO NOT CAUSE boredom with LONG lists of questions DECREADING the RELIABILITY LEVEL *how many extra items are similar to the ones already in the test will be needed to increase the reliability coefficient (factor) to a required level E.g.: Visual materials are very useful to learn something. E.g.: I love to see teachers use power- points, posters and visual materials. Every additional item represent a fresh start for the candidate

How can performance of candidates be made reliable? 2-Do not allow candidates too much freedom *to offer candidates to choose the questions – no choice *to offer candidates to choose how to answer – restricted in terms of possible answers

How can performance of candidates be made reliable? 2-Do not allow candidates too much freedom Example: Which one is more reliable? Write a composition on tourism. Write a composition on tourism in this country. Write a composition on how we might develop the tourists industry in this country. Discuss the following measures intended to increase the number of foreign tourists coming to this country: 1-Better advertising and more information (where &what form) 2-Improve facilities (hotels, transportation, communication…) 3-Training of responsible people(guides, hotel managers...)

How can performance of candidates be made reliable? 3- Write an unambiguous items * the meaning of items should be clear * acceptable answer should also be clear both for the test writer and test taker * No place for a test taker to make any interpretations for the question in different ways on different occasions

How can performance of candidates be made reliable? 3- Write an unambiguous items How can we prepare such clear items Write a draft for items Give them to a peer / colleague to check it Pilot it among non-participant students If no chance for piloting, scorers will take the responsibility.

How can performance of candidates be made reliable? 4- Provide clear and explicit instructions *Peer / colleague correction and criticism TO AVOID complaining about students’ being unintelligent, stupid ;) 5- Ensure that the tests are well laid out (edited) and perfectly legible (easy to read) NO badly typed items or no legible handwriting NO too much text in too small a space NO poorly re-produced copies WHY??? LOWER THE RELIABILITY

How can performance of candidates be made reliable? 6- Candidates should be familiar with format and testing techniques *Ask familiar questions and format and structures that they have been familiar with 7- Provide uniform and non-distracting conditions of administration *administrating the difference in uniformity, timing (the same strictness)- acoustic conditions – a quiet setting with no distracting sounds or movements

How can score reliability be achieved? 1- Use items that permit scoring which is as objective as possible 2- Make comparison between candidates as direct as possible 3-Provide a detailed scoring key 4-Train scorer

How can score reliability be achieved? 5-Agree acceptable responses and appropriate scores at outset of scoring 6- Identify candidates by number not name 7- Employ multiple, independent scoring

How can score reliability be achieved? 1- Use items that permit scoring which is as objective as possible *How can objectiveness be accomplished? -Multiple choice (not easy to prepare) Open-ended questions with has a unique, possibly one-word correct response candidates produce (spelling may be problematic) Example: In a listening test: Q: what was different about the results? A)_________________________________________. B)_______was more closely associated with_______.

How can score reliability be achieved? 2- Make comparisons between candidates as direct as possible *How can this be done? *Asking the same questions and expecting the same answers *Limit the students’ way of answering and choosing the quesitons NO MUCH FREEDOM ;)

How can score reliability be achieved? 3- Provide a detailed scoring key -A clear explanations for possible acceptable answers -A clear scoring for the correct answers -Prepare as much detailed answer key as possible: partially or totally accepted and how grading will be done for them half or full point?

How can score reliability be achieved? 4- Train scorers *If the scoring will be subjective , take writing courses, SCORERS should be familiar with the material and questions because it needs special training. *Double marking should be done.

How can score reliability be achieved? 5- Agree acceptable responses and appropriate scores at outset of scoring *Every scorer should all agree on the scores for the test or else scoring can’t start *open to new possible answers and be flexible *Remind the possible answers and make everybody agree on the scoring *If so, all scorer should be given knowledge about it and those should be added to the answer key

How can score reliability be achieved? 6- Identify candidates by number not name Knowing students’ name, gender, nationality, physical appearance affect scoring process Blind reading = to be objective 7- Employ multiple / independent scoring Scoring should be done by two different independent scorers =Double markers And the difference should be observed and a final decision is made by a third person

Gürşen SARIALTIN & Tuba DEMİR THANK YOU Gürşen SARIALTIN & Tuba DEMİR