1 LANGUAE TEST RELIABILITY

2 What Is Reliability? Refer to a quality of test scores, and has to do with the consistency of measures across different time, test form, raters, and other characteristics of the measurement context.

3 Con. Test reliability is related to high variance of the true score distribution. (person separability) reliability is a measure of accuracy, consistency, dependability or fairness of scores resulting from administration of the particular examination.

4 The Measurement Model Observed score= True score + Error score X = T + E Observed score: a score that a test taker actually received on a test. (Raw or Obtained score). True Score: as there is always some error in any measurement an individual true score on a test would be his observed score minus some error. T = X - E

5 Standard Error of Measurement The standard error of measurement (SEM) is an estimate of error to use in interpreting an individual’s test score. SEM = s  1 – r) S = the standard deviation for the test r = the reliability coefficient for the test

6 Standard Error of Measurement For example, A test has a split-half reliability coefficient of.96 and a standard deviation of 15 calculate the SEM for this test.

7 Standard Error of Measurement SEM = s  ( 1 – r ) = 15  ( 1-.96) = 15 .04 = 15 x.2 = 3

8 Threat to test Reliability Sources of Error What are some of the factors that introduce error into measurement? 1) Student Factors 2) Construction of the Items 3) Test administration- 4) Scoring 5) Length, difficulty and boundary effect of the Test 6) Regulatory Fluctuation 7)Discriminability, Speediness, and Homogeneity 8)Fluctuation in Response

9 Sources of Error (1) Student Factors--Student fatigue, illness, or anxiety can induce error and lower reliability because they affect performance and keep a test from being a measure of their true ability or achievement. 2) Construction of the Items -- A major threat to reliable measurement is poorly worded or ambiguous questions or tricky questions.

10 Sources of Error 3) Test administration--Environmental factors such as heat, light, noise, confusing directions, and different testing time allowed to different students can affect students' scores.

11 Sources of Error 4) Scoring – An objective test is more reliable because the test scores reflect true differences in achievement among students and not the judgment and opinions of the scorer. subjectivity in score or mechanical errors in scoring process may introduce inconsistency in score and produce unreliable measurement, that usually occur with in or between the rater themselves.

12 Scoring A. Intra- Rater Reliability(mark/er-mark reliability) (Bachman, 1990) when an individual subjectively judges or rates the adequacy of a given sample of language performance for at least two times and gives consistent results, we say that this rating have intra- rater reliability. B. Inter-rater reliability Which refers to consistency of rating given by different raters to a sample of language performance.

13 Sources of Error (5) length, difficulty and boundary effect of the Test A- reliability is affected by number of item in the test. More items in the test make a grater range of score and grater reliability. B- A test that is either too easy or too difficult for the class taking it will typically have low reliability. This occurs because the scores will be clustered together at either the high end or the low end of the scale, with small differences among students( boundary effect).

14 Sources of Error (6) Regulatory Fluctuation –Differences in the clarity of instructions, the time of test administration, test administrator interaction with examinees, prevention of cheating behavior, and reporting of time remaining are all potential source of measurement error.

15 Sources of Error (7)Discriminability, Speediness, and Homogeneity A- Discriminability: the degree to which a test or an item of the test distinguishes among stronger and weaker test taker. Great discriminate = Great reliability

16 Sources of Error B- Speediness: Speed test: A test in which the items are easy but the time limits are so short that a few or non of the test takers can complete all the items. such a test aims to determining the speed of the testees to do certain task Power test :A test in which item difficulty generally increase gradually but ample time is given to all candidate. The aim is determine how much an individual is able to do, not how rapidly.

17 Sources of Error In power test failure to allow examinees a reasonable amount of time to complete the test will reduce the reliability. If the test becomes more difficult as a result of the element of speedness, reliability will diminish. C- Homogeneity We can increase reliability and reduce error by including items of similar format and content.(e.g split half method)

18 Sources of Error (8)Fluctuation in Response A- Response arbitrariness B- Wiseness and familiarity Response

19 Methods of Reliability Computation The choice of the method of computation of reliability will depend on such factor as Nature of threats to reliability present Ease of computation Nature of the test Testing situation

20 Methods of Reliability Computation Test-Retest Method Parallel Form Method Inter – Rater Reliability Split Half Reliability KR-20 KR-21

21 Methods of Reliability Computation 1-Test-Retest Method Refer to correlation of two sets of score for the same persons. An approach to estimating reliability in which we administer the test twice to the group of individuals and then compute the correlation between two sets of scores. R= r1,2

22 Methods of Reliability Computation Test-Retest Method disadvantage: 1-time consuming. it is difficult to arrange two testing session an preparing similar condition for the same group of examinee. 2- test effect.students may learn or memorize some question

23 Methods of Reliability Computation 2-Parallel Form Method Two tests of the same ability, and with equal length and difficulty that are administrated to the same sample of persons. disadvantage: constructing two parallel forms of a test is not an easy task.

24 Methods of Reliability Parallel Form Method Equated test: Any two sets of scores from different test (assuming that the same trait is being tested) that have been reduced to a common scale to facilitate comparison.

25 Methods of Reliability Parallel Form Method Random parallel tests: It has been used to described tests that have been composed of items drawn randomly from the same population of items ru = rA,B ru = the reliability coefficient rA,B = the correlation of form A with the form B of the test

26 Methods of Reliability Computation 3-Inter – Rater Reliability Estimation based on the correlation of scores between/among two or more raters who rate the same item, scale, or instrument. the actual level of reliability will depend on number of raters or judges. the more rater present in the determination of the mark, the more reliable will be the mark.

27 Intra- Rater Reliability(mark/er-mark reliability) (Bachman, 1990) when an individual subjectively judges or rates the adequacy of a given sample of language performance for at least two times and gives consistent results, we say that this rating have intra- rater reliability.

28 Methods of Reliability Inter – Rater Reliability there are two steps in the estimation of inter-rater reliability: 1-an average of all correlation coefficients 2-Spearman Brown Prophecy Formula

29 Methods of Reliability Computation 4-Split Half Reliability Obtained from a single administration by dividing the tests into two comparable halves and comparing the resulting scores for each individual (split into odds and evens). an approach to estimating the internal consistency of a test.

30 Methods of Reliability Split Half Reliability disadvantage: a- reliability can be change according to the manner in which the test is divided. (split into odds and evens) b- homogeneous item. Because assuming the equality between the two halves is not always the safe assumption.( different subsection, in a test e.g. grammar, vocab, reading,…will change test homogeneity and thus reduce the test score reliability )

31 Methods of Reliability Split Half Reliability advantage: it is more practical than other. Because: 1-no need to administer the same test twice. 2-not necessary to develop two parallel forms of the same test. 3-single administration will be enough.

32 Methods of Reliability Split Half Reliability Spearman Brown Prophecy Formula. e.g. if the reliability coefficient of half of the test is computed to be 0.80.what would be the reliability of the total test?

33 Methods of Reliability Split Half Reliability it should be logically clear that the reliability of the total test will always be higher than the reliability of half of the test.

34 Methods of Reliability Computation 5-KR-20 Kuder-Richardson Formula 20 Permit us to arrive at the same final estimate of reliability without having to compute reliability estimates for every possible split half combination.

35 Kuder-Richardson Formula 20 It is based on number of item on the test = n or K difficulty of the individual items variance of the total test score = V

36 Methods of Reliability Computation 6-KR-21 Kuder-Richardson Formula 21 is a formula that is easier to use but less accurate than KR 20. This formula is based on the assumption that all item in the test are designs to measure a single trait.

37 Kuder-Richardson Formula 21 This formula, known as KR-21

38 Kuder-Richardson Formula 21 e.g. Suppose we gave a 50-item test and the mean score was 43 and the variance was 25 Putting these values into KR-21. K= 50 X= 43 V= 25

39 Kuder-Richardson Formula 21 Solving for r obtains: r= (1.02) (0.76) = 0.78 the reliability coefficient is greater than 0.70, so we can use this test with some degree of confidence.

40 Correction for Attenuation Henning,1987 A way of holding reliability constant when making comparison among correlation coefficient. It is made by dividing the correlation coefficient by the square root of the cross-product of reliability.

41 Correction for Attenuation E.g If a test of composition writing correlated 0.55 with the test of grammar usage, disattenuate this correlation, assuming that KR20 reliabilities of the tests were 0.70 for composition writing and 0.80 for grammar usage.

42 Correction for Attenuation