1. Michigan Assessment Consortium Common Assessment Development Series Module 18 Reliability
MAC CAD-PD Mod-6 BRF [comp].ppt 1

2. Edward Roeber Michigan State University
Narrated By:
Edward Roeber
Michigan State University 2

3. Previous Modules By now, you have
Developed your test blueprint
Written your test items
Assembled the items into a field test form
Field tested the items
Looked at the performance of students on the field test
Now it is time to look at the test itself

4. Reliability This is a concept that pertains to the entire test.
Reliability is an important concept because it shows the degree to which the results from your test are stable
Stable test results are a necessary, but not sufficient condition, for the use of the test results to be valid

5. Importance of Stability
Why is stability important?
If we weighed ourselves repeatedly and got wildly different results, we would conclude it was time to buy a new scale
If your test yielded wildly different scores if it was given more than once, we wouldn’t trust it, either
Hence, test results need to be stable

6. Importance of Stability
If the test results are not stable, how can you possibly trust them?
If the first time the student took the test, she got a 99% and second time, she got a 23%, we would not be able to say what her real achievement is
We would conclude that because the test is unreliable, we cannot draw a conclusion about her achievement

7. Test-Retest Reliability
This is one type of reliability – if we gave the same test to students, would they get the same scores?
This is done sometimes by giving the same test about two weeks apart, which is enough time to students to forget the items but not learn new material

8. Test-Retest Reliability
Test-retest reliability is not a major concern with achievement tests
However, with higher stakes tests (e.g., a test to award high school course credit), it is worth doing this
The scores on the two tests would be correlated (something that some ISDs and universities could help you with)

9. Internal Consistency
Another way to look at reliability is the internal consistency of the test
This type of reliability is important if we have a test that is supposed to measure one content area or learning target
An example is a 60-item reading test measuring one part of reading, such as inferential comprehension

10. Internal Consistency
On this sort of test, students should do very well or very poorly on all of the items
However, if we built a reading test that measures 10 learning targets with six items each, we would expect
High internal consistency within each six-item measure,
Less internal consistency across the 10 mini-tests

11. Internal Consistency
This means that the internal consistency values you receive need to interpreted in light of your test blueprint for the common assessment
Were you trying to measure one single concept in your test? Or
Were you trying to measure multiple concepts in that test?

12. Internal Consistency
One way to measure internal consistency is to split the test in half and compare the halves
An easy way to do this is to compare the odd versus even items

13. Statistical Analyses
If you have access to statistical software, you can calculate Cronbach’s alpha, a measure of internal consistency
Some ISDs/RESAs have access to this software
Some universities do, too
SPSS or even MS Excel can be used to calculate Cronbach’s alpha

14. Reliability Levels
Reliability is reported on a scale that ranges from .00 to 1.00
0.00 indicates a test that is completely unreliable
1.00 indicates a test that is completely reliable

15. Reliability Levels
For achievement tests, you want the reliability (internal consistency) to be about 0.8 or higher
Longer tests are more reliable than shorter ones
However, longer tests take longer to administer, so there is a trade-off in using longer tests to increase reliability

16. Summary There are two ways of computing reliability:
Test-retest (stability)
Internal consistency
Neither is easy to compute by hand
If the common assessment has consequences for the student, calculating reliability is essential

17. Summary
When you administer a test for the first time, you will learn more about the test than about the students who took it
You should review the data from the first administration to determine if the test is working
Item difficulties
Qualities of the entire test – reliability/validity

18. Series Developers Kathy Dewsbury White, Ingham ISD
Bruce Fay, Wayne RESA
Jim Gullen, Oakland Schools
Julie McDaniel, Oakland Schools
Edward Roeber, MSU
Ellen Vorenkamp, Wayne RESA
Kim Young, Ionia County ISD/MDE

19. Development Support for the Assessment Series
The MAC Common Assessment Development Series is funded in part by the Michigan Association of Intermediate School Administrators
In cooperation with
Michigan Department of Education
Ingham and Ionia ISDs, Oakland Schools, and Wayne RESA
Michigan State University