1. Technical Adequacy
Session One
Part Three

2. Reliability We all have friends, some are reliable and some are not
With your partner, discuss what a reliable friend is,
List three qualities you would use?

3. Reliability
In Laymen’s term, reliability is being able to depend that the results are accurate for that test.
If you did it again, would you get the same score?
There are many factors that affect reliability

4. Error in measurement Two types of error in measurement Systematic-Bias
Random

5. Bias
Generally bias refers to raising a persons score because they were advantaged in some way
However, the groups that was not advantaged, was affected negatively by the bias
Boys score better on multiple choice questions than girls, the boys were advantaged, the girls were disadvantaged

6. Random error Random error – is very different
It is hard to predict who it is affecting,
hard to predict by how much,
Hard to predict by what magnitude
Reliable test try and eliminate most types of error

7. Reliability Coefficient
We can measure how reliable tests are by the reliability coefficient
A test free from error- has perfect 1.0
A test filled with error –has a 0
Since every test has error then a reliability around .85 or above

8. Types of reliability Item reliability Stability
Inter-rater reliability or interobserver agreement

9. Item reliability
Item reliability affects the prediction of understanding of the knowledge in several ways
Imagine a study trying to predict how the population of a country or state will vote in the next election
The prediction is only as good as the sample it selects, if it select from one area, it will not be representative of the population
This same concepts applies to developing a test
Test developers cannot possible select all the items they need to test, the more accurate the representative is of the total knowledge, the more reliable the test

10. Item reliability
Your goal is for the student performance on the sample items would be the same as if he/she took all of the items ( if that were a possibility)
The goal of the test is to be able to generalize the students ability to what they know of the entire realm of knowledge in that area
When we over estimate their ability, our test is unreliable

11. Item reliability
There are two main approaches to determining item reliability
Alternate form reliability-
Internal consistency

12. Item reliability Alternate form reliability-
two forms of a test are developed, each from the same knowledge base but each with different questions
You then test a large sample with the test
Half take one form , half the other
They should have similar scores
Scores from the test are correlated and form the correlation coefficient

13. Item reliability Internal consistency
There are many ways to test internal consistency
On popular way is to develop a test that can be split with a similar level of difficulty
Administer the test and see how the students did
Say the test was split by first half an second half, grade half of the class on the first half and the other half on the second half and compare scores.
Can also do if for specific items

14. Stability
In many cases, we expect out tests to produce information that when tested later, will yield the same results
A child tested for colorblindness- should reveal being colorblind later in life since the problem is not curable, if not the test was unreliable because it is unstable

15. Stability A test should produce similar results
I you give a set of students a test and then wait a while, then readminister the test, it should produce similar results
The more similar the results, the more stable and the more reliable

16. Stability Stability is not affected by, interventions.
If you test a child and it shows he is weak in a certain area, then you provide and intervention and the child does better on the next test, that is not considered a weakness in stability

17. Inter-rater reliability
Inter observer/inter-rater reliability
The concept is simple and easy to understand-
It is analogous to a piece of music, a book or a movie,
Two people see, read or watch the same thing and have a different opinion
Watch the next clip, what do you think?

19. Inter-rater reliability
Now Watch the next clip, an count how many people test the mattress
Do people have similar answers

21. Inter-rater reliability
Inter-rater reliability needs to be developed in several places and can be measured in several ways
Different raters/observers need to be trained on what to watch, need to have a clear criteria for what is a positive incident of what you are observer
If you are looking for out of seat behavior, is it standing, squirming, leaning over, or being two feet from the desk

22. Inter-rater reliability
Inter-rater reliability can be measured in several ways, by comparing two people scores from the same
Or by doing an item by item analysis and comparing the difference observation

23. Standard Error of Measurement

25. Standard Error of Measurement
Imagine you gave a test to a kindergarten student on his letter sound recognition
You developed 100 test of ten items
After giving the child about ten of these test, the scores would be about the same.
Some of the test he would know the sounds, some he would not, but the average would be accurate
SEM tries to predict what that error between the test would be if you only gave him one test, remember it could be a test he scored well on, or it could be a test he scored poorly on
It is a similar concept to Standard Deviation, but related specifically to error

26. Estimate of True Scores
This is more of a conceptual concept, that a statistical unit
Imagine you take a fifty question test and you do not know ten answers questions
You guess on them and being a very lucky person, you get 8 right- These eight answers are really not your true score
If you are unlucky, you get a lower score

27. Confidence Intervals
Given the fact that true scores are difficult to obtain, the concept of confidence intervals was created.
When it is combined with SEM it relays very accurate scores
The level of confidence tells us how certain the score is within the range

28. Confidence Intervals
If a child has a score of 90 ± 5 ( SEM) the we are saying the child score is somewhere between 85 and 95.
If we say that a child has a score of 90 ± 5 ( SEM) with a 95% confidence level, we are saying that there is only a 5% chance that the child score is somewhere above or below 85 and 95.
The lower the confidence, the smaller the range the child score is somewhere between 88 and 92. at a 80% confidence level

29. Validity
This refers to the degree to which the evidence and theory support the interpretation of the test scores by the proposed uses of tests
Often test are interpreted for uses they were not designed.
Therefore, Validity is a fundamental consideration

30. Validity
The fundamental question that you need to ask, is, Does the testing process lead to the correct inferences about a specific person.

31. Validity
First assume you give an IQ test in English to a non English speaking person
You give a test that measures cultural items a that a person was not exposed to
You use a test designed for national standards that does not align to a local standards ( social studies)

32. Validity
Content validity- Is the content of the measure representative of the domain of content it is suppose to assess? Experts look at the content and compare it to what they feel it should contain.

33. Validity Appropriateness of included items-
Should the questions be here
Do they represent what it is trying to measure ( different than content validity) are the questions from a too high of a grade level, like middle school stuff on an elementary test
Is the presentation of the items appropriate, are the questions worded properly?

34. Validity
Content not included- is there important content missing that should be there?
How are the items measured
Are the multiple choice,
Open ended where you must show work

35. Validity
Criterion Reference Validity- references a tests ability to describe a test takers ability in two ways
Present- Concurrent Criterion Referenced Validity
Future- Predictive Criterion Referenced Validity

36. Validity Concurrent Criterion Referenced Validity-
Is the test/assessment a good predictor of what the students currently know based on the criterion of the knowledge base?
If a child takes an achievement test. Is it a valid measure of how well he did in fourth grade?

37. Validity Predictive Criterion Referenced Validity
Does the test have the ability to predict what it say it will predict
A reading readiness test- if a students scores high, does he learn to read easily?
If a child scores poorly, does he struggle to learn to read?

38. Validity
Construct Validity refers to the extent to which a procedure or test measures a theoretical trait or characteristic
construct validity refers to whether a scale measures or correlates with the theorized psychological construct ( such as intelligence) that it purports to measure.