1. Validity and Reliability II: The Basics
EDU 300 | Newberry College Jennifer Morrison
Picture:

2. Validity and Reliability II
What is the difference between reliability and validity? Why are they important concepts?
How can you make your assessments more reliable?
How can you make your inferences from those assessments more valid?
By the end of class you should be able to answer these questions.

3. Reliability Definition #1
Statisticians say an assessment is reliable when it gets consistent results over time. That means if Susie takes Assessment X in January she will get the same results when she takes the assessment (even if it’s a different version) in June, given that she does not learn anything that’s on Assessment X between tests.
Discuss the statistical definition of reliability.

4. Reliability Definition #2
For our purposes, let’s use James Popham’s definition of reliability – that the assessment measures what it is supposed to measure. That means when Susie takes Assessment X that it actually assesses what Susie learned in Course X. Do your assessments measure what they are supposed to measure? How do you know?
Reveal the first question and ask participants – “Are your classroom assessments reliable? In other words, do they measure what they are supposed to measure?” Participants will probably answer yes.
Ask participants what our classroom assessments are supposed to measure. The general consensus should be SC Academic Standards and objectives.
Reveal the second question – “How do you know?” – and state that most teachers’ classroom assessments are not actually reliable because they are not on target with SC Academic Standards or even the teacher’s stated objectives. This is one good reason to write your standards and objectives on your assessments.

5. There is a general feeling that data from teachers’ classroom tests are not reliable. Why?
People feel that teachers’ classroom tests are not reliable. Why?
Reliable is defined by many as meaning trustworthy.
Difference between students’ grades and achievement on standardized tests
Curriculum and assessment inconsistency between teachers
Teachers teach and test what they like..
Tests we’ve seen don’t look good..
No one knows what really happens in a teachers classroom (the door is shut)
Students tell parents they aren’t learning anything
Parents aren’t getting any other information
No one can figure out where grades come from
Picture:

6. On Target vs. On Topic
For classroom assessments to be reliable in regard to state standards, assessment items must be on target, not just on topic.
Discuss these two points. Ask participants to give examples of how teachers’ assessments might be on topic but not on target.

7. Example
Standard = Use context clues to determine the meaning of technical terms and other unfamiliar words. (SC E4-3.1)
Something used to confine a dog is… A) a cage, B) training, or C) identity tags.
How would the assessment question have to be structured in order to assess the standard effectively?
How would the assessment question have to be structured in order to assess the standard at an advanced level?
Show and discuss this example.
The example question is not on target because the are no context clues for students to use in order to determine the meaning of the word confine. In fact, in this instance, the word had been previously taught and memorized. Therefore, even if there were context given, students would simply have to remember what the word meant, not use context clues. This standard cannot be met unless students must apply their knowledge of context clues to determine the meaning of an unfamiliar word.
To assess the standard effectively, the word must be unfamiliar and students must use context clues in a sentence or passage to determine meaning. The sentence or passage must contain context clues students should use. The assessment might ask the student to also explain what context clues he/she used.
To assess the standard at an advanced level, the student would have to create context clues for unfamiliar words or evaluate the ways in which an author supplies context clues with his/her word choice. The student could also design a mental tool for locating and analyzing context clues.

8. How can we make sure our assessments are reliable?
Be sure assessment items are on target (not just on topic).
Follow the rules when designing your assessment items.
Assess your assessment.
DIY: Do an item analysis; Have two people evaluate (inter-rater reliability)
Use statistics: Find the reliability coefficient (using test-retest , equivalent forms, or internal consistency methods)

9. SEM
A standard error of measurement is a statistical number that indicates the amount of error to allow for when interpreting assessment scores. The SEM shows how many points we must add to or subtract from an individual’s test score in order to estimate the range of that individual’s true score, or score free from error.
We use the SEM to create a score band or confidence band.

10. Validity Are the inferences we make from the data accurate?
We are most likely to draw valid inferences when we know…
the assessment and assessment procedure.
how the assessment results were determined and what they mean.
what was assessed.
the consequences of using the assessment.

11. How can we make sure our inferences are valid?
Make sure the assessment is reliable (that it assesses what it is supposed to assess).
Make sure the assessment has an adequate sample (items, tasks)
Have a procedure and rationale in place for scoring. The “numbers” need to be useful and mean something.
Think about your interpretations.
DIY: Compare performance to other measures, class averages, across multiple groups; Go over the assessment and have students share their thought processes
Use statistics: Correlation coefficient (degree of relationship between two measures); Expectancy table
Correlation between SAT scores and college grades Correlation between absences and course grades

12. What’s Due?