1. Unit 5: Improving and Assessing the Quality of Behavioral Measurement
PS 522: Behavioral Measures and Interpretation of Data
Lisa R. Jackson, Ph.D.

2. Indicators of Trustworthy Measurement
Validity
Directly measures a socially significant behavior
Measures a dimension of the behavior relevant to the question
Helps you determine that you are testing what you think you are testing

3. Indicators of Trustworthy Measurement
Accuracy
Observed values match the true values of an event
Reliability
Measurement yields the same values across repeated measurement of the same event
This is a measure of consistency, stability
Does it reliably produce the same results over time? Is your assessment reliable within itself?
If you use different versions, are they all equally reliable?

4. Types of Validity: Face Validity
This is all about how the tests looks, not what it really measures.
Does the test measure what the test items suggest it will?
High face validity can increase test-taker’s confidence, interest and motivation in the test.
Low face validity could disguise the real purpose of the test to decrease self-report bias
Good for measuring something people don’t want to admit to
It is the least scientifically accurate measure of validity

5. Types of Validity: Content Validity
How well the test samples behavior representative of the whole universe of behavior it is designed to sample.
Ex: A test of depression needs to measure thoughts, emotions, motivation and behavior, not just mood.
An assessment for autism cannot just assess language skills
Affected by who writes the test

6. Types of Validity: Criterion Related
This is an external measure – how well does the test measure up against another standard?
How well can a score on this test be used to infer an individual’s likely standing on the criterion?
Concurrent: How well is a score on this test correlated to an individual’s likely standing on a criterion of interest in the present?
Ex: if you test low on one depression test, do you also test low on the Beck Depression Inventory?
 Predictive: How well is a score on this test correlated to an individual’s likely standing on a criterion of interest in the future?
Ex: Does SAT predict college achievement?

7. Types of Validity: Construct Validity
How appropriate are conclusions drawn from test scores regarding an individual’s standing on a construct?
Does this test really assess the construct?
This takes time to determine.
Ex: Over time, the Beck Depression Inventory has acquired construct validity. Most experts agree that it tests for depression.

8. Threats to Measurement Validity
Indirect measurement
Measuring a behavior other than the behavior of interest
Example: Using children’s responses to a questionnaire as a measure of how often and how well they get along with their classmates
Measuring a dimension that is irrelevant or ill suited to the reason for measuring behavior
Example: Using a ruler in a pot of water to measure temperature
Trying to measure reading endurance in oral reading by counting the number of correct and incorrect words read, but not counting how long the student read

9. Types of Reliability: Internal Consistency
This one is really all about the questions, not the test.
The goal is to see if the questions are consistent with one another.
Do test items reliably produce the same results?
Do all of the questions measure the same idea?

10. Types of Reliability: Split Half
Divide a test in half equally
Give one half to one group, the other half to an equal group
The goal is to judge the consistency of the test
Are the 2 halves strongly correlated to each other? If they are, the test is reliable.
One way is to use odd/even reliability: questions 1,3,5,7 on one half, 2,4,6,8 on other
Use when impractical to have two tests or two administrations; saves time and expense

11. Types of Reliability: Test/Retest
Using the same instrument to measure the same thing at two different points in time.
 The goal is to determine the consistency of the test over time
Does it produce similar results each time?
If so, the test is reliable
Good for a stable construct such as personality
Won’t work to assess a reading test if subjects improve in reading between administrations

12. Types of Reliability: Parallel and Alternate Forms
A make-up test is an example
You might not want to give the same one
Minimizes memory effects if administering the same test to the same person
The person won’t have memorized answers from testing to testing
The mean on each form must be equivalent to the original
Can be time consuming, expensive

13. Assessing the Reliability of Measurement
Measurement is reliable when it yields the same values across repeated measures of the same event
Not the same as accuracy
Reliable application of measurement system is important
Requires permanent products for re-measurement
Low reliability signals suspect data

14. Using Interobserver Agreement to Assess Behavioral Measurement
The degree to which two or more independent observers report the same values for the same events
Determine competence of new observers
Detect observer drift
Judge clarity of definitions and system
Increase believability of data

15. Requisites for IOA Observers must:
Use the same observation code and measurement system
Observe and measure the same participants and events
Observe and record independently of one another

16. Methods for Calculating IOA
Percentage of agreement is most common way to calculate
Event Recording methods compare:
Total count recorded by each observer
Mean count-per-interval
Exact count-per-interval
Trial-by-trial

17. Methods for Calculating IOA
Timing recording methods:
Total duration IOA
Mean duration-per-occurrence IOA
Latency-per-response
Mean IRT-per-response
Interval recording and Time sampling:
Interval-by-interval IOA (Point by point)
Scored-interval IOA
Unscored-interval IOA

18. Considerations in IOA
Obtain and report IOA at the same levels at which researchers will report and discuss in study results
For each behavior
For each participant
In each phase of intervention or baseline

19. Considerations in IOA
Believability of data increases as agreement approaches 100%
History of using 80% agreement as acceptable benchmark
Depends upon the complexity of the measurement system

20. Considerations in IOA Reporting IOA
Narrative form
Table
Graphs
In all formats, report how, when, and how often IOA was assessed

21. Assessing the Accuracy and Reliability of Behavioral Measurement
First, design a good measurement system
Second, train observers carefully
Third, evaluate extent to which data are accurate and reliable
Measure the measurement system
Accuracy means the observed values match the true values of an event
You can’t base research conclusions or treatment decisions on faulty data

22. Assessing the Accuracy of Measurement
Four purposes of accuracy assessment:
Determine if data are good enough to make decisions
Discovery and correction of measurement errors
Reveal consistent patterns of measurement error
Assure consumers that data are accurate

23. Accuracy Assessment Procedures
Measurement is accurate when observed values match true values
Accuracy determined by calculating correspondence of each data point with its true value
Process for determining true value must differ from measurement procedures
Accuracy assessment should be reported in research

24. Threats to Measurement Accuracy and Reliability
Inadequate observer training
Explicit and systematic
Careful selection
Train to competency standard
On-going training to minimize observer drift

25. Threats to Measurement Accuracy and Reliability
Unintended influences on observers
Observer expectations of what the data should look like
Observer reactivity when she/he is aware that others are evaluating the data
Measurement bias
Feedback to observers about how their data relates to the goals of intervention

26. Final Points
The data that you gather is used to help improve the lives of real people
Others may use your data as the basis of intervention
Measurements need to be accurate, consistent and relevant

27. Questions?? Thanks for participating!
I am sure you have been asking questions here in seminar!  Great job!
But, if you have more, me:
These slides are posted in the Doc Sharing area for your review.