1. Chapter 5: Improving and Assessing the Quality of Behavioral Measurement
Cooper, Heron, and Heward
Applied Behavior Analysis, Second Edition

2. Indicators of Trustworthy Measurement
Validity
Directly measures a socially significant behavior
Measures a dimension of the behavior relevant to the question
Ensures the data are representative
Cooper, Heron, and Heward
Applied Behavior Analysis, Second Edition

3. Indicators of Trustworthy Measurement
Accuracy
Observed values match the true values of an event
Reliability
Measurement yields the same values across repeated measurement of the same event
Cooper, Heron, and Heward
Applied Behavior Analysis, Second Edition

4. Threats to Measurement Validity
Indirect measurement
Measuring a behavior other than the behavior of interest
Requires inferences be made about the relationship between those behaviors
Must provide evidence that the behavior measured is directly related to behavior of interest
Cooper, Heron, and Heward
Applied Behavior Analysis, Second Edition

5. Threats to Measurement Validity
Measuring a dimension that is irrelevant or ill suited to the reason for measuring behavior
Measurement artifacts
Cooper, Heron, and Heward
Applied Behavior Analysis, Second Edition

6. Measurement artifacts
Misleading data that result from the way behavior is measured:
Discontinuous measurement
Poorly scheduled observations
Insensitive or limiting measurement scales
Cooper, Heron, and Heward
Applied Behavior Analysis, Second Edition

7. Threats to Measurement Accuracy and Reliability
Human error
Poorly designed measurement systems
Cumbersome
Difficult to use
Complex
Cooper, Heron, and Heward
Applied Behavior Analysis, Second Edition

8. Threats to Measurement Accuracy and Reliability
Inadequate observer training
Explicit and systematic
Careful selection
Train to competency standard
On-going training to minimize observer drift
Cooper, Heron, and Heward
Applied Behavior Analysis, Second Edition

9. Threats to Measurement Accuracy and Reliability
Unintended influences on observers
Observer expectations of what the data should look like
Observer reactivity when she/he is aware that others are evaluating the data
Measurement bias
Feedback to observers about how their data relates to the goals of intervention
Cooper, Heron, and Heward
Applied Behavior Analysis, Second Edition

10. Assessing the Accuracy and Reliability of Behavioral Measurement
First, design a good measurement system
Second, train observers carefully
Third, evaluate extent to which data are accurate and reliable
Measure the measurement system
Cooper, Heron, and Heward
Applied Behavior Analysis, Second Edition

11. Assessing the Accuracy of Measurement
Accuracy means the observed values match the true values of an event
No one wants to base research conclusions or treatment decisions on faulty data
Cooper, Heron, and Heward
Applied Behavior Analysis, Second Edition

12. Assessing the Accuracy of Measurement
Four purposes of accuracy assessment:
Determine if data are good enough to make decisions
Discovery and correction of measurement errors
Reveal consistent patterns of measurement error
Assure consumers that data are accurate
Cooper, Heron, and Heward
Applied Behavior Analysis, Second Edition

13. Accuracy Assessment Procedures
Measurement is accurate when observed values match true values
Accuracy determined by calculating correspondence of each data point with its true value
Process for determining true value must differ from measurement procedures
Accuracy assessment should be reported in research
Cooper, Heron, and Heward
Applied Behavior Analysis, Second Edition

14. Assessing the Reliability of Measurement
Measurement is reliable when it yields the same values across repeated measures of the same event
Not the same as accuracy
Reliable application of measurement system is important
Requires permanent products for re-measurement
Low reliability signals suspect data
Cooper, Heron, and Heward
Applied Behavior Analysis, Second Edition

15. Using Interobserver Agreement to Assess Behavioral Measurement
The degree to which two or more independent observers report the same values for the same events
Cooper, Heron, and Heward
Applied Behavior Analysis, Second Edition

16. Benefits of Interobserver Agreement (IOA)
Determine competence of new observers
Detect observer drift
Judge clarity of definitions and system
Increase believability of data
Cooper, Heron, and Heward
Applied Behavior Analysis, Second Edition

17. Requisites for IOA Observers must:
Use the same observation code and measurement system
Observe and measure the same participants and events
Observe and record independently of one another
Cooper, Heron, and Heward
Applied Behavior Analysis, Second Edition

18. Methods for Calculating IOA
Percentage of agreement is most common
Event Recording methods compare:
Total count recorded by each observer
Mean count-per-interval
Exact count-per-interval
Trial-by-trial
Cooper, Heron, and Heward
Applied Behavior Analysis, Second Edition

19. Methods for Calculating IOA
Timing recording methods:
Total duration IOA
Mean duration-per-occurrence IOA
Latency-per-response
Mean IRT-per-response
Cooper, Heron, and Heward
Applied Behavior Analysis, Second Edition

20. Methods for Calculating IOA
Interval recording and Time sampling:
Interval-by-interval IOA (Point by point)
Scored-interval IOA
Unscored-interval IOA
Cooper, Heron, and Heward
Applied Behavior Analysis, Second Edition

21. Considerations in IOA During each condition and phase of a study
Distributed across days of the week, time of day, settings, observers
Minimum of 20% of sessions, preferably 25-30%
More frequent with complex systems
Cooper, Heron, and Heward
Applied Behavior Analysis, Second Edition

22. Considerations in IOA
Obtain and report IOA at the same levels at which researchers will report and discuss in study results
For each behavior
For each participant
In each phase of intervention or baseline
Cooper, Heron, and Heward
Applied Behavior Analysis, Second Edition

23. Considerations in IOA More conservative methods should be used
Methods that will overestimate actual agreement should be avoided
If in doubt, can report more than one calculation
Cooper, Heron, and Heward
Applied Behavior Analysis, Second Edition

24. Considerations in IOA
Believability of data increases as agreement approaches 100%
History of using 80% agreement as acceptable benchmark
Depends upon the complexity of the measurement system
Cooper, Heron, and Heward
Applied Behavior Analysis, Second Edition

25. Considerations in IOA Reporting IOA
Narrative form
Table
Graphs
In all formats, report how, when, and how often IOA was assessed
Cooper, Heron, and Heward
Applied Behavior Analysis, Second Edition

26. Assessing the Quality of Measurement
Indicators of the quality of data include:
IOA
Accuracy
Reliability
Can report multiple indices to assess data quality
Cooper, Heron, and Heward
Applied Behavior Analysis, Second Edition