Reliability of ADA- CAT for ADA Compliance Denis Anson Director of Research and Development Assistive Technology Research Institute Denis Anson Director of Research and Development Assistive Technology Research Institute

What is ADA-CAT? ADA-CAT is a combination of a website and set of physical tools ADA-CAT is a screening tool for accessibility of the environment, based on ADA standards ADA-CAT was designed to allow non- engineers and non-architects to perform reliable and accurate assessments of compliance with ADA requirements

The Website The ADA-CAT website provides assessment checklists (Audits) for features of the environment

The Website Each item in an audit includes: The ADA (or other) requirement, in understandable language A link to the ADA-ABA requirement on the web An explanation of why this feature is important Just-In-Time Training

Just-in-Time Training Training you get today will be forgotten when you need it ADA-CAT allows you to review how to assess an item as you are assessing This includes both text and pictures of the process The training is based on the ADA-CAT toolkit

The ADA-CAT Toolkit The toolkit includes eleven tools in a custom bag The tools include three off-the-shelf tools 25-foot tape measure 24 inch spirit level Stopwatch

Three Off-The-Shelf Tools The tools are easily found in sports and hardware stores The assumption might be that everyone knows how to use them Watching novices attempt to use these common indicates that this isn’t the case

The ADA-CAT Toolkit The toolkit contains three hard to find, but off-the-shelf tools Light meter reading in Lux A sound meter reading decibels A force gauge

Three Hard-To-Find Tools It’s unlikely that many have ever used tools like this Most people have little understanding of the concepts being measured Inverse Squared Rule? Although the tools are “simple” to use, if not used correctly, they won’t give good results

The ADA-CAT Toolkit The toolkit contains five custom tools not available anywhere else Magic Slope Block MultiTool Key Torque Tool The Font Guide The StoryStick

Five Custom Tools These tools were created specifically for use in ADA-CAT assessments Although the principles are hidden from the user, they can be complex No-one who has not used ADA-CAT will have any experience with these tools.

Little or No Training Required? The assertion is that ADA-CAT, the combination of the website and the tools, will produce reliable results when used by non-architects This is a bold claim, and requires support It is also testable

What is Reliability? There are two types of reliability that are commonly recognized in tests and measurements Test-Retest Reliability Inter-Rater Reliability

Test-Retest Reliability Test-Retest reliability means that when a measurement is taken repeatedly, the outcome is the same When you measure the width of a door, you should always get the same width for the same door Tools that have large subjective components will not show this reliability

Inter-Rater Reliability Formally, this means that the results of a measurement are not dependent on the person doing the measurement If I measure the weight of a glass of water, then you do, we should get the same result In accessibility assessment, the “I was/wasn’t able to do it” test depends on who “I” am

Testing ADA-CAT Reliability A group of four OT research students undertook this task The students received an introduction to the tools of the toolkit, but no observed trial in the real world The students used the Elevator Audit as their sample test

Why Elevators? The Elevator Audit uses almost all of the tools of the toolkit We didn’t use the Font Guide, but did use all the other tools Elevators are commonly available There is a wide range of accessibility in elevators in daily use

Method Identify eight elevators to be assessed Each researcher measures each elevator After at least three weeks, each researcher measures each elevator again

Test-Retest Reliability Each researcher’s measurements can be compared with their measurements three weeks later. Assuming no damage or repair to the elevators, there should be good agreement between the results

Inter-Rater Reliablity The measurements of each researcher can be compared to show the inter-rater reliability When the scores for all four researchers were compared, the inter-rater reliability was.6 Statistically significant, but not exciting Why?

Inter-Rater Reliability

Examining the data shows that, in most cases, three of the researchers had closely related data, and one was different. Removing this individual increased the inter-rater reliability score to above.9 So why was this researcher’s scores different? We’ll look at that in a minute.

Inter-Rater Reliability (Filtered)

Test-Retest Reliability If the scores of the researchers on first measurement are compared with the measurements 3 weeks later, the correlation is.97

Test-Retest Reliability

So, What’s Wrong with Inter-Rater? Individual items were compared between researchers where there was not good agreement The differences clustered around two features Sound levels Braille

Sound Levels The Elevator Audit includes standards for the relationship between elevator signals and background noise, and for background noise levels The outlier researcher did her evaluations during times when there was a lot of surrounding activities During a basketball game for the athletic center elevator

Sound Levels The background sound levels at busy times represent the people in the area, not the elevator, and cannot be controlled by the manufacturer The sound measurements should be taken at quiet times, as the other three researchers did

Braille? Visually dependent people have a very difficult time determining if Braille is appropriate In some of the older, and more abused elevators used in this study, the “raised” component of Braille was lost due to damage and wear This was interpreted differently by different researchers, resulting in some variability

Conclusions Some additional instructions are needed on when to take sound measurements Additional examples and counter- examples of Braille need to be added to the assessments

Conclusions Overall, the assertion that ADA-CAT reliably measures compliance with accessibility standards is well supported by this project