1. ACCESSIBILITY TESTING, EVALUATING AND REPORTING
AHG 2013
Hadi Rangin
University of Illinois
EASI Track

2. Overview Background on accessibility evaluation and reporting
Evaluation methodology
Common Accessibility Testing Criteria
Essential Accessibility Criteria (Show-stoppers)
Reporting evaluation results
Identify audience
Closing remarks
Contacts and FAQ

3. Background Increasing attention to accessibility by administrators
Checking for accessibility after adoption
Some checks for accessibility in the RFP process
Inconsistency in accessibility evaluation and reporting

4. Statement of the Problem
No normalized evaluation and reporting
Depends very much on who has performed & furnished report
Tests miss essential features and include testing for non-critical elements
Reports include raw data from automated accessibility checkers without explanation
Reports are not targeted for their audience
Lack of impact of the issues, prioritization of issue and possible solutions
Administrators need reliable data to make strategic decisions

5. The Goal Define guidelines for accessibility evaluation & reporting
Define a methodology for accessibility testing & evaluation
Communicate how to write effective reports
Engage vendors in constructive collaboration

6. Evaluation Methodology
Create a list of functional tasks for the application.
Define Accessibility/Usability Requirements.
Divide functional tasks into a logical series of steps
Start with Testing & Evaluation based on the functional tasks and criteria identified
Watch for Show-stoppers, recommended, and other potential technical criteria.
Record your observations and findings

7. Create A List of Functional Tasks
An application is functionally accessible if the user can:
Accomplish applicable tasks successfully, independently and reliably in a reasonable time.
Functional tasks are unique to each application
Important: work with responsible entities to learn how the application is going to be used in your setting
Prioritize functional tasks
Functional tasks require manual testing
Be ready to spend a lot of time on it

8. Functional tasks example: Webmail
Logging in and out
summary
Finding desired message(s)
Reading an
Tagging or moving an to desired folder
Sending an
Misc. tasks

9. Define Accessibility/Usability Requirements (1)
Keep in mind at all times:
Consider the differing needs of users with disabilities:
Physical disabilities, low-vision, color-blind, blind, deaf, cognitive impairments
Goal is functional accessibility; answer the following questions:
How usable is the application for diverse groups of users?
What are the impacts of accessibility issues in completing the task?

10. Define Accessibility/Usability Requirements(2)
3. Consider the following from the perspectives of users with differing needs:
Perceivable – Information and user interface components must be presentable in ways that the user can understand.
Operable – User interface components and navigation must be operable.
Understandable - Information about and the operation of the user interface must be understandable.
Robust – Content must be robust enough that it can be interpreted reliably by a wide variety of user agents, including assistive technologies.

11. Divide functional tasks into steps(1)
No magic recipe that would work for all applications.
Key factor is that all the relevant functionality must be available to all AT users.
Example: Reading an
Can user identify that he is in the right folder (Inbox)?

12. Divide functional tasks into steps(2)
Can user navigate to the desired message(s)? The user needs to know:
Message status (read, unread, replied to, flagged)
Subject
Date
Sender
Recipients
Presence of attachments
Can user open the message and read the message body?
Can user access and open the attachments?

13. Before you begin Specify: Computer system characteristics
Browser information
Assistive technology used
Any third-party software information
Accessibility testing tools
Study all manuals, configuration/optimization documentation
Study specific configuration information for users with disabilities
Engage the IT professionals in charge of the applications
Have a copy of the Show-stoppers & common technical requirements
Have a second person to observe/take notes

14. Begin with Testing and Evaluation
Testing and evaluation require good knowledge of accessibility and familiarity with APPLICATION development.
As you go through each step for each functional task, check for technical accessibility of that step
You can use automated testing tools to determine selected technical accessibility issues:
FAE, WAVE, Accessibility Evaluator for Firefox, OAA Accessibility Inspector (in development), etc.
Never use an AT like Jaws to determine the technical accessibility issues.
Take good notes & make screenshots whenever possible
Mark any functional tasks that violate Show-stopper criteria
Stop if you see too many Show-stopper violations
Use the Common Testing Categories & Criteria as a guide

15. Common Accessibility Testing Categories
Structure, Navigation, Orientation
Styling
Content
Forms
Help and Documentation
Personalization and Customization

16. Page Structure, navigation, orientation
The most important elements of accessibility.
Logical structure is critical for effective & efficient interaction with an application.
User must always know where he/she is.
User must be able to locate and move to the desired components
User must be able to interact with the page with keyboard
Visual layout and order must match the keyboard order

17. Styling Use of table for layout Text zooming and window resizing
High-contrast mode
Stylistic images

18. Content Link and link text Data table Grouping relevant items
Informational graphics
Use of color
Audio/video
Embedded objects
Challenge-response mechanism

19. Forms Primary method for user to interact with application
User needs to concentrate on the data being entered
Not being challenged or distracted by the complexity of the interface and navigation
Entering data with certainty
Notification of warning, error, or successful form submission
Being able to identify easily places where errors have occurred and navigate to them
All form controls must be accessible

20. Help and Documentation
New, complicated, and challenging applications
Understanding the layout and general features of the application
Some tasks require complicated instructions
Quick guide in the application
Presence of supported accessibility features
Quick accessibility guide

21. Personalization and Customization
Users have different needs and ways of viewing and interacting with application
Adaptation of application to user's individual needs instead of user adapting to the application
Many global settings and personalization/customization features can improve or limit accessibility/usability

22. Essential technical accessibility criteria (Show-stoppers) (1)
All critical functionality of a given application must be operable via the keyboard without relying on keyboard shortcuts
This includes all menus, other navigation, and dynamic objects (such as image slideshows)
A clear visual indicator of the current item in focus must be displayed (similarly to how mouse hover works in a webpage)
Form fields and buttons must have valid programmatic labels that are available to assistive technologies
Color alone must not be used to convey application state and other information

23. Essential technical accessibility criteria (Show-stoppers) (2)
The application must have a logical and programmatically discernible structure that follows the visual structure of the application.
The reading order and tab order for an application must be logical for the program.
The user interface must remain usable in high-contrast color modes.
Informational images, charts, and graphs must have descriptive alternative text. Charts and graphs must also have additional detailed descriptions of their content.

24. Essential technical accessibility criteria (Show-stoppers) (3)
Flash objects and other embedded applications must accept keyboard focus and never trap it. All contents and possible controls in those objects must be also accessible and operable with keyboard.
Audio content must have transcripts, video with audio must have captions and audio description (where appropriate), and any associated media player controls must be operable by keyboard.
CAPTCHAs (generically referred to as a challenge-response mechanism) must be accessible.

25. Reporting Evaluation Results
An effective evaluation report concisely does three things:
It communicates whether or not the application is accessible.
It communicates the impact of any issues found, in regards to the specific application.
It prioritizes the results for remediation, from most impactful to least.

26. Effective Report Structure
Like a Funnel: Moving from a broad conceptual overview to detailed, conformance-level information.
Executive Summary
Evaluation Methods
Include a list of identified functional tasks that were tested
Prioritized Issues
With applicable conformance criteria from accessibility standards and guidelines
Recommendations for remediation (as appropriate)
Include links to applicable resources for best practices and applicable techniques

27. Executive Summary
Good for administrators, purchasers, and account representatives.
An executive summary should include:
The name of the application being evaluated
Include major and minor version number if appropriate.
A statement of impact
Can the application be considered accessible?
Who will be affected by any issues found, and what is the severity of the impact?
A statement of how well the application conforms to legal accessibility requirements
Helps readers determine legal risk of deployment (especially administrators)

28. Executive Summary (continued)
An executive summary should include:
A brief outline of major functional issues
Ex: users cannot log in successfully or cannot read without difficulty
A remediation statement
How difficult would it be to correct the issues found, and who would need to fix them?
Needed by administrators and purchasers to estimate the cost involved to perform remediation and the possible effect on project timelines
Suggestions of viable alternative applications (for internal reports)
If there are no alternative solutions, administrators and purchasers would likely want to know this.

29. Evaluation Methods Establishes the validity of the evaluation
The functional task list lets others know what areas of the application were evaluated.
Assists others in reproducing the results. Include:
Information about what tools were used to evaluate the application.
Include version or model numbers where possible.
Identify web browsers that were used, including version numbers.
Identify any mobile devices used for testing, including the major and minor revision of the OS on the device.

30. Prioritized Issues Helps create a concrete checklist for remediators
Group issues by functional task
Prioritize the tasks from most critical to least critical
For each task:
Identify any problems found
Include the applicable conformance criteria from the accessibility standards
Prioritize the conformance criteria, from most impactful to least impactful

31. Prioritized Issues (continued)
Many conformance issues arise multiple times and are application-wide.
Applies to most Show-stopper issues.
Ex: Application menus are not keyboard accessible
Call out application-wide issues in their own section.
Group by issue, from most impactful to least.
Include applicable conformance criteria.

32. Recommendations for Remediation
Helps developers with limited or no knowledge about accessibility.
Helps to create a positive partnership with vendors.
Provides best-practice recommendations for remediation of issues.
Links to resources:
Help developers understand accessibility issues
See working examples

33. Identify the Audience Internal (your organization) Vendors
Administrators and purchasers
Technical support and documentation staff
Development teams
Vendors
Top managers / CEO
Sales representatives and account managers
Project managers and development teams
General Public
Online journals promoting accessibility awareness
End-users
Disability advocacy groups
Articles and FAQs for accessibility professionals

34. Reports for Upper-Level Administrators and Purchasers
Usually non-technical
Give them the Executive Summary
Let them know more detailed report is available if desired.

35. Reports for Project Managers, Account Representatives, & the General Public
Need a general overview & functional understanding of the issues
Give them:
Executive Summary
Methodology
Issues by Task

36. Reports for Development Teams
Need to remediate any issues found
Give developers the full report
Let them know that you are willing to help

37. Closing Remarks Define unique functional tasks.
For an effective evaluation report:
Define unique functional tasks.
Test & evaluate the application based on the functional tasks.
Identify your audience.
Target the report accordingly.

38. Hadi Rangin, hadi@illinois.edu
Contact Information
Hadi Rangin,
Questions?