1. A Cross Methodology for Accessibility Evaluation Using a Combination of Automatic Tools and Crowdsourcing Tools
By Irwin Ramirez

2. Abstract
This project suggests a methodology to make web site accessibility testing and evaluation more efficient, cost effective, scalable, reliable, and sustainable. This is done by using a combination of automatic testing tools and crowdsourcing tools. This technique achieves better results since it benefits from advantages of both types of tools such as the scalability and human judgement from automatic and crowdsourcing tools respectively. The procedures provided in this paper aim to be a representation of the methodology concept. When choosing a combination of tools factors to consider are web site architecture, platform, target audience, business requirements and objectives. The more suitable tools must be chosen according to those factors.
Depending on the scope and budget of the company alternative methods of accessibility might be used. This paper does not specifically recommend a particular tool. However, the tools might be chosen according to the organization’s requirements, resources, and intended audience. This in turn might be more efficient for accessibility editing. This paper only aims to suggest a methodology or model that can be used with a combination of tools depending on the organization’s needs. Thus, a company can achieve better results when tools that are more suitable for the project requirements are used.

3. Background on Accessibility
Accessibility: the degree to which a product, device, service or environment is available to as many people as possible.
Accessibility Addresses:
Visual: Visual impairments, blindness, color blindness. Motor: mobility impairments. Auditory: Hearing impairments. Seizures: photo epileptic. Cognitive: learning disabilities
Components of Accessibility: Web content, web browsers, Assistive technologies: screen readers, dictation software, user’s knowledge, and developers, authoring tools, evaluation tools and web accessibility standards.
The estimated number of individuals with a disability is approximately 15% of the world population [World Health

4. Accessibility Evaluation
Assessing conformance to accessibility standards. Accessibility testing aims to find issues with accessibility and provide results to fix those errors.
Evaluation Methodologies
Automatic: Using only web accessibility reporting tools
Manual: Performing accessibility testing using human judgement. Used case scenarios.
Crowdsourcing: It is referred to the act of outsourcing tasks, traditionally performed by an employee or contractor, to an undefined, large group of people or community (a crowd), through an open call.

5. Accessibility Requirements
Scalability
Human testing
Degree of severity
Catching the most common errors
Quality of results
Different tools yield different results
over reliance on automatic tools
Content / alternative text - Appropriate alternative text in images

6. Motivations Using existing tools more efficiently
Closing the gap between automatic tools and human evaluation
Accessibility is a right for everyone
Why Accessibility?
Accessibility typically is an after thought
Accessibility tools are not completely accurate
Human verification is always required
Identification of a need for a methodology

7. Methodologies and Tools
Types of Accessibility Integration
Design Stages:
Automatic Tools only
Combination of Crowdsourcing and automatic tools.

8. Pros and Cons of Automatic Testing
Table 4.1: Even Grounds, Accessibility Consulting.
Manual Testing
Automated Testing
Slower
Faster
Completely accurate
Accuracy depends on errors and testing standards
Easier to miss a link
Guaranteed check of all links
Easier to find content which depends on user input
Content can be missed when automated user input is used

9. Automatic Tools
Accessibility tools are applications or web site services that allow running a computer assessment in a web page code to identify accessibility issues.
Examples:
508 Checker by Formstack
AATT (Automated Accessibility Testing Tool) by PayPal
Accessibility Checker by CKSource
Accessibility color wheel by Giacomo Mazzocato
Accessibility Developer Tools by Google Accessibility
Accessibility Management Platform (AMP) by SSB BART Group

10. Advantages Scalability allows evaluating a large number of pages.
Degree of severity: at a glance a test can determine if the site needs to be fixed or built from scratch.
Testing against common accessibility standards
Future automatic tools will be more efficient
Identifying the big issues

11. Disadvantages Using automatic tools is not complete
Different tools yield different results
High sensitivity
Issues with updates
manual testing is always required
over reliance on automatic tools
False positives: appropriate use of color, appropriate content in alternative text in images.
The major disadvantage of using solely an automatic tool is the lack of human judgment

12. Crowdsourcing Crowdsourcing tools examples
Serotek C-Saw Community-Supported Accessible Web.
Web Visum
Firefox plug-in
Community driven tagging and page enhancements.
Automated and instant CAPTCHA image solving
IBM Social Accessibility Project
Provided a tool to facilitate collaboration between “screen reader users” and “volunteers”
Included rewards for volunteers (e.g., rankings)
FixTheWeb.net
Greasemonkey
Firefox extension that allows users to run scripts that modify HTML web page content every time a specified page is loaded
User script repository: userscripts.org
Scripts tagged “accessibility”:
My scripts:

13. What is UTT?
The User Testing Tool (UTT) is an open source application that combines user testing and automated testing to check accessibility of websites [4]. This application integrates with an existing automatic testing tool to check against WCAG 2.0 guidelines.
UTT is an interface prototype that integrates with an existing automatic tool that presents questions that require human verification such as weather an image contains appropriate description or if there is appropriate use of color The backend uses the Node.js platform, and the frontend uses the Bootstrap framework for the user interface, and the Backbone.js fra mework for handling interactivity. UTT is built based on an iframe technique that promises a path for development.

14. Methodology Proposal
I propose a cross methodology for web site evaluation and testing to leverage the advantages of both types of tools to enhance, improve, and reinforce the process of web site accessibility evaluation.
Current state of automatic tools is not complete, efficient, and reliable because both aspects of the tools need to be combined to maximize the capabilities of the tools. There always exists the need for expert evaluation to verify results against accessibility guidelines. By using a combination of both techniques one can improve quality of reporting and maximize advantage of automatic tools in terms of scalability and improve accuracy by human judgment.
The proposed model does not intend to be the all-in-one method for accessibility evaluation. Many sites and organizations have different requirements for their intended clients and stake holders. It is the intention of this project to suggest a set of procedures from this methodology that might serve as a guideline or model to be adopted according to the organization’s needs and requirements. Thus, this model presents a scenario in which a set of tools are used but it is important to realize that different tools would be more appropriate with particular web site platforms and this should be taken in consideration when applying this methodology. Thus, this project is based on the methodology for testing and guidelines rather than the tools themselves to improve accessibility evaluation.

15. Model - Methodology Process
Planning
Development Plan
Define Timeline, Budget, Scope, Quality
Outline names, dates, milestones, and deliverables.

16. Organization Delegating tasks Names, task, responsibility Timelines
Milestones

17. Execution or implementation
Verifying plan execution and implementation
Choose alternative course of action if needed

18. Data Processing Periodically Manual Data Analysis
Collecting reports from automatic tool and UTT
Compile reports into one file
Determine discrepancy

19. Verification and Presentation of Results
* Verifying Results
Findings and presenting results

20. What are the issues that have to be addressed but ignored now?
Dynamic content
Rate of adoption
Security
Quality

21. Conclusion
This methodology aims to be a represented set of procedures to conduct accessibility evaluation depending on the organization requirements, resources, and constrains. Due to the nature of the site complexity, architecture, and platform the developers and managers should choose the appropriate tools that are more suitable for their site web sites as well as the level of accessibility required. This set of procedures is not meant to be all inclusive but rather a model that might serve as a starting point or guideline to provide a conceptual procedure that can be applied based on the different organization requirements.

22. References [1] World report on disability
[2] V. L. Hanson and J. T. Richards. Progress on website accessibility? ACM TWEB, 7(1):2:1–2:30, Mar
[3] A lightweight methodology to improve web accessibility_ p30-greeff'
[4] Beyond Specifications: Towards a Practical Methodology for Evaluating Web Accessibility
[5] A proposed architecture for integrating accessibility test tools
[5] An approach to the integration of accessibility requirements into a user interface development method  
[6] Benchmarking web accessibility evaluation tools: measuring the harm of sole reliance on automated tests
[7] User testing tool : towards a tool for crowdsource-enabled accessibility evaluation of websites
[8[ Table 4.2 Table: differences of tools
[9[ UTT
[10] Web content accessibility guidelines
[10] "CWE - Common Weakness Enumeration". Cwe.mitre.org. Retrieved