A Cross Methodology for Accessibility Evaluation Using a Combination of Automatic Tools and Crowdsourcing Tools By Irwin Ramirez

Abstract This project suggests a methodology to make web site accessibility testing and evaluation more efficient, cost effective, scalable, reliable, and sustainable. This is done by using a combination of automatic testing tools and crowdsourcing tools. This technique achieves better results since it benefits from advantages of both types of tools such as the scalability and human judgement from automatic and crowdsourcing tools respectively. The procedures provided in this paper aim to be a representation of the methodology concept. When choosing a combination of tools factors to consider are web site architecture, platform, target audience, business requirements and objectives. The more suitable tools must be chosen according to those factors. Depending on the scope and budget of the company alternative methods of accessibility might be used. This paper does not specifically recommend a particular tool. However, the tools might be chosen according to the organization’s requirements, resources, and intended audience. This in turn might be more efficient for accessibility editing. This paper only aims to suggest a methodology or model that can be used with a combination of tools depending on the organization’s needs. Thus, a company can achieve better results when tools that are more suitable for the project requirements are used.

Background on Accessibility Accessibility: the degree to which a product, device, service or environment is available to as many people as possible. Accessibility Addresses: Visual: Visual impairments, blindness, color blindness. Motor: mobility impairments. Auditory: Hearing impairments. Seizures: photo epileptic. Cognitive: learning disabilities   Components of Accessibility: Web content, web browsers, Assistive technologies: screen readers, dictation software, user’s knowledge, and developers, authoring tools, evaluation tools and web accessibility standards. The estimated number of individuals with a disability is approximately 15% of the world population [World Health

Accessibility Evaluation Assessing conformance to accessibility standards. Accessibility testing aims to find issues with accessibility and provide results to fix those errors. Evaluation Methodologies Automatic: Using only web accessibility reporting tools Manual: Performing accessibility testing using human judgement. Used case scenarios. Crowdsourcing: It is referred to the act of outsourcing tasks, traditionally performed by an employee or contractor, to an undefined, large group of people or community (a crowd), through an open call.

Accessibility Requirements Scalability Human testing Degree of severity Catching the most common errors Quality of results Different tools yield different results over reliance on automatic tools Content / alternative text - Appropriate alternative text in images

Motivations Using existing tools more efficiently Closing the gap between automatic tools and human evaluation Accessibility is a right for everyone Why Accessibility? Accessibility typically is an after thought Accessibility tools are not completely accurate Human verification is always required Identification of a need for a methodology

Methodologies and Tools Types of Accessibility Integration Design Stages: Automatic Tools only Combination of Crowdsourcing and automatic tools.

Pros and Cons of Automatic Testing Table 4.1: Even Grounds, Accessibility Consulting. Manual Testing Automated Testing Slower Faster Completely accurate Accuracy depends on errors and testing standards Easier to miss a link Guaranteed check of all links Easier to find content which depends on user input Content can be missed when automated user input is used

Automatic Tools Accessibility tools are applications or web site services that allow running a computer assessment in a web page code to identify accessibility issues. Examples: 508 Checker by Formstack AATT (Automated Accessibility Testing Tool) by PayPal Accessibility Checker by CKSource Accessibility color wheel by Giacomo Mazzocato Accessibility Developer Tools by Google Accessibility Accessibility Management Platform (AMP) by SSB BART Group

Advantages Scalability allows evaluating a large number of pages. Degree of severity: at a glance a test can determine if the site needs to be fixed or built from scratch. Testing against common accessibility standards Future automatic tools will be more efficient Identifying the big issues

Disadvantages Using automatic tools is not complete Different tools yield different results High sensitivity Issues with updates manual testing is always required over reliance on automatic tools False positives: appropriate use of color, appropriate content in alternative text in images. The major disadvantage of using solely an automatic tool is the lack of human judgment

Crowdsourcing Crowdsourcing tools examples Serotek C-Saw Community-Supported Accessible Web. http://help.samobile.net/C-SAW.html Web Visum http://webvisum.com/ Firefox plug-in Community driven tagging and page enhancements. Automated and instant CAPTCHA image solving IBM Social Accessibility Project http://services.alphaworks.ibm.com/socialaccessibility Provided a tool to facilitate collaboration between “screen reader users” and “volunteers” Included rewards for volunteers (e.g., rankings) FixTheWeb.net Greasemonkey Firefox extension that allows users to run scripts that modify HTML web page content every time a specified page is loaded http://wiki.greasespot.net User script repository: userscripts.org Scripts tagged “accessibility”: http://userscripts.org/tags/accessibility My scripts: http://terrillthompson.com/gmscripts/

What is UTT? The User Testing Tool (UTT) is an open source application that combines user testing and automated testing to check accessibility of websites [4]. This application integrates with an existing automatic testing tool to check against WCAG 2.0 guidelines. UTT is an interface prototype that integrates with an existing automatic tool that presents questions that require human verification such as weather an image contains appropriate description or if there is appropriate use of color. The backend uses the Node.js platform, and the frontend uses the Bootstrap framework for the user interface, and the Backbone.js fra mework for handling interactivity. UTT is built based on an iframe technique that promises a path for development.

Methodology Proposal I propose a cross methodology for web site evaluation and testing to leverage the advantages of both types of tools to enhance, improve, and reinforce the process of web site accessibility evaluation. Current state of automatic tools is not complete, efficient, and reliable because both aspects of the tools need to be combined to maximize the capabilities of the tools. There always exists the need for expert evaluation to verify results against accessibility guidelines. By using a combination of both techniques one can improve quality of reporting and maximize advantage of automatic tools in terms of scalability and improve accuracy by human judgment.   The proposed model does not intend to be the all-in-one method for accessibility evaluation. Many sites and organizations have different requirements for their intended clients and stake holders. It is the intention of this project to suggest a set of procedures from this methodology that might serve as a guideline or model to be adopted according to the organization’s needs and requirements. Thus, this model presents a scenario in which a set of tools are used but it is important to realize that different tools would be more appropriate with particular web site platforms and this should be taken in consideration when applying this methodology. Thus, this project is based on the methodology for testing and guidelines rather than the tools themselves to improve accessibility evaluation.

Model - Methodology Process Planning Development Plan Define Timeline, Budget, Scope, Quality   Outline names, dates, milestones, and deliverables.

Organization Delegating tasks Names, task, responsibility Timelines Milestones

Execution or implementation Verifying plan execution and implementation Choose alternative course of action if needed

Data Processing Periodically Manual Data Analysis Collecting reports from automatic tool and UTT Compile reports into one file Determine discrepancy

Verification and Presentation of Results * Verifying Results Findings and presenting results

What are the issues that have to be addressed but ignored now? Dynamic content Rate of adoption Security Quality

Conclusion This methodology aims to be a represented set of procedures to conduct accessibility evaluation depending on the organization requirements, resources, and constrains. Due to the nature of the site complexity, architecture, and platform the developers and managers should choose the appropriate tools that are more suitable for their site web sites as well as the level of accessibility required. This set of procedures is not meant to be all inclusive but rather a model that might serve as a starting point or guideline to provide a conceptual procedure that can be applied based on the different organization requirements.

References [1] World report on disability http://www.who.int/disabilities/world_report/2011/report/en/ [2] V. L. Hanson and J. T. Richards. Progress on website accessibility? ACM TWEB, 7(1):2:1–2:30, Mar. 2013. [3] A lightweight methodology to improve web accessibility_ p30-greeff' [4] Beyond Specifications: Towards a Practical Methodology for Evaluating Web Accessibility http://uxpajournal.org/beyond-specifications-towards-a-practical-methodology-for-evaluating-web-accessibility/ [5] A proposed architecture for integrating accessibility test tools https://ezproxy.sunyit.edu/login?url=http://search.ebscohost.com/login.aspx?direct=true&db=cph&AN=17970603&site=ehost-live [5] An approach to the integration of accessibility requirements into a user interface development method   https://ezproxy.sunyit.edu/login?url=http://search.ebscohost.com/login.aspx?direct=true&db=aci&AN=95386394&site=ehost-live [6] Benchmarking web accessibility evaluation tools: measuring the harm of sole reliance on automated tests http://dl.acm.org/citation.cfm?id=2461124 [7] User testing tool : towards a tool for crowdsource-enabled accessibility evaluation of websites https://brage.bibsys.no/xmlui/handle/11250/136183?show=full [8[ Table 4.2 Table: differences of tools http://evengrounds.com/articles/manual-and-automated-web-accessibility-testing [9[ UTT [10] Web content accessibility guidelines https://www.w3.org/WAI/intro/wcag.php [10] "CWE - Common Weakness Enumeration". Cwe.mitre.org. Retrieved 2013-10-18.