Agile and the automated testing of accessibility

Agile has changed a lot since it was first introduced last decade with frameworks like this scaled agile framework that cover all aspects of how agile interacts with the entire organization. © 2015 - All Rights Reserved

However, if you boil it down to its core, agile is about the simple iteration process that consists of a backlog that is groomed and broken-down into sprints (or iterations). These iterations focus on team communication and adaptation to learning. They generally contain feedback loops on a daily basis to facilitate collaboration and coordination and then feedback that improves the overall process. © 2015 - All Rights Reserved

Impact of Agile There have been a lot of changes that agile has caused, many of these are the tools that we use. These tools in turn have changed agile. Tools like GitHub have become the center of the coding portions of agile © 2015 - All Rights Reserved

Agile Tools There have been a lot of changes that agile has caused, many of these have resulted in new tools that help us to manage this agile process. These tools in turn have changed agile. Tools like GitHub and JIRA have become the center of the entire process with other tools integrating into these hubs to facilitate various portions of the process. © 2015 - All Rights Reserved

Shift Left Testing Catching a defect in the design and requirements phase can reduce cost to fix by up to 100:1 Software Defect Reduction Top 10 List, Barry Boehm and Victor R. Basili, January 2001 https://www.cs.umd.edu/projects/SoftEng/ESEG/papers/82.78.pdf The result of this agility is that testing has become more and more important. As we have learned more about software development, we have realized that catching and correcting defects early can save huge amounts of effort later in the process. © 2015 - All Rights Reserved

Rise of Continuous Lag between committing code and getting feedback allows damage to magnify Costs rise Velocity decreases Continuous Integration Continuous Deployment http://www.startuplessonslearned.com/2009/06/why-continuous-deployment.html This is not only true for defects themselves, but impacts can be on personal relationships too. The end result of late feedback is rising costs and lower value provided by development teams. This has led to the rise in continuous feedback. Continuous integration allows feedback on code quality and regression testing. Continuous deployment allows feedback on customer adoption of new features which feeds the backlog. © 2015 - All Rights Reserved

Automation This continuous revolution can only work if the repetitive portions of the process can be automated. So continuous integration has led to high levels of unit testing, high levels of automated integration, performance, security and acceptance testing. This has led to the widespread adoption of technologies like Selenium, Qunit, Jasmine, Mocha, Cucumber etc. © 2015 - All Rights Reserved

Accessibility Testing’s Image We are now asked to introduce accessibility into this environment and when we do, the need for manual testing in accessibility means that we are viewed as the “stick in the mud” in the process. How do we change this? If we cannot change this, we make it difficult for the widespread adoption of accessibility. © 2015 - All Rights Reserved

Aim of Software Testing Reduce the risk of the existence of a serious problem on a supported configuration to an acceptable level Define “serious problem” Define “supported configuration” Define “acceptable level” First we need to be clear about what software testing is about? What is the aim? The aim of software testing is not about eliminating defects altogether. There is no software product without defects. The aim is to optimize the quality of the product, given the resources available. To this end, we find it useful to define the aim of software testing as the reduction of the risk of the existence of a serious problem on a supported configuration to an acceptable level. This means that we need to define the term “serious problem”, we need to define what a “supported configuration” is and we need to define what an “acceptable level” of risk is. There is no single answer to any of these questions. It will depend on the resources of the team. The demographics of the users and the importance of the application. © 2015 - All Rights Reserved

So, even though this may differ for each specific organization, lets look at the general market in which we find ourselves and how this is changing over time. In 2009, Windows had the majority of the market with approximately 70%, Apple and Blackberry had about 6% and other operating systems about 20%. Fast forward to 2013 and the picture has changed dramatically. Android already had over 50% of the market with Windows about 25%, Apple 20% and the rest disappearingly small. © 2015 - All Rights Reserved

Android Fragmentation Looking just at the largest segment of that market: Android – what does it look like? Well it is tremendously fragmented. © 2015 - All Rights Reserved

Android Fragmentation We have seen 24,093 distinct devices download our app in the past few months. In our report last year we saw 18,796. In 2013 we saw 11,868 OpenSignal 2015 The android fragmentation can represented in numbers too as pointed out by a report by OpenSignal. © 2015 - All Rights Reserved

Combinations 4 operating systems, 2 supported versions 5 Assistive technologies 4 Browsers, 2 supported versions 3 Responsive breakpoints 216 combinations excluding device combinations If you then ignore devices, and simply look at the software components like the assistive technologies, browsers and operating systems and combine that with your own software variants – like responsive breakpoints, then you still arrive at a very large number of combinations. 216 combinations using the numbers presented here: 4 operating systems with support for 2 versions, 5 assistive technologies, 4 browsers with 2 supported versions and 3 responsive breakpoints. © 2015 - All Rights Reserved

Most of our customers have trouble testing a single browser-AT combination – so how do we get our arms around this problem? © 2015 - All Rights Reserved

Sparse Testing Matrix Browser/AT Operating Sytem Chrome KB IE KB Firefox KB Safari KB Dragon IE NVDA FF Jaws IE VO Safari TB FF Orca FF Operating Sytem Windows   OS X iOS Android Linux One of the ways to do this is to look at this set of combinations, and using the process of reducing risk to an acceptable level, we start to eliminate combinations that do not help us to reduce that risk by a significant amount. We then combine that with our user demographics. For example, we my have an application that is only accessed on mobile platforms. In this case, we can ignore all the desktop operating systems and this simplifies our picture significantly. Another example of reduction can be done in the area of keyboard testing. Most keyboard platforms are exactly the same and so it becomes unnecessary to test all of the keyboard combinations. Furthermore, we can reduce testing requirements by defining our supported configurations. It is totally acceptable, for example, to say that we will only support the most recent version of iOS – because 90% of iOS users upgrade their operating system immediately. When you do that, then you will arrive at a much smaller number of platforms. Furthermore, when you look at the functionality of your application, you will realize that you do not need to test all of the functionality in order to cover the areas of risk for (for example) keyboard testing. This methodology will result in a much smaller set of tests that need to be performed to get an acceptable level of risk. © 2015 - All Rights Reserved

The other key to dealing with the need for manual testing, is to leverage automation wherever possible. There are two types of automation when it comes to accessibility automation. This first type of automation is the create specific automation for testing accessibility. This is what I call “Accessibility Acceptance Automation” © 2015 - All Rights Reserved

Accessibility “Acceptance Automation” Here is an example of this acceptance testing automation. This is testing an ARIA menu. As you can see in the example, the test code is simulating key presses. For example the first action sets the focus into the menu. The next action simulates pressing the LEFT arrow key. This should cause the focus to shift to the other end of the menu. Then the test asserts that this is happening. The rest of the test carries on in the same vein. © 2015 - All Rights Reserved

Acceptance Automation Comes from the “requirements” Designers mark up their wireframes and comps Stories must contain specific accessibility acceptance criteria Can go to any level of detail Test the specific alt attributes on images Test the DOM order specified in the wireframe Etc. The important thing is that the tests, test the requirements created during the design and story writing process. It is best done by having these requirements specified by the UX designers, marked-up in the design comps and then translated into acceptance criteria and unit tests. The testing can be taken to any level of detail – even to the point of validating the value of alt attributes, the reading order of the DOM content etc. etc. Once again you use a process of analyzing the acceptable risk level versus the required effort tradeoff, as well as evaluating the immediate versus the future effort. © 2015 - All Rights Reserved

Open Source Rules Engine The second type of automated testing is “generic” automated testing. There are many different technologies that can be used for this. The one I will be showing is aXe – Deque’s open source rules engine. © 2015 - All Rights Reserved

aXe Manifesto Must have a zero false positive rate Must be lightweight and fast Must work in all modern browsers on all platforms Must be tested automatically There are a lot of things we have learned over the years in terms of what is required by developers when using an automation library. The primary concerns here are that when the library finds a problem, it better be a real problem. Developers hate it when their CI builds break. They therefore hate it when they break for no good reason. It is thi slearning that has led us to the aXe manifesto. © 2015 - All Rights Reserved

aXe-Core Small ~100KB Fast Intelligent Unlimited Reliable Secure No network connection, no new browser start Intelligent Supports cross domain iframes Turns document level rules on/off automatically Unlimited No throttling, no server limits Reliable No double-execution or missing CSS problems No network connectivity problems No server capacity problems Secure No server access across the wire No sending data to a server Mozilla Public License 2.0 Available via GitHub, npm and Bower © 2015 - All Rights Reserved

aXe Chrome extension © 2015 - All Rights Reserved

aXe Firefox extension © 2015 - All Rights Reserved

aXe extension Developers Testers Accessibility Experts Use when coding Use during functional testing Accessibility Experts Use for validation © 2015 - All Rights Reserved

aXe API Two major calls Getting started axe.getRules() – returns a list of the rules loaded axe.a11yCheck() – analyze a page or portion of a page Getting started Install package bower install –-save-dev axe-core Include library <script src=“(path_to)/axe.js”></script> Call function a11yCheck(target, options, callback); © 2015 - All Rights Reserved

aXe API demo © 2015 - All Rights Reserved

Integrations Java Selenium JavaScript Selenium (axe-webdriverjs) Grunt (grunt-axe-webdriver) Cucumber and R-Spec (axe-matchers) Firefox Dev Tools Chrome Dev Tools Ember-aXe (ember-axe) more… © 2015 - All Rights Reserved

Summary Manage your matrix Leverage automation Optimize for resources and risk Leverage automation Generic automation Specific automation Optimize for up-front vs. ongoing effort © 2015 - All Rights Reserved

Questions https://addons.mozilla.org/firefox/addon/axe-devtools/ Chrome: search Google for “chrome axe extension” © 2015 - All Rights Reserved