1. WAI-Tools Project Norwegian National Observatory 20 MAY 2019
Introduction.
Talk a bit about Difi and our tasks in this project.
Horizon 2020 Project
Grant Agreement

2. Responsible for enforcing the Web Accessibility Directive in Norway
Geir Sindre and I work for Difi – full name Agency for public management and egovernment – is a national agency, or directorate, in Norway. We are about 300 people, working on digitization, digital services, innovation, efficiency in the public sector, leadership, public procurement and universal design of ict.
The two of us work in the unit called the Norwegian Authority for universal design of ict. We are 14 people in total.
Norway has had regulations on universal design of ict since That is both websites, mobile applications and self service machines (vending machines, ticket machines etc.)
Monitor compliance through audits
Provide information and guidance, mainly through our website
Perform area surveillance and analyses to generate statistics and knowledge, so that we can target our efforts
Participate in developments of the legal framework, both nationally and internationally
Responsible for enforcement and monitoring of WAD in NO.

3. For websites and mobile applications, the requirements are 35 success criteria at levels A and AA in WCAG 2.0.
All public sector bodies and private businesses in Norway are obligated to follow the current regulations.
This means several hundred thousand solutions are relevant for testing and audits. And further down the line, monitoring websites and mobile applications as required by the web accessibility directive.

4. Key figures – audits & monitoring
Until now
After WAD
Audits: 30 (2015 – 2018)
Monitoring 2014: 304 websites
Monitoring 2018: 278 websites
Simplified: ~200 websites
In-depth: ~30 websites and mobile applications
Annual monitoring
Audits are part of our enforcement procedure, where it is possible for us to issue fines for non-compliance
Monitoring is a way to generate statistics, so that we can direct our efforts in raising awareness and audits.
2014: 304 websites
2018: 278 websites. Over single tests performed in total. 60 % compliance on average.
Other than test data, we also collect more meta data about the websites, website owners, sectors, business, geography and user accessibility needs.
We’ve had to make some tough priorities. Because of our current resource situation and methods, we are unable to both monitor and perform audits in a large scale simultaneously.
However, this is about to change with WAD. We have to monitor
About 200 websites in simplified monitoring
Websites and some mobile applications, 30 in total, in in-depth monitoring, where we have to test all of the requirements.
Not only testing, but also planning, sampling, data analysis and reporting. In the monitoring in 2018, testing the websites was about 40 % of the total workload.
This is to be performed annually

5. Current test method 51 test procedures 35 success criteria in WCAG 2.0
Some automation, mostly manual
Excel sheets
Web based format
We have a test method that we have developed ourselves.
When we developed the test procedures, our approach was to start with the standard. We worked our way thorugh WCAG, SC by SC.
We have read WCAG as a legal document. We need test methods that provide test results that tell us whether the web content conforms to the SC in WCAG or not.
For each SC, we similarly worked our way through the understanding article and all sufficient techniques and failures.
The method and test procedures are tried and tested. Th test procedures produce consistent and reliable data.
We have 51 test procedures that cover the 35 success criteria in WCAG 2.0 that are obligatory according to Norwegian law.
Each test procedure is a form, where the tester inputs data, and produces a predetermined outcome and documentation of compliance or non-compliance for each element tested.
We have some degree of automation in the test procedures today. We have automated tests to check the HTML syntax and contrast between text and its background. There is no secret that our.
The test procedures were developed in Excel, but we are gradually moving away from this format into our web based observatory – our testlab, that Geir Sindre has constructed.

6. Challenges
Where we are standing now, the directive faces us with some substantial challenges.
We have to monitor both websites and mobile applications to a much greater extent than we do today.
There are a number of new success criteria in WCAG 2.1
Not only when it comes to testing, but also in preparing monitoring, sampling, analysis and reporting.
We need to test more, at lower costs. Our current, manual methods produce high quality data, but they are simply too expensive
We are looking at automation. What goes on in WAI-Tools is a major part of that effort.
We want to use tools to a higher degree than we have until now.
This is where an interpretation of WCAG 2.1, development of test rules and implementation of those rules is of particular interest. That is why we are a part of the WAI-Tools project.

7. Mapping test rules Map test rules developed by the project
Document interpretation of WCAG 2.0/2.1
Compiled documentation
Basis for sampling of success criteria and test rules in monitoring
Develop a document detailing an interpretation and mapping of the test rules developed by the project to WCAG 2.0/2.1.
(This documentation could be added to existing Difi documentation, or be provided as separate documentation)
Database, input info
WCAG
Test rules developed by the project
Our own test rules
Metadata about user acc needs, test mode etc.
Get
Documentation: Compliled interpretation of WCAG – what is covered
Tool: Which test rules we can use to test certain SC and to cover user accessibility needs ++ **
Sikre oss at vi har eit effektivt verktøy for å enkelt kunne vedlikehalde metadata og status på testreglar slik at vi kan gje dei rette innspela på kva testreglar som bør bli utvikla. Denne leveransen inkluderer sjølve databasen og tilhøyrande dokumentasjon av denne.
Vi har behov for oversikt over kva testreglar som er implementert i verktøya hjå prosjektpartnarane SI, DQ og AMA/FCID og korleis dei ulike suksesskriteria i WCAG er dekka med testreglar.

8. Pilot monitoring Q2 2019: Prepare pilot
Q3-Q4 2019: Set up infrastructure
Q1 2020: Pilot simplified monitoring
Q2 2020: Pilot in-depth monitoring
Q3-Q4 2020: Document and write deliverable
Our biggest task in the project is the pilot monitoring.
Perform a pilot for simplified and in-depth monitoring, in line with WAD, on a limited set of websites. Test the websites using tools developed by DQ and SI that have adopted the test rules developed by the project. The pilot will cover the entire monitoring process.
This plan is still “under construction”. We are looking into different ways of solving this task the best way possible. For now, this is the outline.
(Read diagram, explain)
Prepare
Meetings with project partners
Discuss solutions
Get info on how tools work. Data format.
In addition summarize our experiences with monitoring up until now, to learn from experience.
Set up infrastructure
Our own testlab. Scale up.
Use tools developed by SI and DQ. To produce test data
Get ready to receive data
Pilot simplified
Sampling
Test rules. Automated tests.
Test
Analysis
Summary/report
Pilot in-depth
Similar to simplified, but testing to a larger extent.
Document the pilot and use of test rules and tools in a deliverable. **
Document the pilot. This includes a description of sampling approach, testing, analysis, data model and reporting. It will also include an evaluation of Difi’s previous experience with monitoring compliance with web accessibility regulations, conducted in 2018.
Pilot a simplified and in-depth monitoring, in line with WAD
Use tools developed by the project
Document the pilot (deliverable)

9. https://uu.difi.no/om-oss/english
Further Information
Thanks
Questions?
Horizon 2020 Project
Grant Agreement