Manual or automated accessibility testing?

You can tell whether a site or app is accesible by checking it against Web Content Accessibility Guidelines (WCAG) 2.1 by either:

1. using automated testing tools or
2. carrying out a manual audit.

Automated evaluation can be carried out by anyone. There are a wide range of tools available:

• Accessibility insights
• aXe
• SortSite
• Tenon
• WAVE

Our preferred tool is Silktide - which in addition to accessibility testing also highlights usability issues on a site or app. You can take a look at the Silktide accessibility report for this site here to see how it works.

Accessibility testing tools run a set of scripts checking web content against certain criteria based on WCAG 2.1. For example, do <img> tags all have alternative text, or do the colours on screen meet certain contrast criteria.

Benefits

• Automated testing can be done by anyone with access to a testing tool
• Automated tests can return results in minutes for hundreds of pages
• Some of the tools are free or relatively low cost
• The tools can proactively check your site and notify you of issues that have been introduced either by new code or new content

Drawbacks

The main drawback is that many of the guidelines require human judgement in order to determine whether the guidelines has been met. For example, an automated tool can determine if an image has alternative text b- it can't determine if that alternative text is accurate or useful.

A pass from an automated tool simple determines that a site has passed all the checks that the tool is able to carry out - it does not follow that the site is actually WCAG 2.1 compliant.

Manual testing is carried out by an accessibility expert who checks a subset of your site or app against WCAG 2.1 criteria. Every page element and their underlying code is manually evaluated for issues related to:

• non-text content (images, audio, video),
• use of colour,
• keyboard accessibility,
• descriptive links,
• labels and instructions,
• correct HTML structure and many others.

Benefits

A manual audit is:

• more thorough
• can check all WCAG 2.1 criteria
• more reliable
• tailored to your site - with a report to match

Drawbacks

There are a few significant drawbacks to manual testing:

• it takes longer and is therefore more expensive
• it won't cover all pages on your site/app
• the value depends on how good your accessibility expert is and how well they know your system

You can work around these drawbacks by:

• choosing as representative sample of pages as possible
• working with accessibility experts who know your Content Management System

In our view this is really a false choice - automated and manual testing should be used to complement each other. The strengths of each can minimise the weaknesses inherent in the other approach. Our accessibility approach is:

• Use an automated tool to give a basic idea of the level of accessibility of a site
• Carry out a manual audit on preselected pages - we can work with you to pick relevant pages that cross content types and functional components
• Create a customised report with potential fixes for each of the outstanding issues

Once you have carried out all the recommended fixes we can schedule a retest and can help you create your accessibility statement to highlight the work that has been carried out.

Following an accessibility review we would recommend that that you consider regular usability testing with people with disabilities so that further improvements can be made to your site.

If this isn't possible then we would recommend that you carry out:

1. regular automated testing to ensure technical standards remain at the required level
2. training sessions for your developers and content editors to maintain the hard-won level of accessibility and prevent new issues appearing.

If you'd like to know more about our approach to accessibility and how we can help your organisation just contact us.