Automated tools vs manual audit: what each one detects in accessibility

10 min read J. Vicente Training

What automated accessibility tools detect and what only expert manual review finds, criterion by criterion in WCAG 2.2, and how to combine the two.


Why running a validator is not enough

Running an automated tool over the website is the first step for almost any team that cares about accessibility. It is fast, free in many cases and returns a list of errors that looks very objective. The problem arises when that report is taken as the final verdict: a page can come out “clean” in the validator and still be impossible to use with a keyboard or a screen reader.

The W3C itself says so plainly in its guide to choosing evaluation tools: tools cannot determine accessibility, they can only assist in doing so, they cannot check every aspect automatically and human judgement is required. It also warns that they sometimes produce false or misleading results.

In this article we are not going to repeat lists of tools. We focus on a more useful question: which part of each WCAG 2.2 criterion a machine can check and which part needs an expert, and what the W3C and Spanish legislation say about it.


What the W3C and the legislation say

Combining automated and manual testing is not a consultants’ preference: it is written into the reference documents. These are the most relevant:

Source What it establishes about tools and manual review
W3C, “Selecting Web Accessibility Evaluation Tools” (updated 13 May 2024) Tools cannot determine accessibility, they can only assist; they cannot check everything automatically and human judgement is required.
W3C, WCAG-EM 2.0 (Group Note of 23 July 2026) Most accessibility checks are not fully automatable; tools mainly help to explore and to select the sample, and also during manual evaluation.
Royal Decree 1112/2018 (Spanish Real Decreto 1112/2018), Article 17.3 Accessibility reviews must cover all requirements and combine aspects of automated review with aspects of expert manual review.
Commission Implementing Decision (EU) 2018/1524 Distinguishes simplified monitoring, to detect non-compliance and carried out as far as reasonably possible with automated tests, from in-depth monitoring to verify conformance.

The new version of the W3C evaluation methodology, WCAG-EM 2.0, was published on 23 July 2026. It builds on version 1.0 from 2014, is designed for WCAG 2 and uses WCAG 2.2 as its reference. It extends its scope beyond websites to cover web applications, native and hybrid applications, kiosks and documents. Its five steps remain: define the scope, explore the product, select a representative sample, evaluate it and report the findings.


What an automated tool does well

Automated tools are essential, provided they are used for what they are good at. Their strength is objective checks, those with a yes-or-no answer that can be read in the code:

• Presence or absence of something mandatory: alternative text on an image, an accessible name on a button or form field, a page title or the document’s language attribute.

• Calculable values, such as the contrast ratio between a text and its background when both colours are defined in the code.

• Technical consistency: broken references between cells and headers in a table, a page frame without a name or a setting that prevents zooming the page on mobile.

• Scale and repetition: reviewing hundreds of pages or running the same tests on every version of the site, something unfeasible by hand.

That is why WCAG-EM 2.0 recognises that large-scale evaluations rely mainly on automated tools, although it warns that they do not usually reach the qualitative depth the methodology aims for.


Where it falls short: human judgement

Most WCAG 2.2 criteria do not ask whether something exists, but whether it serves its purpose. Criterion 1.1.1 makes this clear: all non-text content must have a text alternative that serves the equivalent purpose. A machine can know that there is a text; it cannot know whether that text properly describes a sales chart or whether the image was decorative and should have been ignored.

Among the W3C ACT test rules, some check whether something exists and others check whether it is appropriate. A few pairs illustrate this very well:

Objective check (usually automated) Check that requires judgement
Image has non-empty accessible name Image accessible name is descriptive
HTML page has non-empty title HTML page title is descriptive (proposed rule)
Link has non-empty accessible name Link in context is descriptive (proposed rule)
Form field has non-empty accessible name Form field label is descriptive (proposed rule)

There are also barriers that only appear when you use the page: focus getting lost when a modal window opens, a drop-down menu that cannot be closed with the keyboard, an error message that is not announced or a reading order that makes no sense. To find them you have to navigate with a keyboard, with a screen reader, with zoom and at different screen sizes.


Criterion by criterion: what each technique detects

The following table lists common WCAG 2.2 criteria and divides the work between the two techniques. It is indicative: what each specific tool detects varies. The criterion names are the official W3C ones.

Criterion (level) What a tool can check What needs manual review
1.1.1 Non-text Content (A) Images without alternative text Whether the text is equivalent and whether the image is decorative
1.2.2 Captions (Prerecorded) (A) Signs of video without a caption track Whether captions exist, are complete and are synchronised
1.3.1 Info and Relationships (A) Tables with broken header references Whether what looks like a heading, a list or a table is also one in the code
1.4.3 Contrast (Minimum) (AA) Text contrast with colours defined in the code Text over images or gradients and states such as mouse hover
2.1.1 Keyboard (A) Little: some technical indications Going through the whole page and its components using only the keyboard
2.4.2 Page Titled (A) That the page has a non-empty title That the title describes the topic or purpose
2.4.4 Link Purpose (In Context) (A) Links without an accessible name Whether the link text explains where it leads
2.4.7 Focus Visible (AA) Very limited Seeing the focus indicator on every interactive element
2.4.11 Focus Not Obscured (Minimum) (AA) Very limited That sticky headers or cookie notices do not hide the focus
2.5.8 Target Size (Minimum) (AA) Measuring the size of some controls Applying the exceptions, such as spacing or links within a text
3.1.1 Language of Page (A) That the language attribute exists and has a valid code That the declared language is the actual one
3.3.8 Accessible Authentication (Minimum) (AA) Almost nothing That logging in does not depend on cognitive tests without an alternative
4.1.2 Name, Role, Value (A) Buttons and fields without an accessible name That custom components properly announce their role and state

The table is easy to read: the automated part usually covers the presence of elements; the manual part, their quality and their behaviour. Bear in mind too that criterion 4.1.1 Parsing no longer exists in WCAG 2.2: it is listed as obsolete and removed, although anyone who must meet WCAG 2.0 or 2.1 due to a regulatory requirement may need to keep checking it.


The public sector case: simplified and in-depth monitoring

Directive (EU) 2016/2102 requires Member States to check the accessibility of public sector websites and applications periodically using a common methodology. That methodology was set by Commission Implementing Decision (EU) 2018/1524, which combines both approaches:

• Simplified monitoring: used to detect cases of non-compliance and carried out as far as reasonably possible with automated tests. It applies only to websites, on large samples.

• In-depth monitoring: used to verify conformance and applied to a smaller sample of websites and applications.

In Spain, the 2022 methodology of the Web Accessibility Observatory (Observatorio de Accesibilidad Web) for in-depth monitoring explains it as follows: simplified monitoring automatically analyses a subset of requirements, while in-depth monitoring analyses all requirements, manually. The same document acknowledges that there are elements that could be improved which cannot be detected by automated analysis.

For the bodies concerned, Article 17 of Royal Decree 1112/2018 goes further: reviews of their own websites and applications must cover all requirements and combine automated review with expert manual review. A report generated with a tool alone does not meet that requirement for expert manual review. Remember that this article is for information only and does not replace legal advice on your specific case.


A figure to put tools in context

In February 2017, the accessibility team at the UK Government Digital Service published a very illustrative test. They created a page with 143 accessibility failures grouped into 19 categories and analysed it with 10 automated tools.

GDS test result (2017) Percentage of barriers detected
Most effective tool (also counting manual review warnings) 41%
Least effective of the tools cited 17%
All tools combined 71%

Tools have improved since then and these figures should not be read as a current percentage. What still holds is the GDS conclusion: tools are most effective when combined with manual testing, and the recommended approach is to bring together automated tools, manual checking, an accessibility audit and user testing.


How to combine both techniques step by step

A realistic workflow does not pit tools against people: it puts them in order. This is a sequence you can adapt to your team:

1. Define the scope and the level you must meet (usually AA) and which parts of the website, application or documents are included.

2. Run a broad automated analysis to find obvious and repeated errors, and fix them first: they usually come from templates and shared components.

3. Select a representative sample: home pages, different templates, forms, complete processes such as registration or payment, and downloadable documents.

4. Review the sample by hand: keyboard, focus, screen reader, 200% zoom, contrast in states and images, alternative texts and error messages.

5. Test with people with disabilities whenever possible: WCAG-EM 2.0 strongly recommends it because they can help uncover barriers that expert review alone does not easily detect.

6. Build automated testing into development so that objective errors do not return with every update.

7. Repeat the review periodically, reusing part of the previous sample to compare results.

Common mistakes to avoid

• Taking a tool’s score as a compliance percentage.

• Accepting “check manually” warnings without checking them.

• Relying on accessibility widgets or overlays: they do not fix the code or guarantee compliance.

• Evaluating only the home page and forgetting processes and documents.


What you gain from an audit that combines both

The automated tool tells you where to look first; manual review tells you whether people can actually use your website, your application or your documents. Together they give you a complete diagnosis, a prioritised remediation plan and a solid basis for your accessibility statement.

Need Automated tool only Tool and expert manual review
Detect repeated errors quickly Yes Yes
Assess the quality of alternative texts and links No Yes
Check keyboard, focus and screen readers No Yes
Combine automated and expert manual review, as required by Article 17.3 of Royal Decree 1112/2018 No Yes

If you want to know what your validator is missing, the Tu Web Accesible digital accessibility audit combines automated analysis and expert manual review against the WCAG 2.2 criteria, and if you then need help fixing what is found, you can rely on our accessibility remediation service.


Sources

• W3C WAI: Selecting Web Accessibility Evaluation Tools (https://www.w3.org/WAI/test-evaluate/tools/selecting/)

• W3C: WCAG Evaluation Methodology (WCAG-EM) 2.0 (https://www.w3.org/TR/WCAG-EM/)

• W3C: Web Content Accessibility Guidelines (WCAG) 2.2 (https://www.w3.org/TR/WCAG22/)

• W3C WAI: ACT Rules (Accessibility Conformance Testing) (https://www.w3.org/WAI/standards-guidelines/act/rules/)

• BOE (Spanish Official State Gazette): Royal Decree 1112/2018 on the accessibility of public sector websites and mobile applications (https://www.boe.es/buscar/act.php?id=BOE-A-2018-12699)

• EUR-Lex: Directive (EU) 2016/2102 (https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:32016L2102)

• EUR-Lex: Commission Implementing Decision (EU) 2018/1524 (https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:32018D1524)

• PAe (Spanish e-Government Portal): Methodology for in-depth monitoring of websites (2022) (https://administracionelectronica.gob.es/pae_Home/va/dam/jcr:d6819e20-361a-47a9-8659-8ba0d5cb9d4e/2022_Metodologia_OAW_SitiosWeb.pdf)

• GOV.UK Accessibility blog: testing automated tools (2017) (https://accessibility.blog.gov.uk/2017/02/24/what-we-found-when-we-tested-tools-on-the-worlds-least-accessible-webpage/)


Frequently Asked Questions

Everything you need to know.

No. The W3C states that evaluation tools cannot determine accessibility, they can only assist in doing so, and that human judgement is needed. A tool can confirm that it does not find certain objective errors, but it cannot assess whether an alternative text is appropriate, whether everything can be used with a keyboard or whether a screen reader properly announces changes.
Its Article 17 requires the bodies concerned to review accessibility at the design stage, before launch and periodically afterwards. Those reviews must cover all requirements and combine aspects of automated review with aspects of expert manual review, and their results are recorded in a review report.
Both come from Commission Implementing Decision (EU) 2018/1524. Simplified monitoring aims to detect non-compliance across large samples of websites and is carried out as far as reasonably possible with automated tests. In-depth monitoring verifies conformance in more detail on a smaller sample and, according to the 2022 Spanish methodology, analyses all requirements manually.
It is the W3C methodology for evaluating conformance with WCAG, published as a Group Note on 23 July 2026. It builds on version 1.0 from 2014, uses WCAG 2.2 as its reference and extends the scope to web applications, native and hybrid applications, kiosks and documents. It keeps five steps: scope, exploration, sample, evaluation and report.
No. A widget may offer settings such as font size or contrast, but it does not fix the page code: if a button has no accessible name or a form cannot be used with a keyboard, the problem is still there. Neither widgets nor automated tools guarantee compliance; the site itself needs to be reviewed and fixed.
The sensible approach is to build automated testing into every change to the site and to carry out manual reviews periodically and whenever important templates, components or processes change. Royal Decree 1112/2018 requires periodic reviews after launch, and WCAG-EM 2.0 proposes repeating the evaluation, reusing part of the sample to compare results.

Taller de Ideas para la Red S.L. ha recibido una ayuda de la Unión Europea con cargo al Programa de Andalucía FEDER 2021-2027 para la realización del proyecto “Weall Suite: Plataforma integral para la gestión de la accesibilidad digital técnica y normativa”.

Certifications and accreditations.

We have the certifications that endorse our experience in accessibility.

IAAP - International Association of Accessibility Professionals IAAP CERTIFIED
ISO 9001 - Sistema de Gestión de Calidad ISO 9001