Automated Scans or Human Audits? Pick Your Path

The WebAIM Million project has evaluated the home pages of the top 1,000,000 websites for seven years using the WAVE stand-alone API. Despite substantial budgets, many organizations still show persistent accessibility gaps. If you're a compliance officer, this should change how you view the results of automated scans.

Deciding Your Conformance Testing Approach

You need to determine how to structure your Conformance Testing program. The key question is: how much should you rely on automated tools like WAVE versus human evaluation?

The WebAIM Million offers an automated, home-page-only evaluation, useful for identifying large-scale trends but not for conformance determination. While WAVE can find missing alternative text, empty links, and low contrast, it can't assess the meaningfulness of alt text, logical focus order, or keyboard navigation. Many WCAG success criteria require human judgment.

Consider these three paths based on your risk profile, not just your budget.

Key Factors to Consider

Before choosing a path, answer these questions:

• Legal exposure. Are you a Place of Public Accommodation under ADA Title III, serving EU customers under the European Accessibility Act, or subject to ADA Title II and the DOJ Final Rule (2024)? Higher exposure necessitates human verification.
• Content velocity. How frequently do your pages, templates, and documents change? Rapid changes require continuous automated checks to catch issues.
• Digital footprint. Managing a few applications differs from handling a large WordPress multisite network or thousands of PDFs.
• Evidence needs. Do you need to produce an Accessibility Conformance Report for customers or procurement teams? Automated output alone won't suffice.
• Internal capability. Do you have staff with credentials like the Web Accessibility Specialist from the International Association of Accessibility Professionals, or will you need external evaluators?

Path A: Automated-First Monitoring

Choose this path if: your digital footprint is large, content changes frequently, and legal exposure is moderate.

Steps to take:

1. Implement automated scanning in the authoring workflow and on a regular schedule across production.
2. Use the output for triage. Fix detectable issues like missing form labels, empty buttons, and contrast failures under WCAG 1.4.3 Contrast (Minimum).
3. Monitor trends over time, similar to how the WebAIM Million tracks overall web trends. A rising error count on a template is an early warning.
4. Document that automated results are a preliminary screening, not a conformance statement.

Limitation: A clean scan doesn't prove WCAG conformance. If your Accessibility Governance documents suggest otherwise, you risk misrepresentation. Path A alone is only defensible with a plan to verify criteria that tools can't test.

Path B: Hybrid Testing with Human Verification

Choose this path if: you face significant litigation or regulatory exposure, publish transactional experiences, or need to support procurement claims.

Steps to take:

1. Maintain the automated layer from Path A for broad coverage.
2. Add manual Conformance Testing on critical user journeys: authentication, checkout, account management, and forms. Include keyboard-only evaluation and Screen Reader Testing with Assistive Technology.
3. Test against the standard your obligations require. Most organizations target WCAG 2.1 AA. The DOJ Final Rule (2024) directs Title II entities to WCAG 2.1 AA, and EN 301 549 guides European Accessibility Act compliance. Consider WCAG 2.2 criteria like Accessible Authentication for login flows.
4. Manually review ARIA usage. Automated tools can flag some misuse of aria-label, aria-labelledby, aria-describedby, and aria-hidden, but they can't ensure coherent announcements.
5. Record findings to support an Accessibility Conformance Report using the Voluntary Product Accessibility Template.

This is the default for most compliance officers. It applies human effort where tools fall short and automation where volume demands it.

Path C: Full Third-Party Audit with Governance Program

Choose this path if: you're responding to a demand letter or complaint, entering a regulated market, selling to federal buyers under Section 508 of the Rehabilitation Act, or need independent evidence for the board.

Steps to take:

1. Commission an independent expert evaluation scoped to specific pages, flows, and document types, with results mapped to individual WCAG success criteria.
2. Request a remediation roadmap with severity ratings, not just a defect list.
3. Establish formal Accessibility Governance: assign ownership, create policies, provide training, and set Accessible Procurement requirements to prevent new vendor failures.
4. Schedule re-evaluation after remediation and major releases.

Path C is the most costly but provides the strongest record. If you report Partial Conformance, specify which criteria fall short and when you plan to address them.

Summary Matrix

What the Data Should Change in Your Planning

The lesson from seven years of WebAIM Million results isn't that tools fail. It's that detectable errors persist Assistive Technology scale even though they're the easiest to find. If organizations with access to scanners still ship these defects, the issue is ownership and process, not detection.

Whichever path you choose, assign a named owner, set a re-test schedule, and document what your testing does and doesn't prove. This record will be essential if your accessibility claims are ever questioned.

ADA Title III