Why Automated Accessibility Scores Miss Half the Problems

There is a number that shows up in accessibility tools, usually somewhere between zero and one hundred, and it exerts a strange pull on people. Teams set targets for it. Managers put it in dashboards. Agencies quote it in proposals. And almost nobody asks the obvious question, which is: a hundred percent of what?

The honest answer is: a hundred percent of the things a machine can decide. That turns out to be a minority of the things that make a site accessible or not. I want to explain why this is, because it’s not a flaw in any particular tool. It’s a property of the problem itself, and once you see it you’ll never read an accessibility score the same way again.

Some accessibility requirements are decidable by inspection. Does this image element have an alt attribute? Does this form input have an associated label? Is the contrast between this text color and this background color above the threshold? These are yes-or-no questions about the markup and the pixels, and a program can answer them perfectly. This is the territory where automated tools live, and within it they are better than humans: faster, more consistent, and never bored.

But most of what the accessibility guidelines actually ask for is not decidable this way, because it depends on meaning. The guidelines say alt text must convey the purpose of the image. A program can see that alt text exists; it cannot know whether “our team celebrating the launch” is accurate or whether the image is actually a chart whose data the alt text omits entirely. The guidelines say headings must describe the sections they introduce. A program can see that you used a heading element; it cannot know whether the heading is a lie. The guidelines say link text should make sense out of context. A program can flag “click here,” because that phrase is famous, but it cannot flag “learn more” repeated nine times pointing at nine different pages, each announcement identical, which for a screen reader user turns the page into a hallway of unlabeled doors.

The pattern here is that machines verify form and humans verify meaning, and accessibility is mostly about meaning. When researchers and practitioners run automated tools against pages with known, cataloged accessibility failures, the tools reliably find the mechanical ones and reliably miss the semantic ones. The exact fraction varies by tool and by page, but the shape of the result never changes: a large share of real barriers pass every automated check. So a score of one hundred does not mean your site is accessible. It means your site contains none of the errors that are cheap to detect. Those are different claims, and the gap between them is exactly where users get hurt.

There is a second, subtler problem: scores can be gamed without anyone intending to game them. If your team is rewarded for the number going up, the fastest route is to fix the detectable issues, and the fastest fix for a detectable issue is often the minimal one. Missing alt text? Add alt text, any alt text. Missing label? Add a label, even a useless one. The score climbs. The site, for an actual screen reader user, is no better and sometimes worse, because a wrong description is more misleading than a missing one. A missing description at least announces itself as a gap. A wrong one is confidently false. The score cannot tell these apart, so optimizing the score can quietly optimize for confident falsehood.

None of this is an argument against measurement. I am, if anything, more pro-measurement than most people; I built GazeSite because I think sites should be audited continuously rather than once a year. But when we grade accessibility, the grade is a summary of specific findings, each with the issue and the fix spelled out, precisely because a bare number invites the wrong response. The right way to use an automated score is as a floor and a trend. As a floor: if the score is low, you certainly have problems, and they’re the cheap kind, so fix them. As a trend: if the score drops after a deploy, something regressed, and automation caught it for free at three in the morning. What the score cannot be is a ceiling, or a certificate, or a substitute for ever watching a person use your site.

The business framing makes the stakes concrete. The visitors who hit the undetectable barriers are real customers with real money, and when they fail, they fail silently. No error appears in your logs when someone gives up on a form whose failure message was never announced to their screen reader. The transaction just doesn’t happen. If you steer only by the automated score, you are steering by an instrument that is blind to precisely these losses, and the better your score gets, the more your remaining problems are concentrated in the invisible half.

So my advice is to hold two ideas at once. Run the automated checks constantly, because the mechanical bugs are embarrassing and free to find. And treat the resulting number the way a doctor treats a temperature reading: useful, objective, worth tracking, and nowhere near a diagnosis. The diagnosis still requires looking at the patient.