How Confident Is the Finding? Reading Between the Lines

When people get an audit report, they tend to read it the way they’d read a test score: as a verdict. But a report is not a verdict. It’s a collection of observations, and observations vary enormously in how confident you should be about them. Learning to read that variation is most of the skill of using a report well.

I noticed this while building GazeSite. The system takes one URL, captures a screenshot and the HTML on both desktop and mobile, and then six reviewers, each focused on one area, produce findings. Every finding comes out in the same shape: a score, a headline, a description of the issue, and a prescribed remedy. The uniform shape is useful, because it makes the report scannable. But it has a side effect. It makes a rock-solid finding and a judgment call look identical on the page. They aren’t.

Some findings are close to facts. If the audit says your images have no alt text, that’s not an opinion. The HTML either contains the attribute or it doesn’t. If it says your page title is missing, or your form inputs have no labels, or your page loaded four hundred requests, those are things you could verify yourself in thirty seconds with a browser’s developer tools. When a finding is grounded in something mechanical like this, you should treat it with high confidence. Arguing with it is a waste of time. Either fix it or consciously decide it doesn’t matter for your site, but don’t debate whether it’s true.

Other findings are judgments. “Your headline doesn’t communicate what the product does.” “The call to action is easy to miss.” “This paragraph is too dense for a first-time visitor.” These can be excellent observations, and often they’re the most valuable part of a report, because they’re the kind of thing a fresh pair of eyes catches and an owner never does. But they’re judgments, and judgments have error bars. The reviewer saw one rendering of one page at one moment. It didn’t watch real users. It doesn’t know that your audience is nuclear engineers who find your dense paragraph refreshingly precise.

So how do you tell the two kinds apart? The tell is usually in the remedy. Findings grounded in fact have remedies that are specific and almost boring: add this attribute, compress that image, set this header. Findings grounded in judgment have remedies that sound like advice: consider rewriting, try moving, test whether. The more the remedy reads like something you’d have to interpret, the more the finding itself is an interpretation.

There’s a second tell: convergence. GazeSite’s six reviewers look at the same page through different lenses, and when several of them independently bump into the same underlying problem, confidence should go up. If the readability reviewer says your text is hard to parse, that’s one data point. If the conversion reviewer also says visitors won’t understand the offer, and the clarity reviewer flags the same section, that’s not three problems. It’s one problem seen three times, and it’s probably real. Findings that show up alone, in one reviewer’s list, with a middling score, are the ones to hold loosely.

The score itself deserves some skepticism too, in a specific way. A score compresses a lot into one number, and compression loses information. What a score is good for is ordering: within one report, a finding scored severe is almost certainly more important than one scored minor. What it’s bad for is absolute meaning. A 6 on your site and a 6 on someone else’s site are not the same amount of broken. Use scores to decide what to look at first, not to decide how bad things are in some cosmic sense.

The mistake I see most often is the opposite of skepticism: people treat the weakest finding as representative and use it to dismiss the whole report. They find one judgment call they disagree with, decide the audit “doesn’t understand their business,” and ignore the three mechanical findings sitting right above it that are unambiguously true and unambiguously costing them. This is like throwing away a medical report because you disagree with the doctor’s advice about diet, when the report also says your blood pressure is 180.

The opposite mistake exists too. Some people implement every finding, in order, as if the report were a specification. This wastes effort on marginal items and, worse, it can hurt. A judgment-call finding implemented literally, against your better knowledge of your own audience, can make the site worse. The report doesn’t know everything. You know things it doesn’t. The right posture is collaboration, not obedience.

What I’d suggest, concretely, is this. Read the report twice. On the first pass, mark every finding you could verify mechanically, and just fix those; there’s no decision to make. On the second pass, take the judgment findings and ask of each one: does this match anything I’ve suspected, or anything a customer has ever said? A judgment finding that echoes something you’ve half-noticed yourself is probably right. One that contradicts everything you know about your users deserves a test, not a rewrite.

An audit is a conversation with a very observant stranger. The stranger notices things you’ve gone blind to, and that’s the value. But it’s still a stranger. Some of what it says is measurement, and some is opinion, and the report format flattens the difference. Your job as the reader is to unflatten it.