We audited our own website. Here is what the scanner got wrong

Any company selling automated inspection faces the same credibility problem: if the scanner is so good, why does every serious report still need a human? The honest answer is that a detector's output and a defensible finding are different things — and the value is in the distance between them.

The cleanest way to show that distance is to walk through it on work we are allowed to embarrass: our own. What follows are four failure classes our engine review found in our own detector output during the adjudication programme — what the evidence actually supported, and the rule each mistake left behind.

One scoping note, because this article is about honesty: these are documented failure classes from our own review, reconstructed here with representative examples. They are not a transcript of a single scan, and the examples are illustrative rather than captured customer or prospect evidence.

A detector output is not a finding

First, the distinction the rest of this article rests on. Inside the engine, detectors propose. A proposal has a check, a location and captured evidence. Whether it becomes a finding a client sees — and what it is allowed to claim — is decided afterwards, by validation steps whose whole job is to argue with the detector.

That architecture exists because of afternoons like the ones below.

Finding one: the contradiction sitting in our own copy

A content check flagged the homepage for missing proof language — the pattern where a site makes strong claims and offers nothing that backs them. Fair check, wrong verdict: the evidence it captured for the finding included the homepage copy that answered it. The proof was on the page; the detector's pattern just did not reach it.

Finding vs. its own captured evidence–Not applicable

Proposed: 'Claims lack supporting proof' Captured in the same pass: the homepage line naming the specific verification behind the claim

The rule this produced: a finding must be checked against the evidence captured alongside it before it survives. A claim contradicted by its own capture is not downgraded — it is removed, and the removal is logged. We call this a contradiction check, and it runs on shared page facts so a detector cannot disagree with what another detector already established. It is implemented (src/lib/audit/contradiction.ts) and gated behind FLAG_CONTRADICTION_GATE.

Finding two: the protected endpoint reported as broken

The link audit probed every collected URL and reported failures. In the pile: authenticated routes answering 401 and 403 to an anonymous probe — refusals doing their job, filed alongside genuine dead links as if they were the same observation.

The full anatomy of this mistake — and the ten-class model that replaced it — is the subject of our pilot investigation. The short version as it applies here: the only honest verdict an unauthenticated probe can reach about a protected route is needs confirmation, and the report now says exactly that instead of "broken".

?Needs confirmationCannot be resolved from public evidence. Someone has to check.

Finding three: the noindexed login page "missing SEO"

An SEO check flagged our login page: thin content, no meaningful description. Technically true. Also completely irrelevant — the page carries noindex, which is not an oversight but the entire point. Auth pages are supposed to be invisible to search.

This one embarrasses the scoring model rather than the detector: the check was written for pages that want to rank, then applied to a page that had explicitly declined to. The rule it produced is an applicability layer — a check must first establish that it applies to a page before its result may count. A noindexed page failing an indexability check is not a defect; it is a category error. Implemented as src/lib/audit/applicability.ts, behind FLAG_APPLICABILITY.

Finding four: two detectors, two opinions about robots.txt

The subtlest of the four. One detector read our robots configuration and proposed a finding based on the file being absent; a different check, reading the same site in the same scan, described robots rules it had parsed. Both were written defensively, each did its own fetch, and their retries landed differently — so a single scan carried two contradictory beliefs about one fact.

Neither finding was individually wrong. The pair was indefensible: a report that says "robots.txt is missing" on one page and reasons about its rules on another has told the client it cannot be trusted on either.

The fix was structural, per RFC 9309's model of the file as a single source of truth: one canonical fact set per scan. Robots presence is established once, and every downstream check consumes that one answer (src/lib/audit/canonical-facts.ts, threaded under FLAG_SHARED_FACTS), so that contradiction becomes impossible by construction rather than discouraged by review.

What changed, in one table

Every rule below is implemented and covered by tests. Each is also behind its own feature flag, which is how a change that can move a customer's score is supposed to ship — written, tested, and released deliberately rather than switched on the day it lands.

We are deliberately not publishing which flags are on in which environment: that state changes without a deploy, and an article that asserts it would be wrong the moment it did.

One more change cuts across all four. The reviewer step that challenges findings is only as honest as the text it reads — so where context-aware copy rewrites a finding's wording for the reader, the reviewer is handed the neutral original, not the rewrite. A reviewer fed persuasive copy is a reviewer being lobbied.

What this does not prove

Corrections of this kind are also not a one-time cleanup. The engine's checks change, and every change can reopen an old mistake in new clothes — which is why the challenge step is part of the release path, not a launch-week ceremony.

The standard this sets for a client report

The reason to publish a list of our own scanner's mistakes is that the alternative pitch — trust the tool — is the thing this product exists to argue against. The standard a paying client should hold us to is the one this article demonstrates: every finding challenged against its own evidence before release, uncertainty labelled as uncertainty, and the mistakes that slip through treated as input to the next rule rather than something to quietly delete.

The sample report shows what survives that process — findings with their evidence attached, and the ones that did not make the cut, absent.