Agentic Visual Testing: Judging and Harnesses

A failing check is not always a regression. This page covers the two things that turn a raw diff into a decision: judging, meaning is this change real, intentional, or noise, and harnesses, meaning driving the page so the right thing gets screenshotted in the first place.

The judging model

The heuristic pipeline puts one of four labels on every failing entry:

Only ambiguous reaches an agent. That is the point: asking a model to rule on every diff, including the obvious ones, is how you get verdicts that contradict each other between runs.

For ambiguous, --judge host writes a JudgmentRequest to .blazediff/judgments/<id>/request.json containing:

• regions[], with bounding boxes, pixel counts and change types per region
• paths.locator (locator.png), a ~400px overview with regions outlined in red
• paths.tiles (regions.png), a vertical stack of [baseline | actual] pairs
• paths.{baseline,actual,diff}, the full-page PNGs, as a fallback

The host agent writes its verdict to .blazediff/judgments/<id>/verdict.json:

{
  "id": "agent",
  "verdict": {
    "label": "intentional-likely",
    "headline": "Em-dash replaced with hyphen in copy",
    "rationale": ["region tile shows only typographic substitution"],
    "action": "rewrite-if-intended"
  },
  "rationale": "Full paragraph explanation...",
  "confidence": 0.95
}

Merge verdicts into the report without re-screenshotting anything:

blazediff-agent check --apply-judgments --json

Accept an intentional change by re-baselining the entry. Mask, viewport and waitFor are preserved; only the PNG is regenerated:

blazediff-agent rewrite agent --json       # by id
blazediff-agent rewrite --failed --json    # all failures from the last check

Why agents fail at visual review

Handing a coding agent two screenshots and asking “what changed” fails reliably. The failures have specific causes, and the judgment request is shaped around each one.

The rule underneath all of it: describe each side separately, then diff the descriptions. A model asked to compare two images in one step fills gaps with what it expects to see. A model asked what a single crop contains is answering a much easier question, and the comparison afterwards is arithmetic. The --judge local backend follows the same shape: Moondream describes, a deterministic word diff runs, then Qwen classifies the result.

If your agent is still returning bad verdicts, check in this order:

1. Is it reading regions.png and locator.png, or reaching straight for the full-page PNGs? The full pages are a fallback, not the input.
2. Is the entry actually ambiguous? A regression-likely entry does not need a verdict, it needs a look.
3. Is the diff real, or is the region non-deterministic? Masking a flake is correct. Asking an agent to rule on a spinner is not.

Harnesses

A harness is a pluggable ESM script in .blazediff/harnesses/<name>.js, attached to an entry through its harnesses: [{ name, params? }] list. Login is one kind of harness. Anything that drives the page before or around a screenshot is a harness. There are two phases:

• setup runs before navigation, to establish a session such as a login.
• interact, the default, runs after the base screenshot. It drives the page and can emit extra named screenshots through screenshot(name), each becoming its own baseline entry <entry>__<name>.

Interaction harness

// .blazediff/harnesses/weather-menu.js
/** @type {import("@blazediff/agent").Harness} */
export default {
  async run({ page, screenshot }) {
    await page.getByRole("button", { name: "More options" }).click();
    await screenshot("menu"); // -> baseline "weather__menu"
  },
};

{ "id": "weather", "url": "/weather", "harnesses": ["weather-menu"] }

Login harness

Routes behind a login capture through a setup harness. Credentials live in environment variables, never in the harness file, the manifest, or an LLM’s context.

/** @type {import("@blazediff/agent").Harness<{ persona?: string }>} */
export default {
  phase: "setup",
  async run({ page, params }) {
    const upper = (params.persona ?? "default").toUpperCase().replace(/[^A-Z0-9]/g, "_");
    const email = process.env[`BLAZEDIFF_AUTH_${upper}_EMAIL`];
    const password = process.env[`BLAZEDIFF_AUTH_${upper}_PASSWORD`];
    if (!email || !password) throw new Error(`missing BLAZEDIFF_AUTH_${upper}_*`);
    await page.goto("http://127.0.0.1:3000/login");
    await page.locator('input[name="email"]').fill(email);
    await page.locator('input[name="password"]').fill(password);
    await Promise.all([
      page.waitForURL((u) => !u.pathname.startsWith("/login")),
      page.getByRole("button", { name: /sign in|log in/i }).click(),
    ]);
  },
};

Attach it per entry, and put the credentials in .blazediff/.env, which is gitignored automatically:

{ "id": "dashboard", "url": "/dashboard",
  "harnesses": [{ "name": "auth", "params": { "persona": "default" } }] }

For OAuth, SSO, magic links, MFA or captcha, record the session interactively instead: blazediff-agent auth init --persona default --login-url http://127.0.0.1:3000/login.

Masking flaky regions

When a diff is noise-likely, or a real-looking diff turns out to come from something non-deterministic, mask it rather than re-baselining. A re-baseline just resets the clock on a flake; a mask removes it.

Mask auto-cycling animations, third-party iframes, timestamps, per-session randomness and personalization noise. Do not mask real content that happens to change often. That is the change you want caught.

The agent always masks any element matching [data-blazediff-agent-mask], with no manifest change needed. Put it on a shared component and it applies on every route:

<div data-blazediff-agent-mask="report-carousel">...</div>

When you cannot edit the source, such as a third-party embed, fall back to a per-entry CSS selector. The mask list replaces the existing one, so include every selector you want to keep:

cat <<'EOF' | blazediff-agent capture --stdin --mode baseline --json
[
  {"id": "examples-vanilla", "url": "/docs/ui-components/vanilla", "mask": ["iframe"]}
]
EOF

Next

• Cross-OS false positives when the same page renders differently on macOS and Linux CI
• Anti-aliasing and 1px shifts when text edges are the source of the churn
• Every command and flag: @blazediff/agent reference