HS logo
back to work
case study · Sep 2025 – Dec 2025

VeriFlow

A QA automation copilot that turns Azure DevOps requirements (or manual user stories) into runnable Playwright browser tests, executes them, and presents structured, reviewable results.

Node.jsExpressPlaywrightClaudeDockerAzure DevOps
system diagramgithub repo

The problem

Requirements live in user stories; automation lives in separate test code. VeriFlow closes that gap by generating executable browser tests directly from requirement text and making the outcomes easy to review.

What it does

  • Imports Azure DevOps work items (WIQL) to pull user stories and acceptance criteria.
  • Generates a test plan, test cases, and a runnable Playwright spec from the requirements.
  • Runs the suite against a target URL and reports totals, durations, and per-test pass/fail/skip.
  • Plain-English run summaries (with a deterministic fallback), persisted history, and scheduled reruns.

how it fits together

VeriFlow · architecture

Requirements → sandboxed, self-healing browser tests

Turns user stories into runnable Playwright tests, executes them safely, and reports results you can trust.

  1. Requirement intake
    Azure DevOps work item (WIQL) or a manual user story.
  2. Claude test generation
    Produces a test plan, test cases, and a runnable Playwright spec from acceptance criteria.
  3. Sandboxed execution
    Docker (dropped caps, no secrets) or hardened process fallback. LLM-generated code is treated as untrusted and isolated from the host.
  4. Playwright JSON reporter
    Authoritative pass / fail / skip totals, not brittle stdout scraping.
  5. AI self-heal (on selector failure)
    Capture live DOM → Claude proposes one replacement selector → rewrite spec → re-run once (bounded).
  6. Structured report + traceability matrix
    Each acceptance criterion mapped to its covering test(s) + pass/fail.
  7. Run summary + persisted history
    Plain-English summary (with fallback); refresh-safe history; scheduled reruns.
Node.js · Express · Playwright · Claude · Docker · Azure DevOps · GitHub Actions

product screens

Test cases generated from a user story and its acceptance criteria.
The runnable Playwright spec Claude generated for the requirement.
Execution results: authoritative pass/fail totals, the sandbox mode, and a requirement → test traceability matrix.

key engineering decisions

Trustworthy result reporting

Success derived from Playwright's JSON reporter (real pass/fail/skip totals), not scraped stdout, which fixed a false-positive where zero-test runs reported as "passed."

Resilient run pipeline

Parses the structured report on both success and non-zero-exit paths, and persists history for refresh-safe review.

results & outcomes

  • Requirements become executable tests; QA becomes repeatable and verifiable.
  • Runs untrusted, LLM-generated code under five Docker hardening controls with secrets stripped from the test process and no shell, so a generated test can't reach the host or leak a key.
  • Eliminated a false-positive that reported zero-test runs as passing by reading Playwright's JSON reporter instead of scraping stdout.
  • Plain-English summaries, persisted history, and scheduled reruns make results easy to review over time.
back to all work