> ## Documentation Index
> Fetch the complete documentation index at: https://docs.reasonos.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Automated QA

> Plain-English QA tests executed by AI agents in a real browser — with per-step screenshots, session replay, and verdicts written as bug reports.

ReasonOS includes automated QA as a platform capability: tests written in plain
English, executed by AI agents that use your product the way a person does. A QA
run drives a real Chrome browser with real mouse and keyboard events — no
selectors, no test scripts — judges each assertion strictly, and leaves behind
evidence for every step: what the agent did, what it saw, why it passed or
failed.

There are two ways to use it:

* **The QA tab** — a full QA surface on your project's pages: author tests in
  plain English, organize them into suites and collections, run them against
  named environments on a fleet of QA workers, and read rich run reports with
  screenshots and session replay.
* **`ai_qa_test` targets** — agent-driven browser tests declared in your
  `BUILD.rbs` files and run with `rbs test`, right in your workspace.

Both sit alongside — not instead of — ordinary tests. `go_test`, `vitest_test`,
and friends verify your code deterministically; QA verifies the running product.

## How a QA run works

Each step of a test is routed to a specialized agent:

* **Action steps** are performed by an action agent: it reads the page (the full
  accessibility tree, page text, and a screenshot), decides the next moves, and
  performs them as real input events. It batches sensibly — a whole form is
  filled and submitted in one pass — and waits adaptively for the page to
  settle rather than sleeping.
* **Assertions** (`/verify`, `Expect:`) go to an assertion agent that judges
  strictly: an exact value you pin down must appear exactly as written —
  currency symbols, separators, and casing included. Garbled or truncated text
  never satisfies an assertion about that text. Assertions about how the page
  *looks* — layout, readability, broken assets — are judged by a design-review
  agent with a defect taxonomy.
* **Extractions** (`/extract`) read one value off the page, following your
  instructions to the letter ("digits only", "without the currency symbol"),
  and bind it to a variable later steps can reference.

Console errors and failed network requests are collected as evidence throughout
the run: the judging agents cite them in verdicts, and a claim like "no error is
shown" is checked against the console, not just the visible page.

A run stops at the first failed step; the remaining steps are recorded as
skipped, so every report reads as "N passed · 1 failed · M skipped" with the
failure front and center.

## Writing tests in plain English

A QA test is a small text file: a frontmatter block, then one step per line.
The step editor in the QA tab writes this format for you, and validates every
line as you type.

```md theme={null}
---
test: Incorrect price or currency for Germany shoppers
suite: checkout
platforms: web
persona: a first-time shopper in a hurry
---

1. Open the storefront at {base_url}, as a shopper in Germany.
2. Add the [Plan:Standard Plan] to the basket and go to checkout.
   Expect: the subtotal reads "€49,00" — euro symbol, comma decimal.
3. /extract order_total = the total amount shown in the order summary
4. /verify the currency stays EUR and the total reads {order_total}
```

### Step forms

| You write                                               | What it means                                                                   |
| ------------------------------------------------------- | ------------------------------------------------------------------------------- |
| `Open /checkout and add the first item`                 | An **action** — the agent performs it in the product                            |
| `/verify the subtotal reads "€49,00"`                   | An **assertion**, judged strictly against the current page                      |
| `/extract order_total = the total in the order summary` | Reads a value off the page and binds `{order_total}` for later steps            |
| `Expect: the cart badge shows "1"`                      | An assertion attached to the action step above it                               |
| `Only: ios, android`                                    | Restricts the step above to the listed platforms                                |
| `Save state as checkout-ready`                          | Snapshots the browser session (cookies + storage) for other tests to start from |

List markers (`1.`, `-`, `*`) are cosmetic, and lines starting with `#` are
comments.

### Frontmatter

| Key             | Meaning                                                                                                                    |
| --------------- | -------------------------------------------------------------------------------------------------------------------------- |
| `test`          | The test's name (required)                                                                                                 |
| `suite`         | The suite it belongs to                                                                                                    |
| `platforms`     | Where it runs: `web`, `ios`, `android` (default `web`)                                                                     |
| `starts from`   | A saved state to restore before step 1 — e.g. start signed in                                                              |
| `persona`       | Who is using the product; the agent acts as that person                                                                    |
| `save state as` | A state this test captures at the end of a passing run, optionally with a lifetime: `save state as: checkout-ready for 7d` |

### Variables and literals

* `{name}` references a variable. Values come from the test's per-environment
  variables plus anything an earlier `/extract` bound. A reference with no
  value **fails the step and names the fix** — it never silently passes
  through.
* `"quoted text"` and `[Label:value]` are **literals** — exact ground truth the
  page must match verbatim. `[Plan:Standard Plan]` reads naturally in the step
  while pinning the exact value.

Parsing is strict and helpful: unknown frontmatter keys, unknown `/commands`,
and malformed `/extract` lines are errors, and every problem in the file is
reported at once with its line number. Using `{a_variable}` before the step
that extracts it is caught at edit time, not at run time. The editor marks the
offending step cards, and **Run** stays disabled until the test parses.

## The QA tab

QA lives on your project's pages in ReasonOS, as a **QA** item in the Build
rail. It is enabled per project under **Project Settings → QA**.

The surface is organized as **collections → suites → tests**:

* **Collections** are the top-level grouping — typically an area of the product
  ("checkout", "onboarding") — and carry schedule settings.
* **Suites** group related tests and define a **run order**: stages that run
  top to bottom, each either in order or in parallel. By default,
  state-producing tests (those that `save state as`) run in a first serial
  stage, then everything that depends on them runs in parallel.
* **Tests** are the plain-English files above, with tabs for **Steps**,
  **Runs**, **Schedule**, **Variables**, and **Config** (target platforms,
  persona, saved-state behavior).

The tree shows live status dots per test (passed, failed, running, not yet
run), failing-count badges per suite, and a platform filter.

### Environments

A QA run executes against a named **environment** — execution context you
define in the QA tab (conventionally `local`, `staging`, `production`):

* For **web**, an environment is a base URL: where that build of your product
  is hosted. Steps can open relative paths ("Open /checkout") and resolve
  against it.
* For **iOS and Android**, an environment registers an app build per platform —
  a URL, its SHA-256 digest, a version, and the app id. The build is fetched
  and digest-verified before anything is installed; a mismatched artifact
  fails the run rather than launching the wrong binary.

Each test's **Variables** tab defines values per environment — `{base_url}`,
test credentials, expected copy — so the same test runs against local, staging,
and production with the right values in each.

### Personas

Personas are defined in **Project Settings → QA** and picked per test. A
persona describes *who* is using the product — "a first-time shopper in a
hurry", "a power user who knows the keyboard shortcuts" — and shades *how* the
action agent behaves: a first-time user reads before clicking. A persona never
changes *what* a step requires; assertions are judged the same for everyone.

### Saved states

Tests hand sessions to each other through **saved states**. A test that ends
with `save state as: shopper-logged-in` captures the browser session — cookies
and local storage — under that name (with an optional lifetime like `for 7d`).
Another test declaring `starts from: shopper-logged-in` restores it before its
first step and begins signed in, instead of re-walking the login flow. Only a
passing run saves its state, so a broken session is never handed on. An
expired or missing state fails the dependent run loudly rather than silently
starting fresh.

<Note>
  Saved states are captured and restored today, and suites order state-producing
  tests first automatically. A browsable list of captured states in the QA tab is
  still in progress — the Saved states panel currently explains the feature
  rather than listing entries.
</Note>

## Running QA

### From the QA tab

* **Run this test** on a test's header.
* **Run the suite** / **Run the collection** on their pages — these execute
  the stored run order, stage by stage.

Runs execute against the environment selected in the QA tab's environment
switcher. Starting a run enqueues it; a fleet of QA workers claims and executes
queued runs, so the browser tab only watches — you can close it and the run
completes and is recorded regardless. Worker pools are typed per platform: web
runs on workers with a browser, iOS on a Simulator pool, Android on an adb
pool. A test declaring `platforms: web, ios` produces one leg per platform
under a single report, which rolls up to the worst status.

If a worker is lost mid-run, the run is not left spinning: the platform hands
the job to another worker, and a run that could not be completed is marked
failed with the reason.

<Note>
  iOS and Android execution is early: today a mobile leg installs the
  environment's registered, digest-verified build and launches it. Agent-driven
  interaction on mobile — tapping, typing, reading the screen — is in progress;
  those steps currently fail with an explicit "not supported yet" reason rather
  than silently passing.
</Note>

### On a schedule, or on events

Schedule settings live on collections (suites can inherit or detach) and
support cadences of every 15 minutes, hourly, and daily. Event triggers can
start a run on pull-request activity or when a push lands on the project's
default branch. Scheduled and event-triggered runs execute against the
`staging` environment.

<Note>
  Scheduling is newly landed and still rolling out; some Schedule pages in the
  UI may still describe runs as manual-only. A deploy-gate trigger ("run before
  promoting a build") is planned but not available yet.
</Note>

## Reading results

### While a run executes

An in-progress run streams itself live: the page replays in the report as the
DOM changes, the model's reasoning appears token by token under the step it
belongs to, and steps flip from running to passed or failed as they settle. A
run's report is available within a second of it starting.

### The run report

Every run — live or historical — has a report:

* A **status tile**: passed / failed / running, the environment, platform, how
  the run was started, and timing.
* A **failed reason** for failing runs — the judging agent's verdict, written
  as a bug report a developer can act on: the exact text, element, or
  console/network evidence observed, and what was expected instead.
* **Per-step results** with elapsed time. Each action step expands into "what
  the agent did" — the individual interactions it performed — with a
  screenshot beside each one. Each step also carries the frame the agent was
  looking at, the assertion's reasoning, and any extracted value. Clicking a
  step opens a detail view with the captured frame, the parsed step, the
  error, and the agent's reasoning, with previous/next navigation.
* A **session replay**: a DOM-level recording of the whole session (not a
  video), replayable with a timeline. The recording is captured invisibly —
  the page under test sees no injected scripts.

Run history is durable and lives with the project: the test's **Runs** tab
lists past runs, and the sidebar's status dots reflect the latest result per
test.

<Note>
  QA test definitions are currently stored with the project and edited in the QA
  tab. Storing them as committed files in the branch's `.reasonos/qa/` tree — so
  tests version with the code they test — is planned.
</Note>

## QA targets in your build: `ai_qa_test`

For QA checks that belong in the workspace itself, rbs ships an agent-driven
test rule. An `ai_qa_test` target starts your app, hands an AI agent a real
Chrome browser (driven through Chrome DevTools), and has it execute your steps
and report pass or fail — as a normal test target.

```python theme={null}
load("@rbs//agent/ai_task.rbs", "ai_qa_test")

ai_qa_test(
    name = "qa_login",
    run_targets = [":dev"],
    todos = [
        "go to the login page",
        "sign in with the test account",
        "verify the signed-in user's name is shown in the header",
    ],
)
```

| Attribute       | Meaning                                                         |
| --------------- | --------------------------------------------------------------- |
| `todos`         | Ordered plain-English steps for the agent to execute (required) |
| `run_targets`   | Targets to start in the background first — your dev server      |
| `headless`      | Run Chrome headless (default `True`; set `False` to watch)      |
| `model`         | Optional model override                                         |
| `timeout`       | Overall time limit in seconds (default `300`)                   |
| `system_prompt` | Replace the default QA-agent instructions                       |

Build the app target first, then run the QA target like any other test:

```bash theme={null}
rbs build :dev
rbs test :qa_login
```

The runtime starts each `run_targets` binary in the background, connects the
browser tooling, runs the agent through the steps, prints its step-by-step
report, and passes or fails the target on the agent's verdict. The background
processes are stopped when the run ends.

Two requirements:

* **A model API key** in the environment — `ANTHROPIC_API_KEY`,
  `OPENAI_API_KEY`, `XAI_API_KEY`, or `GEMINI_API_KEY`.
* **Chrome and a Node.js toolchain** available to the workspace (the browser
  tooling is fetched on first use).

`ai_qa_test` is a thin wrapper over the general `ai_task` rule, which can wire
any MCP tool server into an agent-executed target — the QA wrapper just
preconfigures the browser.

<Note>
  Because `ai_qa_test` targets are ordinary test targets, a broad `rbs test
    //...` sweep will execute them — starting servers, a browser, and a model
  session. Keep them in their own package or CI job and run them deliberately
  rather than inside every unit-test sweep.
</Note>

## QA, `rbs test`, and CI

|                         | Ordinary tests               | `ai_qa_test` targets                | The QA tab                                            |
| ----------------------- | ---------------------------- | ----------------------------------- | ----------------------------------------------------- |
| **What it verifies**    | Your code, deterministically | A locally running app, agent-driven | The deployed product, per environment                 |
| **Where it's declared** | `BUILD.rbs`                  | `BUILD.rbs`                         | The QA tab                                            |
| **How it runs**         | `rbs test`, cached           | `rbs test`, on demand               | QA workers, from the queue                            |
| **When it runs**        | Every build, CI              | When you invoke it                  | On demand, on a schedule, on PR / default-branch push |
| **Where results go**    | Terminal / CI run            | Terminal / CI run                   | The run report in the QA tab                          |

They compose: CI runs your ordinary tests on every change; `ai_qa_test`
targets add agent-level checks you can invoke from the same build graph; and
the QA tab continuously exercises what's actually deployed, with evidence a
person can review.