Automated QA

ReasonOS includes automated QA as a platform capability: tests written in plain English, executed by AI agents that use your product the way a person does. A QA run drives a real Chrome browser with real mouse and keyboard events — no selectors, no test scripts — judges each assertion strictly, and leaves behind evidence for every step: what the agent did, what it saw, why it passed or failed. There are two ways to use it:

The QA tab — a full QA surface on your project’s pages: author tests in plain English, organize them into suites and collections, run them against named environments on a fleet of QA workers, and read rich run reports with screenshots and session replay.
ai_qa_test targets — agent-driven browser tests declared in your BUILD.rbs files and run with rbs test, right in your workspace.

Both sit alongside — not instead of — ordinary tests. go_test, vitest_test, and friends verify your code deterministically; QA verifies the running product.

How a QA run works

Each step of a test is routed to a specialized agent:

Action steps are performed by an action agent: it reads the page (the full accessibility tree, page text, and a screenshot), decides the next moves, and performs them as real input events. It batches sensibly — a whole form is filled and submitted in one pass — and waits adaptively for the page to settle rather than sleeping.
Assertions (/verify, Expect:) go to an assertion agent that judges strictly: an exact value you pin down must appear exactly as written — currency symbols, separators, and casing included. Garbled or truncated text never satisfies an assertion about that text. Assertions about how the page looks — layout, readability, broken assets — are judged by a design-review agent with a defect taxonomy.
Extractions (/extract) read one value off the page, following your instructions to the letter (“digits only”, “without the currency symbol”), and bind it to a variable later steps can reference.

Console errors and failed network requests are collected as evidence throughout the run: the judging agents cite them in verdicts, and a claim like “no error is shown” is checked against the console, not just the visible page. A run stops at the first failed step; the remaining steps are recorded as skipped, so every report reads as “N passed · 1 failed · M skipped” with the failure front and center.

Writing tests in plain English

A QA test is a small text file: a frontmatter block, then one step per line. The step editor in the QA tab writes this format for you, and validates every line as you type.

---
test: Incorrect price or currency for Germany shoppers
suite: checkout
platforms: web
persona: a first-time shopper in a hurry
---

1. Open the storefront at {base_url}, as a shopper in Germany.
2. Add the [Plan:Standard Plan] to the basket and go to checkout.
   Expect: the subtotal reads "€49,00" — euro symbol, comma decimal.
3. /extract order_total = the total amount shown in the order summary
4. /verify the currency stays EUR and the total reads {order_total}

Step forms

You write	What it means
`Open /checkout and add the first item`	An action — the agent performs it in the product
`/verify the subtotal reads "€49,00"`	An assertion, judged strictly against the current page
`/extract order_total = the total in the order summary`	Reads a value off the page and binds `{order_total}` for later steps
`Expect: the cart badge shows "1"`	An assertion attached to the action step above it
`Only: ios, android`	Restricts the step above to the listed platforms
`Save state as checkout-ready`	Snapshots the browser session (cookies + storage) for other tests to start from

List markers (1., -, *) are cosmetic, and lines starting with # are comments.

Frontmatter

Key	Meaning
`test`	The test’s name (required)
`suite`	The suite it belongs to
`platforms`	Where it runs: `web`, `ios`, `android` (default `web`)
`starts from`	A saved state to restore before step 1 — e.g. start signed in
`persona`	Who is using the product; the agent acts as that person
`save state as`	A state this test captures at the end of a passing run, optionally with a lifetime: `save state as: checkout-ready for 7d`

Variables and literals

{name} references a variable. Values come from the test’s per-environment variables plus anything an earlier /extract bound. A reference with no value fails the step and names the fix — it never silently passes through.
"quoted text" and [Label:value] are literals — exact ground truth the page must match verbatim. [Plan:Standard Plan] reads naturally in the step while pinning the exact value.

Parsing is strict and helpful: unknown frontmatter keys, unknown /commands, and malformed /extract lines are errors, and every problem in the file is reported at once with its line number. Using {a_variable} before the step that extracts it is caught at edit time, not at run time. The editor marks the offending step cards, and Run stays disabled until the test parses.

The QA tab

QA lives on your project’s pages in ReasonOS, as a QA item in the Build rail. It is enabled per project under Project Settings → QA. The surface is organized as collections → suites → tests:

Collections are the top-level grouping — typically an area of the product (“checkout”, “onboarding”) — and carry schedule settings.
Suites group related tests and define a run order: stages that run top to bottom, each either in order or in parallel. By default, state-producing tests (those that save state as) run in a first serial stage, then everything that depends on them runs in parallel.
Tests are the plain-English files above, with tabs for Steps, Runs, Schedule, Variables, and Config (target platforms, persona, saved-state behavior).

The tree shows live status dots per test (passed, failed, running, not yet run), failing-count badges per suite, and a platform filter.

Environments

A QA run executes against a named environment — execution context you define in the QA tab (conventionally local, staging, production):

For web, an environment is a base URL: where that build of your product is hosted. Steps can open relative paths (“Open /checkout”) and resolve against it.
For iOS and Android, an environment registers an app build per platform — a URL, its SHA-256 digest, a version, and the app id. The build is fetched and digest-verified before anything is installed; a mismatched artifact fails the run rather than launching the wrong binary.

Each test’s Variables tab defines values per environment — {base_url}, test credentials, expected copy — so the same test runs against local, staging, and production with the right values in each.

Personas

Personas are defined in Project Settings → QA and picked per test. A persona describes who is using the product — “a first-time shopper in a hurry”, “a power user who knows the keyboard shortcuts” — and shades how the action agent behaves: a first-time user reads before clicking. A persona never changes what a step requires; assertions are judged the same for everyone.

Saved states

Tests hand sessions to each other through saved states. A test that ends with save state as: shopper-logged-in captures the browser session — cookies and local storage — under that name (with an optional lifetime like for 7d). Another test declaring starts from: shopper-logged-in restores it before its first step and begins signed in, instead of re-walking the login flow. Only a passing run saves its state, so a broken session is never handed on. An expired or missing state fails the dependent run loudly rather than silently starting fresh.

Saved states are captured and restored today, and suites order state-producing tests first automatically. A browsable list of captured states in the QA tab is still in progress — the Saved states panel currently explains the feature rather than listing entries.

Running QA

From the QA tab

Run this test on a test’s header.
Run the suite / Run the collection on their pages — these execute the stored run order, stage by stage.

Runs execute against the environment selected in the QA tab’s environment switcher. Starting a run enqueues it; a fleet of QA workers claims and executes queued runs, so the browser tab only watches — you can close it and the run completes and is recorded regardless. Worker pools are typed per platform: web runs on workers with a browser, iOS on a Simulator pool, Android on an adb pool. A test declaring platforms: web, ios produces one leg per platform under a single report, which rolls up to the worst status. If a worker is lost mid-run, the run is not left spinning: the platform hands the job to another worker, and a run that could not be completed is marked failed with the reason.

iOS and Android execution is early: today a mobile leg installs the environment’s registered, digest-verified build and launches it. Agent-driven interaction on mobile — tapping, typing, reading the screen — is in progress; those steps currently fail with an explicit “not supported yet” reason rather than silently passing.

On a schedule, or on events

Schedule settings live on collections (suites can inherit or detach) and support cadences of every 15 minutes, hourly, and daily. Event triggers can start a run on pull-request activity or when a push lands on the project’s default branch. Scheduled and event-triggered runs execute against the staging environment.

Scheduling is newly landed and still rolling out; some Schedule pages in the UI may still describe runs as manual-only. A deploy-gate trigger (“run before promoting a build”) is planned but not available yet.

Reading results

While a run executes

An in-progress run streams itself live: the page replays in the report as the DOM changes, the model’s reasoning appears token by token under the step it belongs to, and steps flip from running to passed or failed as they settle. A run’s report is available within a second of it starting.

The run report

Every run — live or historical — has a report:

A status tile: passed / failed / running, the environment, platform, how the run was started, and timing.
A failed reason for failing runs — the judging agent’s verdict, written as a bug report a developer can act on: the exact text, element, or console/network evidence observed, and what was expected instead.
Per-step results with elapsed time. Each action step expands into “what the agent did” — the individual interactions it performed — with a screenshot beside each one. Each step also carries the frame the agent was looking at, the assertion’s reasoning, and any extracted value. Clicking a step opens a detail view with the captured frame, the parsed step, the error, and the agent’s reasoning, with previous/next navigation.
A session replay: a DOM-level recording of the whole session (not a video), replayable with a timeline. The recording is captured invisibly — the page under test sees no injected scripts.

Run history is durable and lives with the project: the test’s Runs tab lists past runs, and the sidebar’s status dots reflect the latest result per test.

QA test definitions are currently stored with the project and edited in the QA tab. Storing them as committed files in the branch’s .reasonos/qa/ tree — so tests version with the code they test — is planned.

QA targets in your build: `ai_qa_test`

For QA checks that belong in the workspace itself, rbs ships an agent-driven test rule. An ai_qa_test target starts your app, hands an AI agent a real Chrome browser (driven through Chrome DevTools), and has it execute your steps and report pass or fail — as a normal test target.

load("@rbs//agent/ai_task.rbs", "ai_qa_test")

ai_qa_test(
    name = "qa_login",
    run_targets = [":dev"],
    todos = [
        "go to the login page",
        "sign in with the test account",
        "verify the signed-in user's name is shown in the header",
    ],
)

Attribute	Meaning
`todos`	Ordered plain-English steps for the agent to execute (required)
`run_targets`	Targets to start in the background first — your dev server
`headless`	Run Chrome headless (default `True`; set `False` to watch)
`model`	Optional model override
`timeout`	Overall time limit in seconds (default `300`)
`system_prompt`	Replace the default QA-agent instructions

Build the app target first, then run the QA target like any other test:

rbs build :dev
rbs test :qa_login

The runtime starts each run_targets binary in the background, connects the browser tooling, runs the agent through the steps, prints its step-by-step report, and passes or fails the target on the agent’s verdict. The background processes are stopped when the run ends. Two requirements:

A model API key in the environment — ANTHROPIC_API_KEY, OPENAI_API_KEY, XAI_API_KEY, or GEMINI_API_KEY.
Chrome and a Node.js toolchain available to the workspace (the browser tooling is fetched on first use).

ai_qa_test is a thin wrapper over the general ai_task rule, which can wire any MCP tool server into an agent-executed target — the QA wrapper just preconfigures the browser.

Because ai_qa_test targets are ordinary test targets, a broad rbs test //... sweep will execute them — starting servers, a browser, and a model session. Keep them in their own package or CI job and run them deliberately rather than inside every unit-test sweep.

QA, `rbs test`, and CI

	Ordinary tests	`ai_qa_test` targets	The QA tab
What it verifies	Your code, deterministically	A locally running app, agent-driven	The deployed product, per environment
Where it’s declared	`BUILD.rbs`	`BUILD.rbs`	The QA tab
How it runs	`rbs test`, cached	`rbs test`, on demand	QA workers, from the queue
When it runs	Every build, CI	When you invoke it	On demand, on a schedule, on PR / default-branch push
Where results go	Terminal / CI run	Terminal / CI run	The run report in the QA tab

They compose: CI runs your ordinary tests on every change; ai_qa_test targets add agent-level checks you can invoke from the same build graph; and the QA tab continuously exercises what’s actually deployed, with evidence a person can review.

The ReasonOS Editor

Platform

How a QA run works

Writing tests in plain English

Step forms

Frontmatter

Variables and literals

The QA tab

Environments

Personas

Saved states

Running QA

From the QA tab

On a schedule, or on events

Reading results

While a run executes

The run report

QA targets in your build: `ai_qa_test`

QA, `rbs test`, and CI

​How a QA run works

​Writing tests in plain English

​Step forms

​Frontmatter

​Variables and literals

​The QA tab

​Environments

​Personas

​Saved states

​Running QA

​From the QA tab

​On a schedule, or on events

​Reading results

​While a run executes

​The run report

​QA targets in your build: ai_qa_test

​QA, rbs test, and CI

How a QA run works

Writing tests in plain English

Step forms

Frontmatter

Variables and literals

The QA tab

Environments

Personas

Saved states

Running QA

From the QA tab

On a schedule, or on events

Reading results

While a run executes

The run report

QA targets in your build: `ai_qa_test`

QA, `rbs test`, and CI