Skip to main content
ReasonOS includes automated QA as a platform capability: tests written in plain English, executed by AI agents that use your product the way a person does. A QA run drives a real Chrome browser with real mouse and keyboard events — no selectors, no test scripts — judges each assertion strictly, and leaves behind evidence for every step: what the agent did, what it saw, why it passed or failed. There are two ways to use it:
  • The QA tab — a full QA surface on your project’s pages: author tests in plain English, organize them into suites and collections, run them against named environments on a fleet of QA workers, and read rich run reports with screenshots and session replay.
  • Tests as code — the same tests committed as qa_* declarations under .reasonos/qa/, reviewed like code and run by the same platform.
Both sit alongside — not instead of — ordinary tests. go_test, vitest_test, and friends verify your code deterministically; QA verifies the running product.

How a QA run works

Each step of a test is routed to a specialized agent:
  • Action steps are performed by an action agent: it reads the page (the full accessibility tree, page text, and a screenshot), decides the next moves, and performs them as real input events. It batches sensibly — a whole form is filled and submitted in one pass — and waits adaptively for the page to settle rather than sleeping.
  • Assertions (/verify, Expect:) go to an assertion agent that judges strictly: an exact value you pin down must appear exactly as written — currency symbols, separators, and casing included. Garbled or truncated text never satisfies an assertion about that text. Assertions about how the page looks — layout, readability, broken assets — are judged by a design-review agent with a defect taxonomy.
  • Extractions (/extract) read one value off the page, following your instructions to the letter (“digits only”, “without the currency symbol”), and bind it to a variable later steps can reference.
Console errors and failed network requests are collected as evidence throughout the run: the judging agents cite them in verdicts, and a claim like “no error is shown” is checked against the console, not just the visible page. A run stops at the first failed step; the remaining steps are recorded as skipped, so every report reads as “N passed · 1 failed · M skipped” with the failure front and center.

Writing tests in plain English

A QA test is a small text file: a frontmatter block, then one step per line. The step editor in the QA tab writes this format for you, and validates every line as you type.

Step forms

List markers (1., -, *) are cosmetic, and lines starting with # are comments.

Frontmatter

Variables and literals

  • {name} references a variable. Values come from the test’s per-environment variables plus anything an earlier /extract bound. A reference with no value fails the step and names the fix — it never silently passes through.
  • "quoted text" and [Label:value] are literals — exact ground truth the page must match verbatim. [Plan:Standard Plan] reads naturally in the step while pinning the exact value.
Parsing is strict and helpful: unknown frontmatter keys, unknown /commands, and malformed /extract lines are errors, and every problem in the file is reported at once with its line number. Using {a_variable} before the step that extracts it is caught at edit time, not at run time. The editor marks the offending step cards, and Run stays disabled until the test parses.

The QA tab

QA lives on your project’s pages in ReasonOS, as a QA item in the Build rail. It is enabled per project under Project Settings → QA. The surface is organized as collections → suites → tests:
  • Collections are the top-level grouping — typically an area of the product (“checkout”, “onboarding”) — and carry schedule settings.
  • Suites group related tests and define a run order: stages that run top to bottom, each either in order or in parallel. By default, state-producing tests (those that save state as) run in a first serial stage, then everything that depends on them runs in parallel.
  • Tests are the plain-English files above, with tabs for Steps, Runs, Schedule, Variables, and Config (target platforms, persona, saved-state behavior).
The tree shows live status dots per test (passed, failed, running, not yet run), failing-count badges per suite, and a platform filter.

Environments

A QA run executes against a named environment — execution context you define in the QA tab (conventionally local, staging, production):
  • For web, an environment is a base URL: where that build of your product is hosted. Steps can open relative paths (“Open /checkout”) and resolve against it.
  • For iOS and Android, an environment registers an app build per platform — a URL, its SHA-256 digest, a version, and the app id. The build is fetched and digest-verified before anything is installed; a mismatched artifact fails the run rather than launching the wrong binary.
Each test’s Variables tab defines values per environment — {base_url}, test credentials, expected copy — so the same test runs against local, staging, and production with the right values in each.

Personas

Personas are defined in Project Settings → QA and picked per test. A persona describes who is using the product — “a first-time shopper in a hurry”, “a power user who knows the keyboard shortcuts” — and shades how the action agent behaves: a first-time user reads before clicking. A persona never changes what a step requires; assertions are judged the same for everyone.

Saved states

Tests hand sessions to each other through saved states. A test that ends with save state as: shopper-logged-in captures the browser session — cookies and local storage — under that name (with an optional lifetime like for 7d). Another test declaring starts from: shopper-logged-in restores it before its first step and begins signed in, instead of re-walking the login flow. Only a passing run saves its state, so a broken session is never handed on. An expired or missing state fails the dependent run loudly rather than silently starting fresh.
Saved states are captured and restored today, and suites order state-producing tests first automatically. A browsable list of captured states in the QA tab is still in progress — the Saved states panel currently explains the feature rather than listing entries.

Running QA

From the QA tab

  • Run this test on a test’s header.
  • Run the suite / Run the collection on their pages — these execute the stored run order, stage by stage.
Runs execute against the environment selected in the QA tab’s environment switcher. Starting a run enqueues it; a fleet of QA workers claims and executes queued runs, so the browser tab only watches — you can close it and the run completes and is recorded regardless. Worker pools are typed per platform: web runs on workers with a browser, iOS on a Simulator pool, Android on an adb pool. A test declaring platforms: web, ios produces one leg per platform under a single report, which rolls up to the worst status. If a worker is lost mid-run, the run is not left spinning: the platform hands the job to another worker, and a run that could not be completed is marked failed with the reason.
iOS and Android execution is early: today a mobile leg installs the environment’s registered, digest-verified build and launches it. Agent-driven interaction on mobile — tapping, typing, reading the screen — is in progress; those steps currently fail with an explicit “not supported yet” reason rather than silently passing.

On a schedule, or on events

Schedule settings live on collections (suites can inherit or detach) and support cadences of every 15 minutes, hourly, and daily. Event triggers can start a run on pull-request activity or when a push lands on the project’s default branch. Scheduled and event-triggered runs execute against the staging environment.
Scheduling is newly landed and still rolling out; some Schedule pages in the UI may still describe runs as manual-only. A deploy-gate trigger (“run before promoting a build”) is planned but not available yet.

Reading results

While a run executes

An in-progress run streams itself live: the page replays in the report as the DOM changes, the model’s reasoning appears token by token under the step it belongs to, and steps flip from running to passed or failed as they settle. A run’s report is available within a second of it starting.

The run report

Every run — live or historical — has a report:
  • A status tile: passed / failed / running, the environment, platform, how the run was started, and timing.
  • A failed reason for failing runs — the judging agent’s verdict, written as a bug report a developer can act on: the exact text, element, or console/network evidence observed, and what was expected instead.
  • Per-step results with elapsed time. Each action step expands into “what the agent did” — the individual interactions it performed — with a screenshot beside each one. Each step also carries the frame the agent was looking at, the assertion’s reasoning, and any extracted value. Clicking a step opens a detail view with the captured frame, the parsed step, the error, and the agent’s reasoning, with previous/next navigation.
  • A session replay: a DOM-level recording of the whole session (not a video), replayable with a timeline. The recording is captured invisibly — the page under test sees no injected scripts.
Run history is durable and lives with the project: the test’s Runs tab lists past runs, and the sidebar’s status dots reflect the latest result per test.
New projects store QA tests with the project, edited in the QA tab. To version them with the code, use Move into the repository in the QA tab’s settings: every test, suite and collection and the run settings are written to .reasonos/qa in one commit (or a change request, if the default branch needs one). From then on, edits in the QA tab are commits made as you, and edits to the files show up in the tab. See QA tests as code.

QA tests as code: qa_test

The same tests can live in the repository as committed files, so test changes are reviewed with the code they test and agents can write them. They are declared with the qa_* rules from @rbs//qa/defs.rbs in .reasonos/qa/**/BUILD.rbs:
qa_collection, qa_suite and qa_module declare collections, suites and reusable steps, and qa_config in .reasonos/qa/BUILD.rbs holds the run settings. The step language is the one described above. These rules declare tests; the ReasonOS platform runs them, rbs test does not. Building a qa_test writes its normalized definition, which the platform reads. From the command line:
Checks every QA file in the workspace. Success prints N QA file(s), no problems.; otherwise it lists each problem and exits non-zero.
Rewrites QA files the way the QA tab writes them, printing formatted <path> for each file it changed. Already-formatted files produce no output.
Starts a run of the test named sign-in-page (a label such as //apps/shop/qa:add also works) on the ReasonOS platform, as this checkout has it, and follows it to the end. It prints Queued … (run …), then a summary such as Sign-in page passed on staging, and exits non-zero when the run fails. rbs qa run takes the project and control plane from the repository (or RBS_CONTROL_PLANE_URL and RBS_PROJECT_NAME) and the credential from RBS_CONTROL_PLANE_TOKEN, which every ReasonOS branch server sets.
The former ai_qa_test rule is retired. For an agent-driven check inside the build graph, use an ai_task target with the Chrome DevTools MCP server.

QA, rbs test, and CI

They compose: CI runs your ordinary tests on every change; ai_task targets add agent-level checks you can invoke from the same build graph; and ReasonOS QA continuously exercises what’s actually deployed, with evidence a person can review.