- The QA tab — a full QA surface on your project’s pages: author tests in plain English, organize them into suites and collections, run them against named environments on a fleet of QA workers, and read rich run reports with screenshots and session replay.
- Tests as code — the same tests committed as
qa_*declarations under.reasonos/qa/, reviewed like code and run by the same platform.
go_test, vitest_test,
and friends verify your code deterministically; QA verifies the running product.
How a QA run works
Each step of a test is routed to a specialized agent:- Action steps are performed by an action agent: it reads the page (the full accessibility tree, page text, and a screenshot), decides the next moves, and performs them as real input events. It batches sensibly — a whole form is filled and submitted in one pass — and waits adaptively for the page to settle rather than sleeping.
- Assertions (
/verify,Expect:) go to an assertion agent that judges strictly: an exact value you pin down must appear exactly as written — currency symbols, separators, and casing included. Garbled or truncated text never satisfies an assertion about that text. Assertions about how the page looks — layout, readability, broken assets — are judged by a design-review agent with a defect taxonomy. - Extractions (
/extract) read one value off the page, following your instructions to the letter (“digits only”, “without the currency symbol”), and bind it to a variable later steps can reference.
Writing tests in plain English
A QA test is a small text file: a frontmatter block, then one step per line. The step editor in the QA tab writes this format for you, and validates every line as you type.Step forms
List markers (
1., -, *) are cosmetic, and lines starting with # are
comments.
Frontmatter
Variables and literals
{name}references a variable. Values come from the test’s per-environment variables plus anything an earlier/extractbound. A reference with no value fails the step and names the fix — it never silently passes through."quoted text"and[Label:value]are literals — exact ground truth the page must match verbatim.[Plan:Standard Plan]reads naturally in the step while pinning the exact value.
/commands,
and malformed /extract lines are errors, and every problem in the file is
reported at once with its line number. Using {a_variable} before the step
that extracts it is caught at edit time, not at run time. The editor marks the
offending step cards, and Run stays disabled until the test parses.
The QA tab
QA lives on your project’s pages in ReasonOS, as a QA item in the Build rail. It is enabled per project under Project Settings → QA. The surface is organized as collections → suites → tests:- Collections are the top-level grouping — typically an area of the product (“checkout”, “onboarding”) — and carry schedule settings.
- Suites group related tests and define a run order: stages that run
top to bottom, each either in order or in parallel. By default,
state-producing tests (those that
save state as) run in a first serial stage, then everything that depends on them runs in parallel. - Tests are the plain-English files above, with tabs for Steps, Runs, Schedule, Variables, and Config (target platforms, persona, saved-state behavior).
Environments
A QA run executes against a named environment — execution context you define in the QA tab (conventionallylocal, staging, production):
- For web, an environment is a base URL: where that build of your product is hosted. Steps can open relative paths (“Open /checkout”) and resolve against it.
- For iOS and Android, an environment registers an app build per platform — a URL, its SHA-256 digest, a version, and the app id. The build is fetched and digest-verified before anything is installed; a mismatched artifact fails the run rather than launching the wrong binary.
{base_url},
test credentials, expected copy — so the same test runs against local, staging,
and production with the right values in each.
Personas
Personas are defined in Project Settings → QA and picked per test. A persona describes who is using the product — “a first-time shopper in a hurry”, “a power user who knows the keyboard shortcuts” — and shades how the action agent behaves: a first-time user reads before clicking. A persona never changes what a step requires; assertions are judged the same for everyone.Saved states
Tests hand sessions to each other through saved states. A test that ends withsave state as: shopper-logged-in captures the browser session — cookies
and local storage — under that name (with an optional lifetime like for 7d).
Another test declaring starts from: shopper-logged-in restores it before its
first step and begins signed in, instead of re-walking the login flow. Only a
passing run saves its state, so a broken session is never handed on. An
expired or missing state fails the dependent run loudly rather than silently
starting fresh.
Saved states are captured and restored today, and suites order state-producing
tests first automatically. A browsable list of captured states in the QA tab is
still in progress — the Saved states panel currently explains the feature
rather than listing entries.
Running QA
From the QA tab
- Run this test on a test’s header.
- Run the suite / Run the collection on their pages — these execute the stored run order, stage by stage.
platforms: web, ios produces one leg per platform
under a single report, which rolls up to the worst status.
If a worker is lost mid-run, the run is not left spinning: the platform hands
the job to another worker, and a run that could not be completed is marked
failed with the reason.
iOS and Android execution is early: today a mobile leg installs the
environment’s registered, digest-verified build and launches it. Agent-driven
interaction on mobile — tapping, typing, reading the screen — is in progress;
those steps currently fail with an explicit “not supported yet” reason rather
than silently passing.
On a schedule, or on events
Schedule settings live on collections (suites can inherit or detach) and support cadences of every 15 minutes, hourly, and daily. Event triggers can start a run on pull-request activity or when a push lands on the project’s default branch. Scheduled and event-triggered runs execute against thestaging environment.
Scheduling is newly landed and still rolling out; some Schedule pages in the
UI may still describe runs as manual-only. A deploy-gate trigger (“run before
promoting a build”) is planned but not available yet.
Reading results
While a run executes
An in-progress run streams itself live: the page replays in the report as the DOM changes, the model’s reasoning appears token by token under the step it belongs to, and steps flip from running to passed or failed as they settle. A run’s report is available within a second of it starting.The run report
Every run — live or historical — has a report:- A status tile: passed / failed / running, the environment, platform, how the run was started, and timing.
- A failed reason for failing runs — the judging agent’s verdict, written as a bug report a developer can act on: the exact text, element, or console/network evidence observed, and what was expected instead.
- Per-step results with elapsed time. Each action step expands into “what the agent did” — the individual interactions it performed — with a screenshot beside each one. Each step also carries the frame the agent was looking at, the assertion’s reasoning, and any extracted value. Clicking a step opens a detail view with the captured frame, the parsed step, the error, and the agent’s reasoning, with previous/next navigation.
- A session replay: a DOM-level recording of the whole session (not a video), replayable with a timeline. The recording is captured invisibly — the page under test sees no injected scripts.
New projects store QA tests with the project, edited in the QA tab. To version
them with the code, use Move into the repository in the QA tab’s settings:
every test, suite and collection and the run settings are written to
.reasonos/qa in one commit (or a change request, if the default branch needs
one). From then on, edits in the QA tab are commits made as you, and edits to
the files show up in the tab. See QA tests as code.QA tests as code: qa_test
The same tests can live in the repository as committed files, so test changes
are reviewed with the code they test and agents can write them. They are
declared with the qa_* rules from @rbs//qa/defs.rbs in
.reasonos/qa/**/BUILD.rbs:
qa_collection, qa_suite and qa_module declare collections, suites and
reusable steps, and qa_config in .reasonos/qa/BUILD.rbs holds the run
settings. The step language is the one described above.
These rules declare tests; the ReasonOS platform runs them, rbs test does
not. Building a qa_test writes its normalized definition, which the platform
reads. From the command line:
N QA file(s), no problems.; otherwise it lists each problem and exits non-zero.
formatted <path>
for each file it changed. Already-formatted files produce no output.
sign-in-page (a label such as //apps/shop/qa:add also works) on the ReasonOS platform, as this checkout has it,
and follows it to the end. It prints Queued … (run …), then a summary such
as Sign-in page passed on staging, and exits non-zero when the run fails.
rbs qa run takes the project and control plane from the repository (or
RBS_CONTROL_PLANE_URL and RBS_PROJECT_NAME) and the credential from
RBS_CONTROL_PLANE_TOKEN, which every ReasonOS branch server sets.
The former
ai_qa_test rule is retired. For an agent-driven check inside the
build graph, use an ai_task target
with the Chrome DevTools MCP server.QA, rbs test, and CI
They compose: CI runs your ordinary tests on every change;
ai_task targets
add agent-level checks you can invoke from the same build graph; and ReasonOS
QA continuously exercises what’s actually deployed, with evidence a person can
review.