Running Tests

Overall status folds the steps as FAIL > INCONCLUSIVE > PASS: one unmeasured claim makes the whole run amber.

Next to every verdict — on a step, a run row, and in the batch report — sits a small tag naming what decided it:

A run’s judge is the weakest judge among its asserting steps, so one LLM-judged claim marks the whole run LLM-judged. This is what the Overview KPI Measured passes counts, and what the Last: pass, not measured filter finds.

Five cards, each with a value, a delta and a 14-day sparkline. The window is the last 14 days: "current" is the last 7 days, "prior" the 7 before it. Blocked runs are excluded from the rate and the duration — a case that never started should not move either number.

A stacked bar chart below the cards, once any runs exist. It shows one bar for each of the last 14 days. A day without runs shows an empty bar. Each bar stacks pass, fail, inconclusive and blocked runs. Errors count as fails. The chart is re-rendered when you switch theme.

The eight most recent runs — Case | Title | Verdict | Duration | Env | When. The verdict cell carries the status pill, the judge tag, and a known badge when the failure is already in the known-issues registry. Identical rows in a row show as one row with a count, for example ×2. Rows are identical when case, verdict and environment match. Clicking a row opens its report panel over the Overview. All runs jumps to Runs.

There are no filters here; the filters live on the run history.

Above both Overview and Runs, one card per batch currently in flight on any worker — yours or a colleague’s. It shows the wave’s name and its project / section, the env, the worker, the owner and the model, a five-segment progress bar (pass / fail / inconclusive / error / blocked), the counts, an ETA, when it started, and the case being executed right now. If the wave is yours there is a Live view button and, while it is still running, Edit run (see test-management.adoc#run-edit).

A paused wave says so, with the reason — most often the Claude usage quota, which the runner waits out rather than abandoning the batch. The wave keeps its worker and its emulator for the whole pause and resumes there by itself, so a run aimed at that worker, or at any env on the same emulator, is refused until the wave ends, exactly as while it is running. Nothing is queued: start it again later, or ask the wave’s owner to stop it.

The header is read top to bottom, and each row answers one question.

Row 1 — what is selected, and how a run of it is made. The title, then the shown / total counter, then the selection pill (N selected, Select all, Clear). Then one bordered run bar holding, left to right, the environment picker, the model picker, the order picker, an optional run name and the run button. It reads Run N on env and names the cases and the environment. After it a Bulk actions menu with Run QA-verified regression and Publish all pending, an icon-only Refresh, and a Library menu holding New test case, New project, Import XML…, Sync runs… and Trash (N).

Row 2 — what the list shows. The search box, then the filters All portals, All projects, All sections, All types, All priorities and Any last result; then the Sort select, Reset filters (shown only while a filter is set) and the More filters toggle, whose label carries the number of active hidden filters.

Row 3 — the rare filters. Shown only while More filters is open: Any ISP mode, Any run, Any claims. The row opens by itself when one of the three becomes active, and remembers whether it was open between visits.

The filters narrow the list, never the selection: a case you have ticked stays ticked when the next keystroke hides it. That is how a batch of two is assembled by id — tick C11653, search for C91355, tick that one as well, and both ride into the run.

What the current filters hide is named rather than dropped: the pill reads N selected and, when some of them are off screen, M hidden beside it, while the run button always counts the whole selection. So a wave is disclosed instead of being trimmed behind you.

A wave of more than ten cases asks once before it starts. The question names the case count, the environment and the model. If the filters hide some selected cases, the question also names how many. Ten cases or fewer start at once.

Reset filters appears as soon as any of the ten is set, and clears all ten at once.

Sort orders the list: Section · ID (the default), Title, Type, Priority, Reference or Last run. Under Title, Type and Reference an empty value sorts last, a case with no priority sorts after every known one, and a never-run case sinks to the bottom of Last run; the default Section · ID is the exception — it compares sections as plain text, so a case with no section sorts first. Priority sorts Critical first rather than alphabetically, and every key breaks ties on a numeric-aware id compare, so C9 precedes C10.

Tick the checkbox or click the row body to toggle. Select all adds every currently filtered row to the selection — it does not replace it, so the case picked under an earlier search survives; Clear empties the selection and is always in place, greyed out while there is nothing to clear. Both sit in the selection pill beside the title, next to the shown / total counter and the M hidden chip.

A single ticked case posts POST /api/gateway/run; two or more post POST /api/gateway/run/batch. The body carries the chosen env plus test_file / test_files, and optional model / dry_run / order.

A 409 means that env’s emulator is already busy — see Selecting the Target Environment.

The icon-only Refresh beside the run bar reloads the list from disk (after an import or an external edit). Everything else lives in the Library menu:

With rows ticked, a bulk bar appears above the list: Add to project…, Set section…, Find / replace…, Export XML and Delete (which moves the cases to the trash). All of them are described in Test Management.

The scheduler is env-exclusive with an emulator mutex: each run holds a lock keyed on the env’s physical emulator backend (its emulator_url), not the env name. Consequences:

If every worker is occupied you instead get 503 — All workers busy.

Run history and Batch reports are a segmented control inside one card; your choice is remembered.

The history header carries a search box (case or title), a verdict filter — All / Pass / Pass · not measured / Fail / Inconclusive / Error / Blocked — a portal filter, and Clear.

The table is Case | Title | Verdict | Duration | Model | Owner | Env | When. Under the title of any non-passed run sits its reason. The verdict cell carries the status pill, the judge tag and the known badge. Clicking a row opens the report panel.

The list is paged on the server, 100 rows at a time; Show 100 more appends the next page. The search and the filters are applied by the worker over everything you may see, not by the browser over the page it happens to hold, so a case whose row sits on the fourth page is still found from the search box.

Archived batch reports, persisted as run-reports/batch-<id>.json on the worker that ran them. The gateway merges them owner-scoped to you. The table is Name | Started | Model | Total | Pass | Fail | Error | Inconclusive | Blocked | Duration | Env | Build | Owner; the Name cell falls back to the batch id and carries the project / section underneath.

The header of that view also has Import XML — take a run exported from another stand (or another install) back in as a report of your own — plus Refresh and Clear all.

Opening a report gives the Batch Report dialog:

Names, comments, defect keys, the exports and the in-flight edit are all described in Test Management → Naming, Commenting and Exporting a Run.

Case id and title, the batch position, the status pill, and:

While a run is live the header also shows the phase stepper — Load → Pages → Execute → Parse → Done — a progress bar over the steps, and, for a batch, a five-colour bar with counts, an ETA and a foldable Queue listing what is still to come.

That batch strip belongs to the wave that is moving: it goes as soon as the wave ends, whichever way the end is noticed, and starting a single case clears any strip left over from a finished one.

For a finished run: the big status pill, the judge tag, a known: <bucket> badge if the registry already explains this red, the case id, elapsed time, model alias, when it ran, the env it ran against and its owner. Then:

One row per step: a mark, the description, the judge tag, and — for a failure — the runner’s own verdict line inline.

A step description beginning [not measured] names the reason the evidence gate downgraded it. [verify], [polling] and [empty-state] mark a deterministic measurement, which the gate never downgrades.

Two sub-tabs on the right:

The bar above them has Jump to FAIL — 3/12 FAIL — which scrolls to the next failed or inconclusive verdict line and tints every one of them; a red report jumps to its first failure automatically. Bottom jumps to the latest line.

When the run produced them:

Force stop hard-kills the underlying Claude CLI subprocess and tears down the MCP connection for that run. The case is recorded as error with "Stopped by user".

Force stop is owner-scoped. POST /api/run/stop fans out to every worker as the calling user, and each worker stops only that user’s run — it is a no-op on runs you do not own. Admins can stop any run.