Architecture

The worker above is normally fronted by a single public gateway (gateway_service/, port :8083 — the only port exposed to the outside). The gateway owns identity and routing; the workers do the actual test work and are not published to the host (they are reached over the test-runner-net Docker network). One gateway, N interchangeable workers, many env profiles.

Per-run reads, batch status / reports and stop are routed to (or fanned out across) the worker that actually owns the caller’s run — owner-scoped, as the caller — so a run that lands on a non-primary worker is still visible and stoppable, and one account can never see or stop another’s run. See Environments & the Gateway for the deployment topology and day-to-day operations.

The gateway and the workers share two small SQLite data layers (pure data modules — no FastAPI imports, unit-testable in isolation) in a single file at run-reports/test_runner.db:

The converter has two independent code paths: convert_case() for simple single-scenario tests and convert_case_scenarios() for multi-phase tests. They build phases independently — fixes must be applied to both.

Before each run, test_server.py performs three checks. Their output is visible in the Output pane of the Runner tab and on the WebSocket stream:

Two small modules turn "the runner said PASS" into something a reader can check.

scripts/evidence_gate.py holds the contract handed to the judge: quote, verbatim, the page text that decides the claim, or answer UNVERIFIED. It then parses the answer back and checks the quote actually occurs in what the judge was shown — an invented quote is not evidence. The outcome is asymmetric on purpose, matching the runner’s existing rule that a false red is cheap while a false green is not:

A verdict a helper measured is exempt: descriptions prefixed [verify], [polling] or [empty-state], and any step already stamped programmatic, are never downgraded, because the judge having nothing to quote says nothing about that measurement. Rollback switch: FT_VERIFY_EVIDENCE=0 — it changes the consequence only, the contract is still sent.

scripts/verdict_source.py stamps every step with a judge: programmatic (a *_verify helper, a typed assertion, a KPI compare, a polling match, the nav executor failing to find an element), keyword (the expected text’s words found in the page), llm (Claude read a snapshot and decided), or none (nothing was asserted). finalize() fills in whatever the verdict sites did not stamp, from the description markers and the step type, always erring on the weaker source — the report must never claim more rigour than there was. A run’s judge is the weakest among its asserting steps.

The run status folds as FAIL > INCONCLUSIVE > PASS, and both facts reach the UI: the amber pill, the judge tag, the Measured passes KPI and the Pass · not measured filter all read these two fields.

On the first failing step the runner keeps one viewport image per run (FT_FAILURE_SCREENSHOT, on by default). It also keeps the device parameter tree (FT_FAILURE_TREE_SNAPSHOT, on by default) when the run applied parameters to the emulator. Both files are written next to the run report as <run_id>-failshot.png or .jpg and <run_id>-failtree.*. The runner serves them to the owner from GET /api/runs/{run_id}/failure-screenshot|failure-tree. The screenshot is kept on failed. The tree is kept on failed or error.

The failure image has two sources. The run field failure_screenshot_source names the source.

The runner uses a step image only when the final result of that step carries the mark shot. A repeated segment therefore never reuses an old image. Before each send, the runner also deletes the old images of the segment.

The first inconclusive step gets its picture from the same step images. The runner keeps it as <run_id>-inconclusive.jpg and serves it from GET /api/runs/{run_id}/inconclusive-screenshot. The script never photographs a screen with a secret, such as the test mailbox settings.

The runner deletes the temporary folder when the run ends. At startup, it deletes folders older than six hours. It deletes a folder only when the name is a run id and the folder holds nothing but step-<N>.jpg files. It never follows a link. The folder root /tmp/ft-stepshots is fixed.

FT_FAILURE_SCREENSHOT=0 switches off the failure image of both sources. FT_STEP_SCREENSHOT=0 switches off the step images and the inconclusive image.

run_test.py decides how to execute a test from its step types (run_test.py:64-124). A single tuple, KNOWN_STEP_TYPES, is the one source of truth for every deterministic ("two-mode") step type. It currently holds 23: nav, interact, phase, emulator_action, acs_task, db_query, log_check, spa_task_verify, cred_length_verify, chat_ask, ai_step, explore, and the eleven programmatic verifiers — duplicate_user_verify, unused_models_scope_verify, file_on_server_verify, device_file_transfer_verify, user_group_permission_verify, ext_system_scope_verify, domain_setting_scope_verify, ext_system_readonly_verify, device_domain_move_verify, event_urls_verify, group_update_import_verify. OWN_SEGMENT_STEP_TYPES is derived from it (everything except nav / interact) so each such step always gets its own segment instead of being grouped with a same-type neighbour.

should_use_two_mode(steps) is the gate: