Environments & the Gateway

The AI Test Agent is no longer a single container on :8083. That port is now a gateway that authenticates every caller and routes each run onto a pool of per-environment worker containers. This page explains the topology, what an "environment" is, how you pick one per run, and the scheduling rules that keep two runs from colliding on the same emulator.

At a glance

  • One public endpoint — :8083 is the gateway. Workers are addressed only by the gateway, not by users.

  • Login is mandatory. Every request needs a session token except the allowlist (/, /health, /api/auth/login|logout, /novnc/*).

  • An "env" is a saved settings blob, stored centrally on the gateway — not a set of URLs you hand-edit per retarget.

  • Pick the env per run via the Env: dropdown on the Tests tab. The gateway materializes that env onto a free worker and runs there.

  • Runs are isolated per account. You see your own runs / history / live view; admins see all.

Why a Gateway

The old model was one test-runner container that you retargeted by editing all four base URLs in Settings before a run. That serialised everyone onto one emulator and one Chromium, and gave no way to run two environments at once.

The gateway (test-runner-gateway image) replaces that. It is the only process users talk to. Behind it sit one or more workers — each an ordinary test-runner container (plus its mcp-tr-emul and bbf-tree-mcp sidecars) that is env-agnostic: it holds no fixed environment of its own. The gateway pushes the chosen env’s settings onto a worker just before it starts the run.

Authentication lives entirely in the gateway. gateway_service/app.py runs an auth middleware that rejects any request without a valid Bearer session token, except the allowlist above and the /novnc/* asset prefix. The gateway then forwards the caller’s identity to the worker as X-Forwarded-User / X-Forwarded-Role headers, signed with the shared X-Gateway-Secret.

Topology

Component Role

Gateway (:8083)

The only public port. Hosts the SPA, terminates auth, owns the central user / stand / profile database (run-reports/test_runner.db), schedules runs, and reverse-proxies everything else to a worker.

Worker (test-runner container)

Executes runs. Env-agnostic — the gateway materializes a profile’s settings onto it per dispatch via PUT /api/settings. Runs TRUST_GATEWAY=1 so it skips its own login and trusts the gateway-forwarded identity. Registered in the gateway as a stand (name, base_url, novnc_url, enabled).

Per-env emulator wrapper (mcp-tr-emul-<env>)

Optional mcp-tr-emul sidecar deployed for an env that targets its own emulator (e.g. kira3 → qa11m). Auto-provisioned when the profile is first saved (see Portainer Auto-Provisioning).
Workers reach the gateway-forwarded headers as the source of truth for who is running. A worker never sees a user password — only the signed identity headers. The X-Gateway-Secret must match on both sides or the worker rejects the forwarded identity.

Environment Profiles

An environment is a named JSON settings blob stored in the gateway’s profiles table (name, settings_json, created_by, updated_at). The settings shape is identical to the per-worker Settings form (support / management / backend / emulator / ACS URLs, credentials, MCP URLs, …) — but it lives centrally instead of on one container.

Manage profiles from the Settings tab, which is now an env-profile editor:

Action Effect

Create / Save-as

POST /api/gateway/profiles (admin). Writes a new profile or overwrites an existing one by name.

Export / Import

Round-trips a profile’s JSON so envs can be copied between deployments.

Delete

DELETE /api/gateway/profiles/{name} (admin). With ?teardown=true it also best-effort deletes that env’s provisioned mcp-tr-emul-<name> wrapper stack.
Listing profiles (GET /api/gateway/profiles) is allowed for any logged-in user so the Env: dropdown can populate; create / read-full / delete are admin-only.

Selecting an Env per Run

On the Tests tab, an Env: dropdown sits next to the run controls. Your choice is remembered in localStorage, so it persists across reloads — see running-tests.adoc#env-select.

When you start a run, the SPA calls POST /api/gateway/run (or /api/gateway/run/batch) with {env, test_file|test_files, model?}. The gateway then:

  1. Picks a free worker from the enabled pool.

  2. Materializes the env — PUT /api/settings onto that worker (as the gateway admin) with the profile’s settings.

  3. Starts the run on that worker as you (your forwarded identity), and records run_id → worker so later reads / WebSocket progress route back to the same worker.

This replaces the old "edit all four Settings URLs to retarget" workflow — you no longer mutate a shared container to switch environments; you just choose a different env in the dropdown.

Scheduling Model

The scheduler (gateway_service/scheduler.py) enforces two independent constraints.

Env-exclusive + emulator-mutex

A worker is "free" only when it is enabled, not already busy, and reports zero active runs upstream. On top of that, exclusivity is keyed on the env’s physical emulator (emulator_url), not its name:

  • Two different envs that point at the same emulator (e.g. Default, qa65, kira2 all on tr069-emulator.test-agent) share one device serial per case, so they are mutually exclusive — the second submit gets 409 "the emulator for env '…' is busy".

  • Envs on distinct emulators (e.g. kira3 → qa11m) carry different mutex keys and run in parallel.

Condition Response

Requested env’s emulator already has a run

409 — env busy, retry shortly (EnvBusy).

No enabled worker is free

503 — all workers busy, retry shortly (WorkerBusy).
Keying the mutex on the emulator (not the env name) turns a silent device-serial collision into a clean 409. If a profile’s emulator_url lookup ever fails, the scheduler degrades to a per-env-name mutex rather than breaking dispatch.

Per-account isolation

Runs, history, force-stop, and the live browser are scoped to the calling user. The gateway fans these reads out to every enabled worker as the caller; the worker endpoints are owner-scoped, so you only ever see your own runs. Admins see all. The reads affected: GET /api/runs, GET /api/runs/active, the GET /api/run/batch* family, POST /api/run/stop, and the noVNC live view (GET /api/live/stand + /novnc/{stand}/websockify, gated so you may watch a worker’s browser only while you own the active run on it).

Workers Tab

Workers (stands) are managed from the Workers tab (admin-only):

Action Endpoint

Register a worker

POST /api/gateway/stands (name + base_url, optional novnc_url).

Enable / disable

PATCH /api/gateway/stands/{name} {enabled}. Only enabled workers are scheduled onto.

Add Worker (provision)

POST /api/gateway/workers — deploys a new worker container (see Portainer Auto-Provisioning) and registers it disabled so you can enable it deliberately.

Deregister / teardown

DELETE /api/gateway/stands/{name}; with ?teardown=true also deletes the provisioned tr-worker-<name> stack.

Portainer Auto-Provisioning

gateway_service/provision.py can deploy stacks straight into Portainer. It is best-effort and inert when Portainer credentials are unset — provisioning simply does nothing and the save / add still succeeds.

Trigger Provisions

New env profile with an emulator_url (and no emulator_mcp_url)

A per-env mcp-tr-emul-<env> wrapper stack on a free host port, pointed at that env’s emulator_url / acs_url. The resulting MCP URL is written back into the profile.

Add Worker

A tr-worker-<name> stack: a test-runner container plus its mcp-tr-emul and bbf-tree-mcp sidecars, with TRUST_GATEWAY=1, the shared GATEWAY_SECRET, and the per-stand persistent volumes wired in.
Provisioned stacks publish host ports the gateway then addresses by base_url. Teardown (?teardown=true on profile or stand delete) removes the matching Portainer stack.

Invariants

Read these before changing the gateway or adding a worker:

  • GATEWAY_SECRET must match across the gateway and every worker. It signs the forwarded-identity headers; a mismatch makes workers reject all proxied requests.

  • Workers run TRUST_GATEWAY=1. This disables their local login and makes them trust X-Forwarded-User / X-Forwarded-Role from the gateway. A worker in this mode must never be exposed to users directly — it has no auth of its own.

  • Per-stand persistent volumes keep each worker’s state isolated: test-cases-<STAND>, run-reports-<STAND>, qa-notes-<STAND>, and /opt/mcp-shared-<STAND> (serial isolation). They survive redeploys.

  • Shared Claude OAuth — the ${CLAUDE_CONFIG_DIR:-/root/.claude} volume is mounted read-write on each worker so a token refresh on one propagates to all.

Gateway Endpoints

The gateway-specific surface (everything else falls through to the catch-all proxy → primary worker). Full request/response detail in REST API.

Endpoint Purpose

POST /api/auth/login / logout, GET /api/auth/me

Session auth. Login returns a Bearer token; all other calls require it.

GET / POST /api/gateway/profiles, GET / DELETE /api/gateway/profiles/{name}

Env profile CRUD (DELETE accepts ?teardown=true).

GET / POST /api/gateway/stands, PATCH / DELETE /api/gateway/stands/{name}

Worker (stand) registration, enable / disable, deregister.

POST /api/gateway/workers

Auto-provision a new worker container.

POST /api/gateway/run, POST /api/gateway/run/batch

Schedule a single / batch run onto a free worker for the chosen env.

GET /api/runs, GET /api/runs/active, GET /api/run/batch*

Per-account fan-out reads across enabled workers.

GET /api/live/stand, POST /api/run/stop, WS /ws/run/{run_id}, /novnc/{stand}/*

Live-view target resolution, owner-scoped stop, run progress stream, noVNC.

See also: Settings (the env-profile editor and field reference), Running Tests (picking an env per run), and API Reference (gateway endpoint details).