Architecture

Engineer-oriented overview of how the AI Test Agent is wired internally.

Components

In the multi-stand deployment the container described here is one of several interchangeable workers sitting behind a central gateway (see Multi-Stand Gateway). Each worker is one container running several long-lived processes plus on-demand subprocesses:

Process Role

test_server.py (FastAPI on :8083)

HTTP / WebSocket gateway. Hosts the UI, exposes the REST API, manages the in-memory run registry, persists batch reports, runs the TestRail auto-sync loop and brokers MCP connections.

run_test.py (subprocess per run)

The actual test executor. Loads the YAML, builds the system prompt, drives the Playwright MCP via deterministic JS for nav segments and via Claude CLI for interact segments, runs the verifier and emits the JSON report.

Claude Code CLI (Max subscription)

Used as a subprocess by run_test.py for interact steps and the LLM verifier fallback. OAuth token at /root/.claude/.credentials.json. Model aliases resolve to: haiku → claude-haiku-4-5-20251001, sonnet → claude-sonnet-4-6, opus → claude-opus-4-6 (see claude_config.py). Both the UI and the API endpoints (POST /api/run, POST /api/run/batch) default to opus — Haiku was dropped from the converter UI as well to reduce multi-step jitter.

Playwright MCP (Node, :3100/sse)

Drives Chromium running under Xvfb. Started via the start.sh entrypoint.

Xvfb + x11vnc + noVNC (:6080)

Headed Chromium displayed via VNC for live debugging.

mcp-tr-emul (sidecar container, :8082)

Wraps the TR-069 emulator HTTP API as MCP tools (set_parameters_batch, start_protocol_emulator, invoke_event …). Shares /opt/mcp-shared/ with the test runner.

bbf-tree-mcp (sidecar container, :7020)

Synthesises TR-181 / TR-098 trees from natural language (used by the Editor’s Generate Tree button) and also serves the converter Stage 2 LLM prompt with TR-181 spec snippets via fetch_bbf_spec_context.

knowledge-mcp (separate stack :7030)

Stateless retrieval over three Qdrant collections: ft_docs_v2 (AsciiDoc + Java spec corpus), ft_ui_map (widget catalog auto-extracted from the Angular widgets-dashboard component), and ft_qa_notes (team-authored behavioural notes, pull-pulled from the test-runner volume on a webhook). Used by both the converter (TR-181 spec context + QA notes during Stage 2 LLM enrichment) and the test-runner Claude (proactive widget catalog
search_qa_notes during interact steps, plus auto-inject of QA notes into LLM verify prompts). Image: hub.friendly-tech.com/mcp/knowledge-mcp:latest. See QA Validation Notes for the user-facing flow.

Multi-Stand Gateway

The worker above is normally fronted by a single public gateway (gateway_service/, port :8083 — the only port exposed to the outside). The gateway owns identity and routing; the workers do the actual test work and are not published to the host (they are reached over the test-runner-net Docker network). One gateway, N interchangeable workers, many env profiles.

Module Role

app.py

FastAPI front door. Central auth (login → an opaque bearer session token, plus an in-memory login throttle), per-request stand / worker resolution, and all local routes (auth, users, profiles, stands, run dispatch, fan-out reads) registered before the catch-all reverse-proxy so they win.

proxy.py

HTTP reverse-proxy helper. Strips hop-by-hop and client-supplied identity headers, re-injects the gateway-owned X-Forwarded-User / X-Forwarded-Role / X-Gateway-Secret, and streams the worker response back unbuffered.

scheduler.py (env-per-run)

Picks a free worker for each run and materializes the chosen env’s settings onto it before dispatch. Exclusivity is keyed on the env’s physical emulator (emulator_url), not the env name: two envs that share one emulator (one device serial) are mutually exclusive and the second submit gets HTTP 409, while envs on distinct emulators run in parallel. No free worker → 503. A background reaper frees the env once the worker reports its run/batch terminal.

WS / noVNC relay (app.py + novnc.py)

/ws/run/{run_id} relays progress frames bidirectionally to the worker that owns the run. novnc.py proxies the live-view: asset passthrough is unauthenticated, but the VNC-stream WebSocket is gated per account — you may watch a worker’s browser only while you own the active run on it (_caller_owns_active_on).

provision.py

Portainer auto-provisioning. Creates per-worker tr-worker-<name> stacks and per-env mcp-tr-emul-<env> emulator-wrapper stacks on demand (host-port allocation, idempotent stack-exists guard, best-effort teardown).

Per-run reads, batch status / reports and stop are routed to (or fanned out across) the worker that actually owns the caller’s run — owner-scoped, as the caller — so a run that lands on a non-primary worker is still visible and stoppable, and one account can never see or stop another’s run. See Environments & the Gateway for the deployment topology and day-to-day operations.

Persistence Layer

The gateway and the workers share two small SQLite data layers (pure data modules — no FastAPI imports, unit-testable in isolation) in a single file at run-reports/test_runner.db:

Module Tables

scripts/auth_db.py

users (PBKDF2 password hashes; role admin / editor / viewer), sessions (bearer tokens), activity (audit log), stands (worker registry), profiles (per-env settings) plus case_meta / case_history (per-case ownership + edit history). See Accounts, Roles & Audit.

scripts/case_db.py

projects — first-class project containers for test cases (a TestRail-style project registry); pairs with `auth_db’s case metadata. See Organizing Cases: Projects & Sections.

Data Flow

Conversion (TestRail → YAML)

TestRail REST → api_case_to_xml_element()
   → convert_case() | convert_case_scenarios()        (dual code path)
      → Stage 1: regex extraction of params from preconditions / steps
      → Stage 2: LLM enrichment (Sonnet/Opus) — fills missing TR-181 paths
      → Stage 3: score_simulator.py — Python port of backend KPI math,
                 verifies that resulting parameters reach the expected
                 widget value; retries with adjusted params if not
   → mesh-interference tree auto-switch (when applicable)
   → _normalize_optical_scale post-processing
   → write tests/cases/<case>.yaml

The converter has two independent code paths: convert_case() for simple single-scenario tests and convert_case_scenarios() for multi-phase tests. They build phases independently — fixes must be applied to both.

Run-Time Pre-Flight

Before each run, test_server.py performs three checks. Their output is visible in the Output pane of the Runner tab and on the WebSocket stream:

Check Behavior

Tree coverage (_validate_tree_coverage)

Reads the test’s kpi_type (e.g. wlan-health-de-mesh, network-score-wan) and looks up the required TR-181 prefixes in _REQUIRED_TREE_PREFIXES. If any prefix is missing from tree_file, emits a [tree-coverage] CRITICAL line and fails the run. Param-vs-tree matching is schema-aware: any concrete instance index in a YAML param (e.g. AccessPoint.2.AssociatedDevice.3.*) is normalised to .{i}. and matched against the schema-level set of paths extracted from the tree, so dynamic instances created by the emulator at runtime do not produce false-positive WARNs. For neighboring_diag-style tests the validator also auto-swaps tree_file to a template that already declares NeighboringWiFiDiagnostic.Result.{i} if the configured tree lacks those instances.

ClickHouse purge

Wipes the per-device monitoring rows across all 9 network_* tables in ftacs_qoe_ui_data (clients, scores, issues, wifi_band, interfaces, device parameters, …) so history accumulated by a previous test cannot leak into the next score calculation. Implemented as ALTER TABLE …​ DELETE mutations with mutations_sync=2 (hard delete that physically rewrites the parts and waits for completion). The earlier lightweight-delete path was unreliable on ClickHouse 24.8 — tombstones (_row_exists=0) were intermittently ignored by SELECT plans during active inserts, so the dashboard kept seeing "deleted" rows mid-test even when the purge logged 9/9 OK. The hard mutation also validates the response body (catches ClickHouse DB::Exception returned with HTTP 200) and re-runs count() post-delete — only counts a table as purged when the rows are actually gone.

Concurrency guard

Refuses to start a new run while another is still active (the agent shares one Chromium / one emulator — concurrent runs corrupt state). The UI greys out the Run Selected button while a run is in flight.
kpi_type is set automatically by the converter and stored at the top level of the YAML (test_case.kpi_type). Hand-authored YAMLs that omit it fall back to a generic preflight that only checks tree_file exists and is non-empty.

Anti-Flap (PeriodicInform scoping)

After ensure_online, _run_emulator_setup may disable Device.ManagementServer.PeriodicInformEnable for the duration of the test. This is not unconditional — the disable applies only when the test contains CREATE_MONITORING in its ui_preconditions or any phases[*].preflight.

Why scoped:

  • Monitoring tests read state through ACS GPV polls. The autonomous Periodic Inform every PeriodicInformInterval (~90 s) carries a different param subset than the GPV. ACS cache-merges them (Not all data received → putting to cache → merging), NetworkProcessor oscillates between client-count states, and per-client Health varies between recomputes. The original symptom was C7626 failing with Health=70 after a long predecessor (e.g. C7625, ~14 min) in the same batch. Disabling the autonomous Inform makes monitoring GPV the single source of param updates and removes the flap.

  • Diagnostic tests (RUN_DIAGNOSTICS, RUN_INTERFERENCE_DIAGNOSTICS, the UI "Run diagnostics" button) and plain UI flows depend on the CWMP Inform cycle. ACS getDiagnosticRoot() returns null without periodic param discovery, and post-SPV state Informs stop firing on schedule. The earlier unconditional disable regressed C7628 / C20436; the scoped fix restored them.

Restoration is automatic between tests: the next run restarts the emulator with a fresh tree and PeriodicInformEnable=true returns. Do not add an in-test re-enable.

Conversely, execute_phase_step forces a Periodic Inform after set_params_batch for non-WiFi-config phases so backend monitoring sees the new state without waiting up to a full PeriodicInformInterval for the next pull. WiFi-config phases skip this because their SPV already triggers a Value Changed Inform on its own.

The setup pane logs one of these lines so the path is auditable:

[emulator] PeriodicInform disabled (anti-flap, monitoring test)
[emulator] PeriodicInform left enabled (no CREATE_MONITORING in test)

QA Notes Pipeline

Team-authored behavioural notes flow into prompts through a separate pull-pattern indexed alongside the spec corpus:

UI Save  ──► POST /api/notes
              ├── /data/qa-notes/<id>.yaml          (atomic write, source of truth)
              └── webhook POST knowledge-mcp/admin/refresh-notes
                       └── QaNotesIndex.refresh()
                              ├── GET test-runner:8083/api/notes/raw
                              ├── (id, version, payload-hash) diff
                              └── embed + upsert into Qdrant ft_qa_notes

Run / Reimport
   ├── converter Stage 2 (`_build_qa_notes_section`)  ─► search_qa_notes
   ├── LLM verify (`_fetch_qa_notes_block`)           ─► search_qa_notes
   └── interact loop proactive tool                   ─► mcp__knowledge__search_qa_notes

Failure of any leg degrades gracefully — the prompt is built as if the note didn’t exist. See QA Validation Notes for the user-facing workflow and the scope-filter semantics.

Execution (YAML → report)

YAML
 ├── emulator_setup
 │    ├── tree_file → factory_reset + push_tree (mcp-tr-emul)
 │    ├── delete_objects, instance_limits, wifi_client_limit (CWMP DeleteObject)
 │    ├── parameters → set_parameters_batch
 │    ├── ui_preconditions → FtApiClient REST (monitoring / group / diagnostics)
 │    └── phases[0] → applied if present
 │
 ├── steps (grouped by type)
 │    ├── nav segment   → nav_script_gen.py → batched Playwright JS via browser_run_code
 │    ├── interact segment → snapshot-action loop driven by Claude CLI
 │    └── phase step    → execute_phase_step() — applies phases[i+1] params via emulator
 │
 └── verify
      ├── programmatic — DOM scan from the rendered page
      └── LLM fallback — Claude reads page text + expected value

Two-Mode Routing (deterministic vs agentic)

run_test.py decides how to execute a test from its step types (run_test.py:64-124). A single tuple, KNOWN_STEP_TYPES, is the one source of truth for every deterministic ("two-mode") step type — nav, interact, phase, emulator_action, acs_task, db_query, log_check, spa_task_verify, cred_length_verify. OWN_SEGMENT_STEP_TYPES is derived from it (everything except nav / interact) so each such step always gets its own segment instead of being grouped with a same-type neighbour.

should_use_two_mode(steps) is the gate:

  • all steps are nav, or nav mixed with any other recognized type → two-mode: the steps are grouped into segments and each runs through its deterministic handler (nav → generated Playwright JS via browser_run_code, the others → their typed executors), with interact verify falling back to Claude only inside its own segment.

  • any unrecognized step type, or no nav step at all → the whole test falls through to the fully-agentic path where Claude drives every step.

a step type wired into the segment dispatch but missing from KNOWN_STEP_TYPES silently regresses the entire test to agentic mode (the C9614 spa_task_verify regression) — add new deterministic types in that one place. Because deterministic phrasing is what keeps a step in two-mode, how a TestRail case is written directly controls which path it takes — see Navigation Grammar.

MCP Integration

The agent talks to five MCP servers. Four are remote (mcp-tr-emul, bbf-tree-mcp, qdrant-mcp, knowledge-mcp) and one is local to the same container (playwright-mcp).

This 5-MCP wiring is per worker behind the gateway: every test-runner worker has its own local playwright-mcp and its own mcp-tr-emul / bbf-tree-mcp sidecars (an env’s emulator is pinned via the gateway-provisioned mcp-tr-emul-<env> wrapper), while qdrant-mcp and knowledge-mcp are shared singletons that all workers point at.

Server Used in Key tools

playwright-mcp

run_test.py (every step)

browser_navigate, browser_click, browser_run_code, browser_snapshot, browser_wait_for

mcp-tr-emul

EmulatorMCP wrapper, used both at preflight and inside phase steps

start_protocol_emulator, set_parameters_batch, get_parameter_value, invoke_event, search_parameters

bbf-tree-mcp

Editor → Generate Tree button (/api/generate-tree endpoint) and converter Stage 2 LLM prompt enrichment (fetch_bbf_spec_context) for TR-181 / TR-098 spec snippets.

generate_tree, edit_tree, provision_device, chat_bbf

qdrant-mcp

Legacy ft_code Qdrant access for the converter (off by default in current versions — see backend_rag in importing-tests.adoc#xml-api).

direct Qdrant collection access (ft_code)

knowledge-mcp

Stateless retrieval over ft_docs_v2, ft_ui_map, and ft_qa_notes. The converter calls it during Stage 2 LLM enrichment (TR-181 spec context plus team-authored QA notes). The test-runner Claude receives mcpknowledge* tools wired via the KNOWLEDGE_MCP_URL env so interact steps can ground themselves in the widget catalog and the QA notes. In addition, run_test.py auto-fetches relevant QA notes by scope (test_id, widget, kpi_type, topology) before building each LLM verify prompt — see QA Validation Notes.

mcpknowledgesearch_docs, mcpknowledgesearch_ui_map, mcpknowledgeget_widget_layout, mcpknowledgelist_widgets, mcpknowledgesearch_qa_notes

Smart Converter

scripts/convert_testrail_xml.py (~12 000 lines). Three deterministic stages with structured retries:

  1. Stage 1 — regex extraction. Scans preconditions and step text for recognisable patterns ("RSSI = -55", "OperatingStandard is 'ax'") and produces a draft parameters map.

  2. Stage 2 — LLM enrichment. If Stage 1 leaves gaps, prompts Claude with the test text plus TR-181 spec snippets retrieved from bbf-tree-mcp / knowledge-mcp (and, when explicitly enabled via backend_rag=true, Java snippets from the legacy ft_code Qdrant collection), asking for additional TR-181 paths to satisfy the expected value. Stage 2 is gated by an explicit skip-LLM whitelist (_should_skip_stage2) for KPI types that are fully solvable by Stage 1 — this keeps deterministic tests from inheriting LLM jitter. An idempotency cache keyed on the converter inputs short-circuits identical re-conversions; bypass it with Force refresh in the TestRail modal (or force_refresh=true on the API). When a team-authored QA note matches the test’s scope (test_id / kpi_type / wifi_health_topo), the converter appends a QA Validation Notes section to the LLM prompt via _build_qa_notes_section — see QA Validation Notes. Bumping _CONVERSION_CACHE_VERSION in scripts/test_server.py invalidates every cached entry on the next reimport (used when a new note is significant enough that older YAMLs should reflect it).

  3. Stage 3 — Score Verification. Runs score_simulator.py (Python port of the backend NetworkScoreProcessor / WlanScoreCalculator / etc.) against the candidate parameters. If the simulated score is within tolerance of the expected widget value, the YAML is committed. Otherwise the converter adjusts and retries.

The TestRail expected value is the source of truth — never edited. If the math says the expected is unreachable on the current backend (integer division, overlapping ranges), the converter flags the case in the import report instead of silently baking a workaround.

Persistent State

Path Volume

/app/tests/cases/

test-cases (YAML test cases — survives redeploy)

/app/run-reports/

run-reports (batch reports + settings.json)

/data/qa-notes/

qa-notes (QA Validation Notes YAMLs — source of truth for ft_qa_notes Qdrant collection)

/opt/mcp-shared/

host bind-mount (shared with mcp-tr-emul)

/root/.claude/

host bind-mount (Claude CLI OAuth credentials)
Under the multi-stand gateway each worker gets its own persistent volumes — test-cases-<STAND>, run-reports-<STAND>, qa-notes-<STAND> and a per-stand /opt/mcp-shared-<STAND> serial-isolation bind — so stands never share test cases, reports or device state. The Claude CLI OAuth volume (/root/.claude) is the exception: it is mounted shared read-write across all workers so a single token refresh propagates everywhere. Every worker runs with TRUST_GATEWAY=1 (it skips its own login and trusts the gateway’s forwarded X-Forwarded-User identity), and its GATEWAY_SECRET must match the gateway’s — a mismatch makes the worker reject every forwarded request.
After converter changes, reimport the affected cases — the test-cases volume is not wiped on redeploy.