Troubleshooting

Common failure modes and how to diagnose them. For an organised debugging playbook (TestRail → backend → frontend → ArgoCD logs), see the Common Pitfalls section of the User Guide.

Where to look first

  1. Live Browser (the button in the UI — gateway-proxied noVNC) — what the browser is actually doing right now, for the run you own.

  2. Runner → MCP Log — every Playwright call with args; great for "why did the click miss".

  3. Runner → Output — full Claude reasoning + verifier diagnostics.

  4. Live spec — C:\F_Projects\ft-device-network\docs for KPI formulas before assuming a backend bug.

  5. QA Notes — known-bug / flap notes get auto-injected if their scope matches; check QA Validation Notes if a verifier verdict looks wrong.

Run Hangs in Executing

Symptom: the phase stepper sits on Executing for several minutes; MCP Log shows the same browser_snapshot repeating without action.

  1. Open the live view via the Live Browser button (gateway-proxied noVNC for the run you own).

  2. Identify what the browser is showing — is the dialog the agent expects there?

  3. If the model in use is Sonnet, force-stop and rerun with Opus. Sonnet occasionally stalls on multi-step interact loops; Opus is the default for this reason.

  4. If Opus also stalls, the test prompt is ambiguous — open Editor, refine the failing step’s action / expected, save, rerun.

401 authentication_error

Claude CLI OAuth token expired (~30 days). Copy local ~/.claude/.credentials.json into the test-runner-agent container (run claude auth login locally first).

This is the Claude OAuth error, raised mid-run by the agent’s LLM calls. A 401 returned by the UI login is a different problem — see Login Fails / 401 on the UI (Not Claude OAuth) below.

Login Fails / 401 on the UI (Not Claude OAuth)

Symptom: the :8083 UI rejects your sign-in, or API calls come back 401 Unauthorized from the gateway (not from a running test).

  • 401 on login means a wrong username/password, a disabled account, or an expired bearer-token session — log in again.

  • After ~5 failed attempts within a minute the gateway rate-limits you and returns 429 Too many attempts; wait a minute and retry.

  • On a fresh deployment the only account is the bootstrap admin admin / admin. If that was changed and lost, an admin must reset it from the Users tab (or re-bootstrap the gateway).

This is unrelated to the Claude 401 authentication_error above, which is about the agent’s LLM OAuth token, not your UI session. See Accounts, Roles & Audit.

"Env Is Busy" / 409 When Starting a Run

Symptom: starting a run returns 409 with a message like the emulator for env '<env>' is busy.

Cause: another run already holds that environment. Exclusivity is keyed on the env’s physical emulator backend (emulator_url), so two profiles that point at the same emulator block each other, while envs on distinct emulators run in parallel.

Fix: wait for the in-flight run to finish, or pick a different env in the Env: dropdown on the Tests tab. See gateway.adoc#selecting-env.

I Can’t See My Run / Someone Else’s Run Is Hidden

Run history and active runs are owner-scoped: each account sees only the runs it started. Admins are the exception — they see all runs. If your history looks empty or a colleague’s run is missing, you are simply logged in as a non-owning, non-admin account. See Accounts & Roles.

A Tab Is Missing (Settings / Users / Workers / Activity)

These four tabs are admin-only and are hidden for editor and viewer roles. If you expect to see them, you are not signed in as an admin — ask an admin to raise your role or perform the action for you. The always-visible tabs are Dashboard, Tests, Runner, Editor, Notes and Learning.

Live Browser Blank / noVNC Won’t Connect

The live view is now gateway-proxied and owner-gated — the old direct noVNC host port was removed. You can only watch the browser of a run you currently own on that stand; the gateway closes the noVNC socket otherwise. Make sure a run of yours is actually active, that you are logged in as its owner, and then open the Live Browser button.

Run Fails With [tree-coverage] CRITICAL

Symptom: the run is marked error immediately after start; the Output pane shows a line like:

[tree-coverage] CRITICAL: tree=181.xml kpi=wlan-health-de-mesh
  | missing required prefixes: Device.WiFi.DataElements

Cause: the YAML’s kpi_type requires TR-181 paths that the configured tree_file does not contain. The pre-flight (see architecture.adoc#preflight) refuses to start a run that is guaranteed to fail later in verify because the data simply isn’t there.

Fixes, in order of preference:

  1. Switch tree_file to a template that covers the required data model (e.g. 181-mesh-de.xml for wlan-health-de-mesh).

  2. If no template fits, click Generate Tree in the Editor with the test text in place and let bbf-tree-mcp produce a fresh tree.

  3. As a last resort, hand-edit the XML under /app/templates/setups/ to add the missing object instances. Keep the tree minimal — extra paths can surface accidental defaults that change scoring.

If the same line is [tree-coverage] WARN instead of CRITICAL, the run proceeds but emulator_setup.parameters reference paths that aren’t in the tree — the emulator will accept them but the parameter will not be visible to the backend until a corresponding object instance is created.

The validator is schema-aware — it normalises any concrete instance index (AccessPoint.2.AssociatedDevice.3.*) to .{i}. and matches against the schema-level path set extracted from the tree, so dynamic instances created at runtime by the emulator no longer trigger false-positive WARNs. For neighboring_diag tests the validator also auto-swaps tree_file to a template that already declares NeighboringWiFiDiagnostic.Result.{i} if the configured tree lacks those instances.

Switched Settings to a New Env, Tests Fail / Device Not Found

Symptom: after changing the URLs in Settings to point at a different environment (e.g. from test-agent to dev-mysql), runs go to the new portal but the device the test expects is missing — the runner can’t find it in the device list, or the portal shows no parameters.

Root cause: portal navigation uses base_url_* and direct emulator REST queries use emulator_url — both honour Settings immediately. But the device-creation path goes through the mcp-tr-emul sidecar, which historically had its target ACS hard-coded.

Fix: set both Emulator MCP URL and ACS URL in Settings so they match the new environment:

After saving Settings, no restart is needed — the next run picks up the new URLs.

Imported Test Has Empty emulator_setup

The TestRail case has no XML tree attached and the converter fell back to a template that had no parameter requirements.

  1. Open Editor, click Generate Tree with the test text in place.

  2. Or set tree_file: 181.xml (or another template) and add parameters manually for the values the test expects to read.

Step Fails With "Expected value not found"

Programmatic verifier scanned the DOM and didn’t see the expected text. Check:

  1. Did the runner navigate to the right page? Use MCP Log — the last browser_navigate URL should match the widget being verified.

  2. Is the widget rendering the value you expect? Open VNC after a failed run to confirm.

  3. Does the backend actually have the data? Hit the emulator API:

    curl 'http://tr069-emulator.test-agent.rd.friendly-tech.com/apiV2/parametersByPrefix?serial=test-runner-001&prefix=Device.WiFi.AccessPoint.1.AssociatedDevice.1.'

  4. Or query the device tree from the portal via FtApiClient. The portal tree reflects what the backend received over CWMP — if a parameter is in the emulator API but missing in the portal tree, the ACS didn’t pick it up.

Backend Score Doesn’t Match TestRail Expected

The widget value computed by the live backend differs from what the TestRail test expects. Order of investigation:

  1. Spec docs — C:\F_Projects\ft-device-network\docs\src\modules\ROOT\pages\ (mesh/client-health-score.adoc, network-score.adoc, etc.). Source of truth for KPI formulas, data-model paths, weights, fallback paths.

  2. Backend Java sources — NetworkScoreProcessor and *ScoreCalculator`s in `C:\F_Projects\ft-device-network. Implementation of the formulas with exact border values.

  3. Frontend — C:\F_Projects\oneiot_angular. How the value is rendered (rounding, transformation).

  4. ArgoCD logs — ft-device-network pod. Look for NetworkScoreProcessor: TH: …​, INT: …​, NH: …​ lines for WLAN; tail to see what the backend computed and whether the ACS delivered the new state.

The expected value in TestRail is the source of truth — never edit it. Fix the converter or emulator parameters.

Tree Pushed But Portal Shows Old State

The portal shows whatever the ACS has cached from the last CWMP Inform. If the emulator was reset but no Inform was triggered, the portal lags.

  1. Check the MCP Log for invoke_event after the tree push.

  2. Manually trigger via the emulator API /apiV2/invokeEvent?serial=…​&type=2 PERIODIC.

  3. Or restart the emulator instance via start_protocol_emulator.

Sidebar / Tab Click Times Out

Selector text doesn’t match what the portal shows. Two common reasons:

  1. Wrong portal — the YAML says portal: support but the case is for management. Check the top-level field.

  2. Localisation / casing — nav_target is matched case-sensitively. Open VNC, copy the exact label, paste into nav_target.

Force Stop Didn’t Clean Up

In rare cases the Claude CLI subprocess survives Force Stop. The next run will queue but never start because Playwright is still bound to the previous session.

  1. docker exec test-runner pkill -f claude (via Portainer terminal).

  2. Or restart the stack from Portainer.

TestRail Modal Shows "Disconnected"

  1. Open Settings → TestRail.

  2. Verify URL, Email, API Key.

  3. Save. The pill should turn green within a couple of seconds.

  4. If still red, check the agent logs for HTTP errors against TestRail — often a stale API key after rotation.

QA Note Saved But Verify Still FAILs

You wrote a known_bug / warning note that should make the agent accept the observed value, but the next run still records FAIL.

  1. Open the Notes tab and click Reindex. Read the summary:

    1. fetched: 0 → knowledge-mcp can’t reach the test-runner pull URL. Verify QA_NOTES_SOURCE_URL on the knowledge-mcp stack.

    2. errors > 0 → Qdrant upsert failed (embedding service or Qdrant down).

    3. unchanged: N for every note → the index already has the same (id, version, payload-hash), so nothing was re-embedded — your note is there.

  2. If reindex says everything is fine, check the scope. The verify path only injects notes whose scope matches the running step:

    1. test_ids must contain the test’s id exactly (e.g. C7626, not 7626 or C7626-1).

    2. widgets is matched against nav_target of nav_action: widget steps in the segment (case-sensitive — Network connected clients, not network connected clients).

    3. kpi_types / topologies match the converter-set kpi_type / wifi_health_topo in the YAML — open the YAML in Editor to see what was actually detected.

  3. If the converter Stage 2 prompt should see the note but doesn’t, your test was last imported before the note existed. Reimport with Force refresh (TestRail modal) or pass force_refresh=true on the /api/testrail/reimport endpoint.

  4. As a last resort, search via the knowledge-mcp REST to confirm the note is queryable:

    curl -X POST http://ai.friendly-tech.com:7030/search/qa_notes \
  -H 'Content-Type: application/json' \
  -d '{"query":"<short keyword from your note>","test_id":"C7626","limit":3}'

    Empty result means the scope/index disagreement — re-check the YAML on disk (GET /api/notes/<id>).

See QA Validation Notes → Troubleshooting for the full table.

Reading the Batch Report Failure Categories

Each failure in a batch report is bucketed into a category by the post-run analyser. This drives the pie chart in the Batch Report modal and the per-row label in the results table.

Category Meaning

setup

emulator_setup failed before any UI step ran (tree push, parameter write, ACS Inform). Typically a missing tree file, malformed parameter map, or the emulator container being down.

mcp_timeout

The Playwright MCP didn’t respond within the per-call timeout. Usually the browser hung on a snapshot or a long network call. Retry or restart the runner if it persists.

nav

A nav step couldn’t find its target. Wrong nav_target text, wrong portal, or the page didn’t load before the click.

verify

Steps executed but the verifier (DOM scan or LLM fallback) didn’t see the expected value. Usually a real backend / converter mismatch — start with the debugging order in user-guide.adoc#common-pitfalls.

converter

The Smart Converter raised an error on this YAML before the run could start (missing required field, malformed parameters map, etc.). Reimport the case after fixing.

stopped

The user pressed Force Stop (or the Force Stop API was called). Not a real failure.

unknown

Couldn’t classify. Check the run’s Output pane for the raw error.