Troubleshooting

Symptom: the run is neither green nor red. One or more steps are marked with a description starting [not measured] ….

This is the evidence gate, live since 2026-09-03. An LLM verdict must quote, verbatim, the text from the captured page that decides the claim, and the runner checks that the quote really occurs in what it captured. A PASS that quotes nothing — or that was reached over an empty capture — is not a product failure and is not a pass either, so it is recorded as inconclusive. A FAIL is applied whatever it quoted: a false red is cheap and visible, a false green is neither.

What to do:

A deterministic verdict is never downgraded: a description prefixed [verify], [polling] or [empty-state] means a helper measured the claim itself, and the judge having nothing to quote says nothing about that measurement.

Rollback, if the gate itself is ever suspected: set FT_VERIFY_EVIDENCE=0 on the worker. It changes only the consequence — the judge is asked for evidence either way.

The verdict tag beside a run says what looked at the claim, not whether the claim held. Unverified means nothing asserted anything (a bare navigation, an auto-filled step, a verify helper that was unavailable). LLM-judged means a model read the page and decided.

Neither is an error, but neither is proof. The Test cases list shows the same information per case as a claims chip (3/7 measured), and the run history has a Pass · not measured filter to sweep exactly these. Cases that keep coming back LLM-judged are candidates for a typed *_verify step.

A failure carrying a known: <bucket> badge is in the known-issues registry — somebody already investigated it and recorded what it is. Read the badge title before spending time on it; GET /api/known-issues returns the whole registry with the bucket, the note and the action for each case.

Symptom: the phase stepper sits on Execute for several minutes; MCP Log shows the same browser_snapshot repeating without action.

A Claude CLI auth failure raised mid-run by the agent’s LLM calls.

Since 2026-07-13 the workers authenticate with a long-lived CLAUDE_CODE_OAUTH_TOKEN set in the Portainer stack environment (from claude setup-token, valid about a year). It overrides any stale .credentials.json on the bind mount, so there is no periodic refresh to run and no laptop-side scheduled task — the old one was removed. A 401 here means that token was revoked, rotated, or never set on this stack.

Symptom: the :8083 UI rejects your sign-in, or API calls come back 401 Unauthorized from the gateway (not from a running test).

This is unrelated to the Claude 401 authentication_error above, which is about the agent’s LLM OAuth token, not your UI session. See Accounts, Roles & Audit.

Symptom: starting a run returns 409 with a message like the emulator for env '<env>' is busy.

Cause: another run already holds that environment. Exclusivity is keyed on the env’s physical emulator backend (emulator_url), so two profiles that point at the same emulator block each other, while envs on distinct emulators run in parallel.

Fix: wait for the in-flight run to finish, or pick another environment in the first dropdown of the Test cases run bar. See gateway.adoc#selecting-env.

Run history and active runs are owner-scoped: each account sees only the runs it started. Admins are the exception — they see all runs. If your history looks empty or a colleague’s run is missing, you are simply logged in as a non-owning, non-admin account. See Accounts & Roles.

Two things are deliberately not owner-scoped, and they are the right way to find out whether a stand is free:

Two independent reasons, and only the first is about roles:

The always-visible sections are Overview, Test cases, Runs and Editor.

The live view is now gateway-proxied and owner-gated — the old direct noVNC host port was removed. You can only watch the browser of a run you currently own on that stand; the gateway closes the noVNC socket otherwise. Make sure a run of yours is actually active, that you are logged in as its owner, and then open the Live browser button.

Symptom: the run is marked error immediately after start; the Output pane shows a line like:

Cause: the YAML’s kpi_type requires TR-181 paths that the configured tree_file does not contain. The pre-flight (see architecture.adoc#preflight) refuses to start a run that is guaranteed to fail later in verify because the data simply isn’t there.

Fixes, in order of preference:

A second, non-fatal validator writes [tree-coverage:intent] lines to the same pane. Those describe what the case’s intent needs and never stop a run.

If the same line is [tree-coverage] WARN instead of CRITICAL, the run proceeds but emulator_setup.parameters reference paths that aren’t in the tree — the emulator will accept them but the parameter will not be visible to the backend until a corresponding object instance is created.

Symptom: after changing the URLs in Settings to point at a different environment (e.g. from test-agent to dev-mysql), runs go to the new portal but the device the test expects is missing — the runner can’t find it in the device list, or the portal shows no parameters.

Root cause: portal navigation uses base_url_* and direct emulator REST queries use emulator_url — both honour Settings immediately. But the device-creation path goes through the mcp-tr-emul sidecar, which historically had its target ACS hard-coded.

Fix: set both Emulator MCP URL and ACS URL in Settings so they match the new environment:

After saving Settings, no restart is needed — the next run picks up the new URLs.

The TestRail case has no XML tree attached and the converter fell back to a template that had no parameter requirements.

Programmatic verifier scanned the DOM and didn’t see the expected text. Check:

The widget value computed by the live backend differs from what the TestRail test expects. Order of investigation:

The portal shows whatever the ACS has cached from the last CWMP Inform. If the emulator was reset but no Inform was triggered, the portal lags.

Selector text doesn’t match what the portal shows. Two common reasons:

POST /api/testrail/import answers 409 import already running (job …). Only one import may be in flight per worker. Poll GET /api/testrail/import/jobs/{job_id} for the running one and wait for it — the modal does this for you and shows Created / Skipped / Errors when it lands.

In rare cases the Claude CLI subprocess survives Force Stop. The next run will queue but never start because Playwright is still bound to the previous session.

Each failure in a batch report is bucketed into a category by the post-run analyser. This drives the doughnut chart in the Batch Report dialog and the per-row label in the results table.