User Guide

This guide walks through the day-to-day usage of the AI Test Agent from the QA engineer’s perspective. If you want a deep field-by-field reference for the in-browser test editor, jump to Editor Guide.

At a glance

  • Run a case → Tests tab, tick, Run Selected.

  • Import from TestRail → Editor → TestRail button (top right).

  • Edit a converted case → Editor, click the row on the left.

  • Document a known backend quirk → Notes → + Add Note. It will feed the verifier and the converter on its own next time.

  • Watch live what the browser is doing → Live Browser button (top right) opens the gateway-proxied noVNC view of the run you own.

Logging In

The :8083 endpoint is the gateway: it requires a login before the UI loads.

  1. Open http://ai.friendly-tech.com:8083/. A login overlay appears.

  2. Sign in with your account. On a fresh deployment the only account is the bootstrap admin admin / admin — change that password immediately (Users tab, admin only).

  3. On success the gateway issues a bearer-token session (kept in the browser); every API call carries it. After ~5 failed attempts in a minute the login is rate-limited (HTTP 429); a wrong username/password returns HTTP 401.

  4. Your role (admin / editor / viewer) decides which tabs and actions you see — see Accounts & Roles.

Roles in one line — a viewer can browse and read reports but cannot run or edit; an editor can author and run tests but cannot see the admin-only Settings, Users, Workers and Activity tabs; an admin sees everything.

Opening the UI

  1. After logging in, the UI brands itself as FT QA agent in the header.

  2. The header shows up to ten tabs: Dashboard, Tests, Runner, Editor, Notes, Learning, and (admin-only) Settings, Users, Workers, Activity. The four admin-only tabs are hidden for editor and viewer roles.

    FT QA agent header tabs

  3. The Learning tab is where the agent’s self-learned navigation fixes and recurring-failure signatures wait for a human to Accept (promote) or Dismiss — nothing is applied automatically.

  4. First-time setup: an admin opens Settings and verifies the base URLs and credentials, or — more commonly now — defines reusable environment profiles so anyone can pick the target env per run (see Picking the Environment Per Run). The defaults target test-agent.

    Settings tab — environment defaults

Picking the Environment Per Run

You no longer retarget the agent by editing all four Settings URLs. Instead, an admin defines named environment profiles once, and every run picks its target from the Env: dropdown next to the run controls on the Tests tab. Your choice is remembered in the browser (localStorage) so the next run defaults to the same env. Each env carries its own portal / backend / emulator / ACS URLs and credentials, and the gateway routes the run to a worker bound to that env.

See Multi-Stand Gateway → Selecting an env for how profiles map to workers, and the busy/409 behaviour when an env is in use.

Four Common Workflows

The agent supports four end-to-end flows. Most QA work falls into one of them.

Workflow A: Import → Run

Use this when there is already a TestRail case and you just need to execute it.

Tests tab — pick a case + Run

  1. Open Tests tab.

  2. Click TestRail (top right). The TestRail modal opens.

  3. Either:

    1. Pick Project → Suite → Section and check the cases to import, OR

    2. Switch to the Reimport by Case ID tab and paste a case number (e.g. 7677).

  4. The converter runs server-side: downloads the attached XML tree, extracts steps, enriches with LLM, generates emulator_setup. A toast confirms the import.

  5. Back on the Tests tab, find the case (use the filter), tick the checkbox.

  6. Pick the target environment in the Env: dropdown (see Picking the Environment Per Run).

  7. Pick the model in the dropdown (default: Opus — recommended; Sonnet may stall on multi-step interact flows).

  8. Click Run Selected (1). The view jumps to Runner.

  9. Watch the phase stepper (Loading → Pages → Executing → Parsing → Done) and the live MCP log on the right. Click Live Browser (top right) to open the gateway-proxied noVNC view of your run.

Workflow B: Edit Existing Test

Use this when an imported test fails because the converter missed something (wrong tree, missing parameter, wrong step ordering).

Editor tab — left list + right form

  1. Open Editor tab. The left list shows all imported cases — search by ID or title.

  2. Click the case. The form populates.

  3. Adjust:

    1. Preconditions / Emulator Serial

    2. Emulator Setup block — tree file, parameters, delete-objects, instance limits, UI preconditions, phases.

    3. Steps — change the type (nav vs interact), edit action / expected, reorder with arrows.

  4. Click Save.

  5. To re-run: switch to Tests, tick it, Run Selected.

If you only want to regenerate emulator_setup from the test text — and keep your manual step edits — click 🔄 Regenerate at the top of the editor (see editor-guide.adoc#regenerate).

Workflow C: Author From Scratch

Use this when there is no TestRail case yet, or you are reproducing a bug without official QA coverage. For conventions on writing the step text and preconditions so the converter produces a clean YAML, see Writing TestRail Cases.

  1. Editor tab → + New (top of the left list).

  2. Fill the Core Fields (Test ID, Portal, Type, Section, Title, Base URL, username/password, Preconditions, Emulator Serial).

  3. Click + Add Emulator Setup to enable the emulator block (see editor-guide.adoc#emulator-setup).

  4. Add steps — start with a nav: login step, then sidebar, widget etc., then interact steps for verification.

  5. Save → Run from Tests tab.

Workflow D: Capture a Known Quirk

Use this when a regression batch flaps on a particular widget value because the backend behaves differently from the spec — for instance "Monitoring widget Health oscillates ±10 between runs after a long predecessor". Write the quirk down once, and the agent stops false-FAILing on every future run.

Notes tab — three example notes

  1. Open the Notes tab.

  2. Click + Add Note.

    Notes editor modal — Monitoring flap example

  3. Fill in Title and Body (one-line summary + full explanation including acceptable deviation, e.g. "Health ±10 PASS").

  4. Set Scope — narrow to the affected test IDs (C7626), widgets (Monitoring), and/or KPI types (monitoring). Or tick Global if it applies everywhere.

  5. Pick Severity — known_bug for accepted backend deviations, warning for flaps, info for non-obvious conventions.

  6. Save. The note is on the volume in ~50 ms and indexed in Qdrant within ~3 s (via the auto-fired webhook). It will appear in:

    1. The LLM verify prompt for any future run whose step matches the scope, prepended to the "VERIFY each step" instructions.

    2. The converter Stage 2 prompt when a TestRail case matching the scope is (re-)imported.

    3. The test-runner Claude’s proactive search_qa_notes tool, when its interact loop is stuck or sees a suspicious widget value.

Day-to-day no manual reindex is needed — see qa-notes.adoc#reindex for the safety-net cases.

Organizing Cases

Each case carries organisational metadata: Project, Section, Reference (the Jira key), Priority, Type and Author. On the Tests tab the list groups by section and the project dropdown filters it, so large suites stay navigable. Set these fields in the Editor, or have them populated on TestRail import. See Organizing Cases for the full model.

Reading the Dashboard

The Dashboard tab summarises the in-memory run history:

  • Four metric cards — Total Runs, Pass Rate, Avg Duration, Last Run.

  • Pass/Fail trend chart across all recorded runs.

  • Run History list with status pill, elapsed time, and the model alias used. Filter by status (Pass / Fail / Error) or by portal (Support / Management).

  • Batch Reports dropdown — open archived batch reports (failure-category pie + results table + CSV export). See running-tests.adoc#batch-reports.

Click on a Run History row to drill into the run report. The Dashboard does not show per-test bar charts of duration — that view lives inside the Batch Report modal.

Reading a Run Report

In the Runner tab, after a run finishes:

  • The header pill shows PASS / FAIL / DRY RUN.

  • The left card lists each step with ✓ / ✗ / — (skipped).

  • The right card has two sub-tabs:

    • Output — full agent stdout (Claude reasoning, retries, verifier output).

    • MCP Log — every Playwright MCP call (browser_click, browser_run_code, browser_snapshot, etc.) with arguments. Use this to understand exactly what the browser did at each step.

For a failed step, the agent writes a short failure reason next to the ✗. Scroll the Output pane to find the full trace and the verifier’s expected vs actual.

When to Touch Settings

Most users never touch Settings — defaults are correct for test-agent. Edit when:

  • You need to switch the test target environment (set new portal / backend / emulator URLs).

  • TestRail credentials rotated.

  • Adding ClickHouse seeding for interference tests.

See Settings.

Common Pitfalls

Symptom Likely cause and fix

Test was imported but emulator_setup is empty

Tree XML was not attached to the TestRail case. Open Editor, click Generate Tree, or set tree_file manually to one of the templates (e.g. 181.xml).

Test stalls in Executing for several minutes on Sonnet

Interact loop is waiting for an element that never appears. Switch to Opus in the Tests tab dropdown and re-run.

Login step works, but sidebar click times out

The portal credentials in Settings / Editor are wrong, or the base URL points at a portal that does not match portal: in the YAML.

Dashboard shows old failures even after fixing the converter

Ensure you reimported the case after the converter change (see importing-tests.adoc#reimport). The persistent volume keeps the old YAML.

Verify still FAILs even after writing a matching QA note

Check the note’s Scope — if it lists test_ids or widgets, the test’s actual id and the segment’s nav_target (case-sensitive) must match. Press Reindex on the Notes tab and read the summary; if errors > 0 the Qdrant upsert failed. See qa-notes.adoc#troubleshooting.

I don’t see my run, or my history looks empty

Runs are owner-scoped — you see only the runs you started (admins see all). Logging in as a different account, or as an editor/viewer, shows a different slice of history. See Accounts & Roles.

"Env is busy" / HTTP 409 when starting a run

Another run already holds that environment (more precisely, its emulator backend). Wait for it to finish or pick a different env in the Env: dropdown. See gateway.adoc#selecting-env.