Editor Guide

The Editor is the visual front-end for the YAML test case format consumed by run_test.py. Every field maps 1:1 to a YAML key — the editor never hides data, it just makes it editable without writing YAML by hand.

It sits in the sidebar under Author. You also land in it from a Test cases row — the pencil and Copy buttons appear on hover — and from a run report (Open in editor).

Editor — left list, form, and the TestRail source toggle
At a glance

  • List on the left = every YAML on the agent, in a four-row header: title and count, search, three filters, selection and bulk actions.

  • Right pane has three modes — list (no selection), new (empty form), edit (a row clicked).

  • Save persists to disk; ▶ Save & Run also triggers a run.

  • More ▾ holds Export YAML, Duplicate, Regenerate, Reimport and Delete.

  • Validate vs TestRail scores the case against the live TestRail text and flags drift since import.

  • TestRail source shows the original case text beside the converted steps.

  • Below the form, an open case carries its Change history, its Execution history and its Attachments — see The Case’s Own Panels.

  • Leaving with unsaved changes is guarded by a dialog — nothing is auto-saved.

Overview

Open the Editor. The layout is split:

  • Left — list of all YAML cases currently on the agent (under /app/tests/cases inside the container, mounted as the persistent test-cases volume). Search box filters by ID or title.

  • Right — the form. Three modes:

    • list — empty placeholder (no case selected, "+ New" not pressed).

    • new — empty form for authoring a fresh case.

    • edit — populated form for an existing case.

The header of the left list is four rows, one job each.

  1. Title, count and actions. The count is how many cases the filters below currently leave on screen. Beside it:

    • + New — clear the form and switch to new mode.

    • ⋯ — an overflow menu (labelled More actions) holding TestRail… (the import modal, Importing Tests), Import (upload a hand-edited .yaml, POST /api/tests/upload-yaml), Sync runs… (ask TestRail which cases each run covers and stamp it on the local YAMLs), and — after a separator — the destructive Delete all tests and Delete by section (see Bulk Delete (the ⋯ menu)).

    Both are hidden from a viewer account: the whole row is write-only.

  2. Search, full width, matching ID or title.

  3. Three filters of equal width — All portals, All projects and Any run — plus a Reset link that appears only while at least one of them is set. All three drop the cases they hide from the sidebar’s selection — this list drives bulk edits and deletes, where acting on a row that is off screen is a different kind of accident from running it. Project and run are shared with Test cases, so a project chosen there is still in force here, but since the 2026-09-09 build neither one touches that tab’s selection: a case picked for a run stays picked.

  4. Selection. A Select all checkbox (it ticks exactly the cases the filters leave on screen) and an N selected count. The sidebar’s own bulk bar appears only once something is ticked: Validate, Reimport, Add to project… and Delete.

Add to project… used to sit among the filters, where it read as a way of narrowing the list. It is a bulk action and now lives with the others, so it appears only when there is a selection to act on.

A row that has been validated carries its score as a coloured chip; clicking the chip opens the validation report. A case nobody has validated yet has no chip at all, which is how the two are told apart. A row also shows the portal, the type and priority, a blocked pill where the case needs infrastructure the stand lacks, and its Jira references as links.

Periodic TestRail refresh is handled by the background sync loop (see Auto-sync Interval).

Right-panel header buttons (visible only in edit mode):

  • Validate vs TestRail — re-reads the case from TestRail live and scores the YAML against it, so drift since the import is visible (the report says whether it compared against the live case or the import snapshot).

  • More ▾ — Export YAML, Duplicate (copy this case under a free id; TestRail run membership is deliberately dropped and the attachments are copied), Regenerate (see Regenerate Emulator Setup), Reimport, and Delete (a move to the trash, not an unlink — see test-management.adoc#trash).

  • Save — persist changes to disk.

At the bottom of the form there is also ▶ Save & Run — saves the case and immediately starts a run with the model and env picked in Test cases.

There is no auto-save. After every change press Save (or Save & Run) at the bottom.
Unsaved changes are guarded, not kept

Switching section, opening another case, or closing the browser tab with unsaved edits raises "Leave the editor with unsaved changes?". Confirming discards them — there is no draft buffer. Save first.

Bulk Delete (the ⋯ menu)

The bottom half of the ⋯ overflow menu, below the separator, holds two sweeping actions:

  • Delete all tests — calls DELETE /api/tests and moves every case out of /app/tests/cases. Use after a major converter rewrite when you intend to fully reimport from TestRail.

  • Delete by section — under the Delete by section label the menu lists every distinct section value in the current YAMLs, with a count next to each. Click a section to remove only those tests (DELETE /api/tests?section=). Useful for cleaning up an obsolete TestRail section without losing the rest.

Both are confirmed before they run, and both are undoable: a delete is a move into the trash, not an unlink. The cases come back from Test cases → Library → Trash — see test-management.adoc#trash. What cannot be undone is emptying that trash, which is admin-only.

Core Fields

These are the top-level keys that drive the runner before steps are even considered.

Field YAML key Notes

Test ID

id

Format C<TestRail-id>, e.g. C7677. Used as a stable identifier and as the report row title.

Portal

portal

support or management. Dictates which portal the runner logs into and which sidebar / widgets are available.

Section

section

Free-text grouping (e.g. TR-181, TR-181 Mesh). Pure metadata, used for list filtering.

Title

title

Human-readable test title, shown in the list and reports.

Project

project

Dropdown of the projects defined under Organizing Cases: Projects & Sections. Groups the case for list filtering and reporting alongside section.

Priority

priority

Low / Medium / High / Critical (free values also allowed). Rendered as a colored pill in the list and edit header (green / blue / amber / red).

Type

type

Test category — Acceptance, Functional, Performance, Regression, Security, Smoke & Sanity, Usability, Other (custom values preserved). Pure metadata.

Reference

reference

Jira ticket reference(s), e.g. DEV-1234. Rendered as a clickable link when Settings supplies a jira_base_url (Settings); otherwise shown as plain text.

Base URL

base_url

Defaults to the value from Settings if left empty. Override only when one specific case must hit a non-default portal.

Username / Password

credentials.username / .password

Same fallback rule — empty here means use the Settings defaults.

Preconditions

preconditions

Plain-text bullet list (one per line). Read by the LLM during conversion to deduce required emulator_setup.parameters, not parsed at runtime.

Emulator Serial

emulator_setup.device_serial

The CWMP serial number of the emulated CPE. Defaults to test-runner-001. Must match the device the agent navigates to in the portal.
In edit mode an attribution strip above the form shows Created by (created_by) and Last edited by (edited_by) with timestamps, plus a per-case edit history. These are written by the server from the logged-in account — they are read-only and not part of the editable form. See Organizing Cases: Projects & Sections for the project / section taxonomy these fields feed into.

Emulator Setup

Most tests need state on the emulator — TR-181 parameters, dynamic instances, diagnostic results — for the dashboard / portal to render the expected value. This is configured under Emulator Setup.

If the block is missing, click + Add Emulator Setup and an empty stub is created (tree_file, parameters, delete_objects, instance_limits, wifi_client_limit, ui_preconditions, phases). All sub-fields are optional.

The block also carries a protocol selector. TR-181 / TR-098 (the default) run the case on the TR-069 emulator through mcp-tr-emul; USP routes it to the IoT emulator through mcp-iot-emul instead (USP/TR-369 over STOMP, plus LwM2M / MQTT / CoAP devices). The ACS target for a USP agent comes from the env profile’s usp_* keys — see settings.adoc#emulator.

Execution order (server-side, before the first step runs):

  1. Push tree_file to the emulator (factory reset → upload tree).

  2. Apply delete_objects (CWMP DeleteObject).

  3. Apply instance_limits (truncates dynamic tabular objects to N instances).

  4. Apply wifi_client_limit (special-case for Hosts.Host).

  5. Set parameters via SetParameterValues.

  6. Run ui_preconditions against the management backend (see UI Preconditions).

  7. Trigger ACS Inform so backend reads the new state.

  8. If phases[0] exists, apply phase 1 parameters and continue.

Tree File and "Generate Tree"

tree_file is the name of an XML file under /opt/mcp-shared/ on the test-runner host. It defines the initial TR-181 tree pushed into the emulator (factory reset baseline). Common templates:

File Purpose

181.xml

Default Fiber CPE: CPU, MEM, 4 radios, full diagnostics. Fallback for unrecognised KPI types.

181-cellular.xml

Cellular CPE.

181-mesh-de.xml / 181-mesh-multiap.xml

Mesh CPEs with DataElements / MultiAP data models.
Selecting a different tree_file does not re-push the tree. It only changes which file is used the next time the test runs. To verify what’s currently on the emulator, query its API directly (see Troubleshooting).

The Tree File row exposes four controls:

  • Tree dropdown — populated from GET /api/trees, which lists every *.xml in /opt/mcp-shared/. The currently selected file is always pre-rendered as the first option so the dropdown reflects the saved value immediately, even when the list is still loading or the file is no longer on disk.

  • Upload — opens a file picker. Selecting an .xml calls POST /api/trees/upload (multipart). The server validates the payload as well-formed XML, sanitises the basename (no path traversal), and writes it into /opt/mcp-shared/. On success the dropdown refreshes and the new name is auto-selected. Re-uploading the same name replaces the existing file — the response carries replaced: true.

  • Delete — removes the selected tree from /opt/mcp-shared/ (DELETE /api/trees/{filename}). Only do this for a tree you uploaded; the templates are shared by every case on the stand.

  • Generate Tree — described below.

Generate Tree

The Generate Tree button calls the bbf-tree-mcp server with a natural- language description of the test (title + preconditions + first step). The MCP returns a fresh XML tree tailored to the case. Workflow:

  1. Edit Title and Preconditions so they describe the device state you want.

  2. Click Generate Tree.

  3. A modal appears with a live status pane (connecting → reading preconditions → calling MCP). Generation typically takes 10–30 s but can run minutes for large mesh topologies.

  4. On success the modal shows the new filename and the tree_file field in the form is updated. Save the test to persist.

  5. On failure the modal shows the MCP error verbatim — fix the preconditions text or fall back to a template under /opt/mcp-shared/ (use the dropdown).

Use this when none of the existing templates fit (unusual KPI mix, exotic data model, multi-radio mesh topology).

Parameters

Key/value list of TR-181 paths that override the tree after factory reset. Examples:

parameters:
  Device.WiFi.AccessPoint.1.AssociatedDevice.1.SignalStrength: "-55"
  Device.WiFi.AccessPoint.1.AssociatedDevice.1.Noise: "-95"
  Device.WiFi.AccessPoint.1.AssociatedDevice.1.OperatingStandard: "ax"

Editing in the UI:

  • + Add opens an inline row — paste the full Device.<…​> path and the value. Both fields are free-text monospace.

  • The big monospace block lists existing entries — click in the value field to edit in place; the x button removes.

  • Values are sent as strings to the emulator; numeric/boolean coercion happens on the emulator side per TR-181 type.

SNR is not a TR-181 parameter — it’s computed as SNR = SignalStrength - Noise. To get a target SNR set both SignalStrength and Noise accordingly.

Delete Objects and Instance Limits

Delete Objects

List of TR-181 instance paths to delete after the tree is pushed. Useful when the template tree contains a parameter the test wants absent (e.g. delete OperatingStandard to test the missing-param weight redistribution path).

Instance Limits

Map of parent path → N. Backend truncates the dynamic tabular collection at parent to keep only N instances. Example: Device.Ethernet.Interface.: 2 keeps only the first two Ethernet interfaces.

WiFi Client Limit

Special integer — caps Device.Hosts.Host.{i} to the given count. Equivalent to setting Device.Hosts.Host.: N in Instance Limits, but exposed separately because it is the most common case.

UI Preconditions

Pre-test actions executed against the management backend’s REST API (not against the portal browser). Each entry has an action and params. The server dispatches them through the _PREFLIGHT_ACTIONS table; the full set is below, grouped by purpose.

Table 1. Monitoring
Action Purpose

CREATE_MONITORING

Creates a monitoring rule on the device so the network-history widget has data points. Mostly auto-applied; rarely edited by hand. (Also the anti-flap trigger — its presence disables the autonomous Periodic Inform for the run.)

DELETE_MONITORING

Removes a monitoring rule (the inverse of CREATE_MONITORING) — for tests that assert monitoring is off.

TRIGGER_MONITORING_TASKS

Forces the backend’s monitoring tasks to run now instead of waiting for the next scheduled pull, so freshly set state is reflected immediately.

CREATE_GROUP_UPDATE

Creates a group update with the selected diagnostics list and optional reactivation flag (periodic re-run). Used for tests that exercise the group update widget.
Table 2. Diagnostics
Action Purpose

RUN_DIAGNOSTICS

Generic diagnostic runner. Supports IPPingDiagnostic, DownloadDiagnostic, UploadDiagnostic, NSLookupDiagnostic, TraceRouteDiagnostic, NeighboringWiFiDiagnostic. For IPPing the host field controls latency band (8.8.8.8 for "good", 203.50.2.71 for "bad").

RUN_INTERFERENCE_DIAGNOSTICS

Triggers NeighboringWiFiDiagnostic and (if INTERFERENCE_DASHBOARD_SEED is on) seeds the ClickHouse interference dashboard.

RUN_NEIGHBORING_WIFI_DIAGNOSTICS

Alias of RUN_INTERFERENCE_DIAGNOSTICS — runs the neighboring-Wi-Fi scan (same handler), named after the diagnostic for readability.

RUN_UPLOAD_DIAGNOSTIC

Runs an UploadDiagnostic so the upload-speed result is populated before the test reads it.

VERIFY_DEVICE_ONLINE

Polls the backend until the device reports online — a guard step so later actions don’t run against a still-booting CPE.
Table 3. ACS / parameters
Action Purpose

ACS_TASK

Drives the ACS via the management REST API (SPV / GPV / add / delete object / reboot / factory reset) as a precondition. Same surface as the acs_task step type (see Steps), run before the first step.

SET_ACS_PARAM

Sets an ACS configuration parameter (e.g. overwriteProvisionParameters) that must be in place before the run.

SET_CPE_LOGIN_GENERATE

Enables ACS Basic / Digest credential generation for the device (sets the CPE login + generateCredentials flags). Required precondition for the credential-length tests driven by cred_length_verify; restored on post-flight.

SET_TARIFF

Assigns a tariff plan to the device (static QoE tariff seed).

SET_TARIFF_DYNAMIC

Assigns a dynamic tariff (used by diagnostic / QoE tests that need a rate-limited plan).

SET_UI_SETTING

Writes a portal UI setting (e.g. an Advanced-View toggle) the test depends on.
Table 4. Files / devices
Action Purpose

ADD_DEPLOYMENT_FILE

Registers a deployment (firmware / config) file in the platform so a file-download / deployment test can select it.

ADD_DOWNLOAD_FILE

Registers a download file entry the device can be told to fetch.

CREATE_NBI_DEVICE

Creates a device via the North-Bound Interface (NBI) before the run — for tests that need a freshly provisioned device record.
Table 5. Domains and domain users (multi-domain / multi-ISP cases)
Action Purpose

CREATE_DOMAIN

Creates the domain the case acts in — nested under a parent when the case needs a hierarchy. Purged on post-flight.

CREATE_DOMAIN_USER

Creates a user inside that domain and injects the matching nav_action: login_as so the case actually runs as the domain user.

MOVE_DEVICE_TO_DOMAIN

Moves the test device into the run’s domain, the way the product does it for the stand’s ISP mode.

CREATE_DOMAIN_SETTINGS

Seeds a domain-scoped settings record, for cases that assert a setting applies to one domain only.

CREATE_EVENT_URL

Seeds an event URL on a domain so the inheritance cases have something to read back.

CREATE_EXT_SYSTEM_OVERRIDE

Seeds an External-System override in the domain.
Table 6. Seeded records
Action Purpose

CREATE_VIEW

Creates a saved view in the views bar.

CREATE_AUTOMATION

Creates an automation / event-monitoring record.

CREATE_GROUP_CONDITION

Creates a group condition the case then edits or filters on.

CREATE_USER_GROUP

Creates a user group with a permission tree — read back by user_group_permission_verify.

CREATE_DEVICE_PROFILE

Creates a device profile and waits for its provisioning task.

CREATE_ACS_LOGIN

Seeds an ACS login record.

CREATE_WHITELIST

Seeds a whitelist entry.

CREATE_USP_SECURITY

Seeds the USP Security row the ACS needs before it will admit a USP agent (an Endpoint-ID mask and an MTP record).

CREATE_MQTT_SECURITY

Seeds the MQTT security record for MQTT devices.

ADD_UPLOAD_FILE

Registers an upload file entry.
Table 7. Guards and utilities
Action Purpose

START_TRACE

Starts a device trace before the run — for the trace-viewer cases.

ENSURE_PARAM_PRESENT / ENSURE_PARAM_ABSENT

Assert a TR-181 parameter is (or is not) on the device before the case looks at it, so a stale stand cannot decide the verdict.

noop

Explicitly does nothing. Used to keep a converted precondition visible in the YAML when it needs no action on this stand.
The dispatch table is strict — an unknown action name raises instead of being skipped, so a typo fails loudly at pre-flight rather than silently producing a green run with no preconditions.

device_serial and portal are auto-filled from the test’s top-level fields, so you only need to choose the action and (if relevant) the diagnostic type, host, or action-specific params.

Multi-Phase Tests

Some TestRail cases are written as a sequence of "set state A → verify; set state B → verify". The converter encodes them as a single YAML with multiple phases. Each phase has a name and its own parameters map.

Editing:

  1. Click + Add Phase to append. The button appears both in the populated Phases header and in the empty-state row when no phases exist yet — fresh tests can grow phases from zero without touching YAML.

  2. Each phase shows an editable name and a parameter map (same UI as the main Parameters block).

  3. Order matters — phases run in array order. Use the x to remove.

  4. The corresponding phase step in the steps list (see Steps) is what triggers the runtime to apply that phase.

Phases share the base parameters — phase entries are additive overrides on top of the initial tree + base parameters, not full replacements.
A phase with an empty parameters map is a scenario marker — the runtime treats the matching step as instant pass and only re-runs any diagnostic preconditions tied to the phase. Such empty phases are preserved through a load → save round trip so converter-emitted markers (common in multi-scenario tests) survive in-place edits.
Converter-emitted phases sometimes carry extra fields like preflight (e.g. SET_TARIFF_DYNAMIC for diagnostic tests) and diag_params. The editor preserves those fields verbatim — they are out-of-band of the visible inputs but round-trip cleanly on save.

Steps

Steps is a flat list. Each step has:

Field Meaning

Type

One of 23 step types, 22 of them in the dropdown. Empty type defaults to interact. See the Step Type reference below for what each one does and its fields.

Action

Free-text. For nav steps it’s a hint for the report; for interact steps it’s the prompt the LLM sees ("Open Wi-Fi tab and verify the SSID is 'TestSSID'"). Used as a label for the deterministic verify types and ignored for phase / emulator_action.

Expected

What the verifier should look for. Programmatic verifier scans the DOM; LLM verifier reads the rendered page text.

nav_action

Only for nav type. About 100 actions in six groups (Navigation, Dropdowns / dialog, File Download, File Upload, Diagnostics / Advanced View / misc, Domains / filters). See the nav_action reference below.

nav_target

Label text for the action (sidebar item, tab name, button text, widget caption, dropdown trigger label, option text). E.g. nav_action: widget, nav_target: Network Score. (Some actions take an action-specific field instead — e.g. export_verify reads report_format = excel / csv.)

wait_for_content_change

Optional checkbox for click_button. The runner takes a content snapshot before the click and waits until it differs.

phase_index

Only for phase type. 1-based index into emulator_setup.phases[] selecting which delta to apply at this step. The editor renders this as a dropdown of existing phases by their name, e.g. 2: Scenario 2: The "LAN" KPI is displayed as "Good". The currently-bound phase is rendered as the first option so the dropdown shows the saved value even before the phase list finishes loading. If no phases are defined an inline hint points back to Emulator Setup → Phases.

emulator_action

Only for emulator_action type. The MCP tool name — typically set_parameters_batch or invoke_event.

params

Only for emulator_action type. Map of parameters / arguments to forward to the MCP tool.

Use the up / down arrows in each step row to reorder; ✕ removes; + Add Step appends.

Step Type Reference

The 23 step types and what each does. For step phrasing (how to word action / expected so the converter and verifier agree), see Writing TestRail Cases.

They fall into three families: the two general-purpose ones (nav, interact), the typed executors that drive something other than the browser (phase, acs_task, emulator_action, db_query, log_check), and the _verify family — deterministic checks that *measure a claim over REST or the DOM instead of asking a model about it. Every one of the latter exists because the same sentence, judged from a snapshot, produced a green that proved nothing; they are what the Measured judge tag reports (see running-tests.adoc#verdicts).

Type Purpose & key fields

nav

Deterministic browser script (fast, never stalls). Drives the portal via a nav_action + nav_target. See the nav_action reference.

interact

Handed to Claude with a snapshot-action loop. action is the prompt, expected is what the verifier looks for. Use when the step needs reasoning.

phase

Apply a multi-phase emulator delta and refresh state. phase_index selects the 1-based entry in emulator_setup.phases[] (see Multi-Phase Tests).

acs_task

Drive the ACS through the management REST API. acs_action = spv / gpv_direct / add_object / delete_object / reboot / factory_reset, with params, push, reprovision, verify_tasks_created, and wait_seconds.

emulator_action

Push parameters / invoke Inform directly through the emulator MCP, no UI. emulator_action is the MCP tool name (set_parameters_batch / invoke_event); params are its arguments.

spa_task_verify

Verify a Device-Activity SPA monitoring task was emitted (e.g. a "Set parameter attributes / Notification" task) after a portal action.

db_query

Assert against the Platform DB (MySQL / Oracle). Requires the Platform DB connection to be configured in Settings.

log_check

Post-test SSH grep of the server logs. Warn-only — it attaches its findings to the report but never fails the test. Requires Log Server settings in Settings.

cred_length_verify

Validate generated-credential min/max length. Fields: set_min / set_max (optionally push credentialMinLength / credentialMaxLength first), min_len / max_len (expected random-length window), oui, and poll_seconds. A second mode takes mode + login / password and checks one credential’s individual behaviour. Used together with the SET_CPE_LOGIN_GENERATE precondition.

chat_ask

Ask the Device AI Assistant a question and judge the answer — by containment and semantically — with an optional confirm / deny follow-up.

ai_step

A natural-language step. What decides its verdict is not the model but its evidence_spec[]: a list of checks of kind ui_text, ui_state, ui_absent, param_gpv, rest_api, file_download or ui_llm_judge. Once a step has run stably it can be compiled down into deterministic steps — see Promoting an ai_step.

explore

A read-only survey of a screen. Produces a report, never a pass or a fail; the report can then be turned into draft cases. Normally synthesised by Explore rather than written by hand, but selectable like any other type — an EXPLORE-* case opens in the editor like the rest.
Table 8. The *_verify family — each measures one claim programmatically
Type What it reads instead of guessing

duplicate_user_verify

Attempts the creation the case says is forbidden and reports the backend’s own answer, instead of describing an empty form.

unused_models_scope_verify

Compares the domain user’s unused-model list against the super domain’s over REST — not the row count, which drifts stand-wide inside one batch.

ext_system_scope_verify

Reads the External System tab as the domain user and as the super domain, so "configuration related to the Super domain is present" is proven rather than asserted as page text.

ext_system_readonly_verify

Compares the writable flag the portal greys the control on, so "greyed out for domain users and active for super domain users" cannot collapse into "both labels are on the screen". Takes compare_defaults.

domain_setting_scope_verify

Re-reads a Settings section as both callers after the Save, so "changes applied separately to a specific domain" is measured on two screens, not one.

user_group_permission_verify

Reads the seeded group’s permission tree over REST (e.g. View + Execute on Domains).

device_domain_move_verify

Re-registers the device with the other domain the way the product does for the ISP mode, waits, and reads it back as both domain users.

event_urls_verify

Reads, over REST as the logged-in domain user, the two lists the Edit Event URLs dialog renders — so "the URL from the parent domain is present / absent" is answered even when the dialog shows an empty list.

group_update_import_verify

Reads the saved update’s rows, its "Send to" and its associated serials after an XML serial-list import.

file_on_server_verify

Fetches the uploaded file from the run domain’s folder on the stand’s file server (FTP / HTTP), rather than judging the form left behind on screen.

device_file_transfer_verify

Drives and reads back a device Upload / Download over REST as the domain user.
Every type has its own inputs in the editor form — each renders a tinted block under the common action/expected pair carrying just its own fields (interact its target hint, emulator_action the MCP tool and its arguments, spa_task_verify the task name and value, explore its starting point). Two unit tests keep it that way: a type the runner dispatches must be selectable in the editor and must have a render block.

When Type is nav, the nav_action dropdown groups about a hundred actions into six sections. The ones you will write by hand are in the first two groups; the rest are overwhelmingly converter-emitted, and most of the _verify ones are *self-verifying — they decide their own step, which is why they show up with the Measured judge.

Table 9. Navigation
nav_action Purpose

login

Log into the portal (always the first step).

login_as

Log in as a domain user — the entry action of the multi-domain wave. Takes login / password / domain.

sidebar

Click a left-sidebar item (nav_target = item label).

tab / sub_tab

Switch a tab (or a nested tab) inside a widget or page.

click_device

Click the row of the test device (auto-targets the emulator serial).

click_button

Click any button by label; optional wait_for_content_change.

modify_and_save

Change a field and save it in one deterministic gesture.

settings_form_edit

Edit the Settings form deterministically, leaving Save alone.

hover_button

Hover a button (reveals tooltips / hover-only UI).

widget

Open a dashboard widget (nav_target = widget caption).

scroll_to_section

Scroll a named section into view.

press_escape

Send Escape (close an overlay / dropdown).

check_row

Tick the checkbox on a table row.

list_not_empty

Assert the list on screen actually has rows.

domain_context_verify

Assert which domain context the screen is in.
Table 10. Dropdowns / dialog
nav_action Purpose

expand_dropdown

Open a Material mat-select dropdown.

select_option

Pick an option in an open mat-select dropdown.

hover_in_dialog

Hover an element inside an open dialog.

open_condition_dialog / create_condition / save_condition

Drive the New condition dialog end to end.
Table 11. File Download (deterministic, self-verifying)
nav_action Purpose

download_dropdown_verify

Open the File-Download Add-form dropdown and assert a value is present / absent.

download_form_verify

Read the File-name dropdown placeholder and assert the default text.

download_fill_verify

Type into Description / Username / Password and confirm each is populated.

download_create_verify

Create a download task and verify a row renders in the list.
Table 12. File Upload (deterministic, self-verifying)
nav_action Purpose

upload_file

Upload a file (URL mode default or explicit URL).

upload_open_form

Open the Upload Add form.

upload_delete

Delete an uploaded file (optionally from FTP).

upload_get_file

Fetch / read back an uploaded file entry.
Table 13. Diagnostics / Advanced View / misc (deterministic)
nav_action Purpose

diagnostic_dropdown_verify

Collect an open diagnostic dropdown’s options and self-verify (min count / max numeric).

diagnostic_field_verify

Assert named fields are present / absent in the diagnostic Add form.

readonly_param_verify

Assert a parameter is rendered read-only.

advanced_view_set_param

Fill named param value(s) in the Advanced-View tree edit mode.

advanced_tree_select

Click a child tree node and assert the selection cascade.

advanced_view_root_verify

Assert which root the Advanced View opened on.

device_activity_method_verify

Open the Device-Activity "Task name" filter and assert the target method is listed.

custom_rpc_method_verify

Assert the Custom RPC method list offers the expected method.

device_settings_edit_verify

Edit a device setting and read the saved value back.

port_mapping_interface_verify

Assert the port-mapping interface list.

export_verify

Click export and confirm a 2xx report request fired. Takes report_format (excel / csv).

focus_field / focus_verify / tab_focus_verify

Place focus, then assert which element has it — including after Tab.

delete_refused_verify

Assert a delete was refused, and on what grounds.

word_wrap_verify

Assert long text wraps instead of overflowing.

search_no_match_verify / search_scroll_verify

Assert the empty-result state, and that the result list scrolls.

settings_save_verify

Re-read the setting after Save and compare it with what was typed.

domain_filter_verify

Read the grid itself and assert which rows the domain filter left.

seeded_row_present / seeded_row_absent

Assert the seeded record is (or is not) in the list, by the seed’s own name.

field_value_verify

Assert a named field holds a given value.

ui_state_verify

Read one named check-box, toggle, radio button or field. Fail unless it is in the state expect_state names. Set control_kind to checkbox, radio or field.

grid_filter_verify

Wait for a live filter, then read its grid. Fail unless the filter input holds a value and every row matches it. Set nav_target to logs or devices. Set column to the grid column to compare, Serial by default.

form_bottom_gap_verify

Scroll the parameter area of a task form to its end. Fail unless the last row ends above the Save and Cancel bar. Set mode to tab or advanced.

parameter_order_verify

Open an object sub-tab of Automation parameters. Fail unless its Parameter name cells are in ascending order.

device_activity_password_verify

Read the Password values of the "Set parameter values" rows on Device activity. Set expect_state to masked to fail on any value that is not only asterisks. Set it to plain to fail on a value that is only asterisks. The step reads only the rows that appear after the run starts. The step is inconclusive when no such value exists.

apply_list_filter and list_filter_verify

Apply a list filter, then read it after the return to the list. Set nav_target to seeded on the Device Profile list. Set it to seeded:group_update, seeded:monitoring or seeded:file_management on those lists. On the Device Profile list, the apply step picks a Profile status other than All that shows the profile of the run. The verify step fails when the status is back to Active or the profile is missing. On the other three lists, the apply step picks a Manufacturer other than All. The pick must show the record of the run and remove a row of the unfiltered list. The verify step fails when Manufacturer is back to All, the record is missing or the removed row is back.

file_url_verify

Assert the file URL the form produced.

default_profile_tabs_verify

Assert the tab set of the default profile.

landing_page_verify

Assert which page the login landed on.

url_validation_verify

Type a URL and assert the validation verdict.

import_result_verify

Assert the outcome the import reported.

software_install_verify

Assert the software-install task was created and completed.

password_policy_verify

Assert the password policy the portal enforces.

network_map_show

Open the network-map modal and collect its topology text for the verifier.
Table 14. Domains / filters (deterministic)
nav_action Purpose

select_domain / select_domain_link

Pick a domain in the picker, or follow the domain link in the header.

picker_confirm

Confirm the picker dialog.

add_subdomain / delete_domain

Create a nested domain, or delete one.

domain_name_validation / domain_dialog_close

Assert the name validation, close the dialog.

domain_picker_scope_verify

Assert which domains the picker offers this account.

list_row_domain_verify

Assert the domain each listed row belongs to.

set_filter / new_filter / view_filter_section / select_filter_row / logic_radio

Build a filter row by row, including its logic operator.

search_criteria

Set the search criteria deterministically.

view_bar / action_bar / open_list_row

Drive the views bar, the action bar, and open a row.

import_file / import_serials / source_type

Import a file or a serial list without the OS file dialog, and pick the source type.

select_radio / select_toggle / select_row_checkbox

Set a radio, a toggle, or a row checkbox.

button_disabled_verify

Assert a control is disabled — and that it is disabled, not absent.

device_model_select / profile_model_select

Pick a device or profile model.

file_management_add / file_management_open

Drive the File Management screen.

send_to / pick_devices / record_name / group_update_form_complete

Fill a group update: target, devices, name, and the rest of the form.

account_info_edit

Drive the Account info dialog.

usp_security_create

Create the USP Security row from the UI.

form_tweak

Change one field of a form without touching the rest.

noop

Do nothing — keeps a converted step visible without acting.

Step Type Cheat Sheet

Goal Use type Example

Log into the portal

nav + nav_action: login

always the first step

Log in as a domain user

nav + nav_action: login_as

login: dom-user, domain: RunDomain

Click an item in the left sidebar

nav + nav_action: sidebar

nav_target: Devices

Switch tab inside a widget / page

nav + nav_action: tab

nav_target: Wi-Fi

Click the row with the test device

nav + nav_action: click_device

auto-targets emulator_serial

Click any other button

nav + nav_action: click_button

nav_target: Save

Open a dashboard widget

nav + nav_action: widget

nav_target: Network Score

Open a Material Design select dropdown (mat-select rendered in cdk-overlay-container)

nav + nav_action: expand_dropdown

nav_target: Configuration

Pick an option in an open mat-select dropdown

nav + nav_action: select_option

nav_target: 5GHz

Apply the next phase from emulator_setup.phases[] (push params + refresh)

phase + phase_index

phase_index = 2

Push TR-181 params directly without UI (no portal click — emulator-only state change)

emulator_action + emulator_action: set_parameters_batch

params: { "Device.WiFi.Radio.1.Channel": "11" }

Verify a value or perform any UI action that needs reasoning

interact

action = "Verify the Health value is 'Good'"
Prefer nav whenever the action has a fixed label and no decision-making — it is roughly an order of magnitude faster than interact and never stalls.
expand_dropdown issues an Escape only when the next step is not select_option, so the open dropdown stays accessible to the option click. The selector logic walks cdk-overlay-container because Angular Material renders the option list at the end of the DOM, not inside the trigger element.

The Case’s Own Panels

Below the step list, an open case carries three panels. They are read-only context for the form above them: none of them is part of the YAML, and none is saved with Save.

Editor — the Change history panel with its versions, diff and restore

Change History

Every write to this case — a Save, a YAML upload, a TestRail import or reimport, a duplicate, a restore, a bulk edit — is one numbered version, whichever screen made it. A delete is logged too, without a number. The panel lists them newest first, and each row carries:

  • the version number (v7) — a deleted row and rows written before versions existed carry none;

  • who saved it and when;

  • the action — created, edited, uploaded, imported, reimported, restored or deleted;

  • what changed — the fields that moved, as field: before → after;

  • the bug reference the edit was made for, rendered as a Jira link when Settings supplies a jira_base_url;

  • the portal build the edit was made against (build master_7.1.2.RC), probed from the portal itself, so a case edited the morning after a deploy is distinguishable from one edited before it.

A row that kept a snapshot offers:

  • Diff — a modal with the unified diff of that version against the case as it is now (GET /api/tests/{filename}/history/{version}). The modal offers Restore this version as well, so you can read before deciding.

  • Restore — overwrite the case with that version (POST /api/tests/{filename}/history/{version}/restore). Write access only, confirmed first, and itself recorded as a new version — so a restore is undone by restoring the version that preceded it.

A snapshot holds the case as that save left it, not as it was before. v7 is what the seventh write produced, which is why restoring v7 reproduces that state exactly.

The panel is hidden for a case with no history at all. Refresh re-reads it.

Execution History

The last twenty runs of this case, newest first: verdict pill, when, how long, which env and the owner. Clicking a row opens that run’s report, so "did my last edit help" is answered without a trip to Runs.

The rows are owner-scoped like the rest of the run history: you see only runs you started. A case nobody has run under your account on this stand says so: "This case has never been run here."

Attachments

The files the case was written against — the device tree, a screenshot, the TestRail export. The panel lists each file with its size and who uploaded it. Everyone can read and download them; Upload, Delete and Use as tree need write access.

  • Upload — one file per press, 20 MB cap, the name reduced to a safe basename. Uploading the same name replaces it.

  • Clicking the file name downloads it.

  • Delete — removes it for good (confirmed).

  • Use as tree — offered on .xml attachments only. It pushes the file into /opt/mcp-shared/ through POST /api/trees/upload and points the case’s tree_file at it, which is the step that used to be done by hand.

    Use as tree needs the case to already have an Emulator Setup block — without one there is no tree_file field to write, and the editor says so instead of acting. And it only puts the name on the form: press Save to keep it.

Attachments survive a delete and a restore — they are removed only when the case is purged from the trash. A Duplicate copies them onto the new case.

Regenerate Emulator Setup

Regenerate, in the form’s More ▾ menu (edit mode only), re-runs the converter pipeline using the current text fields (preconditions, steps, expected, title) and replaces the emulator_setup block while leaving steps and metadata untouched.

A modal appears with live status (saving edits → re-running the converter → score-simulator retry → reload). On success the modal shows the new tree_file, parameter count, and any delete_objects / wifi_client_limit that were derived. On failure the converter error is shown verbatim.

When to use:

  • You changed Preconditions or step Expected values and want the emulator state regenerated.

  • You’re investigating why a test was imported with the wrong tree / parameters.

  • The backend formula changed and you want the score-verifier-driven retry to pick a new parameter combination.

This overwrites manual edits inside emulator_setup. If you hand- tuned the parameters, save a copy first via More ▾ → Export YAML — or lean on the change history, which snapshots every save (see Change History).

Download / Inspect YAML

More ▾ → Export YAML downloads the raw YAML file as it lives on disk. Useful for:

  • Diffing two cases when one passes and the other fails.

  • Pasting into a Jira ticket / chat for handover.

  • Manually editing fields the editor doesn’t expose (rare — usually only experimental converter output keys, e.g. kpi_type, which is read by the tree-coverage pre-flight, the score simulator, and the converter retry loop).

To upload a hand-edited YAML back, use the case list’s ⋯ menu → Import, or POST /api/tests/upload-yaml directly. To move a whole set of cases between stands, Test cases → Library → Import XML… and the Export XML bulk action are the wider door — see test-management.adoc#case-xml.

Saving and Validation

  • Save — POST /api/tests (new) or PUT /api/tests/{filename} (existing). Server normalises the YAML, validates the tree file path, and writes to the persistent volume.

  • On error, the toast shows the server message — typically a missing required field (id, title, portal) or an invalid tree_file reference.

The serializer writes a whitelist — an unknown field is dropped, silently

Saving does not round-trip arbitrary YAML. The server writes only the step fields it knows (_step_fields in scripts/test_server.py), so a field the runner reads but the serializer does not list is stripped on every save, turning a working step into a bare shell with no error anywhere.

This is why adding a new step type or a new field on an existing type is two changes in one commit: the editor’s render block for that type, and the serializer whitelist. If you hand-edit a YAML and the editor "loses" a field on the next save, this is why — upload the file instead of saving the form.

Two unit tests hold the line, and a new field that skips the whitelist fails them: tests/test_editor_verify_field_parity.py derives the rule from the source — a field the converter writes onto a step and an executor reads back must be whitelisted — and tests/test_step_field_roundtrip.py pushes real steps through the serializer and asserts nothing came back missing.

Validation against TestRail

Validate vs TestRail scores the case against the TestRail text as it stands today, so drift since the import is visible. The report shows a score out of 100, the per-section findings, and whether it compared against the live case or the import snapshot. The same thing runs in bulk from the list’s Validate button, and across a whole project via POST /api/validate/scan.

Related, and worth knowing before you accept a green: TestRail source (in the Steps header) renders the original case beside the converted steps — preconditions, every step’s content and its expected result, with a count of how many were converted. A TestRail step that the converter merged or dropped is visible here in seconds.

Promoting an ai_step

An ai_step that has run the same way several times can be compiled into deterministic steps: GET /api/tests/{filename}/promotable lists which steps are stable enough, and the Promote… button (or POST /api/tests/{filename}/promote) rewrites one of them from its latest trace. This is how an exploratory step graduates into a nav sequence.

After You Save

Saved cases are picked up immediately by:

  • The Test cases listing.

  • The Overview pass-rate calculation.

  • Any subsequent batch run that includes this file.

You do not need to redeploy the container; the volume is shared.