Editor Guide

The Editor tab is the visual front-end for the YAML test case format consumed by run_test.py. Every field maps 1:1 to a YAML key — the editor never hides data, it just makes it editable without writing YAML by hand.

Editor tab — left list + right form
At a glance

  • List on the left = every YAML on the agent (filterable by ID/title).

  • Right pane has three modes — list (no selection), new (empty form), edit (a row clicked).

  • Save persists to disk; â–¶ Save & Run also triggers a run.

  • 🔄 Regenerate rebuilds emulator_setup from the test text but keeps step edits.

  • ⬇ Export YAML / ⬆ Import — download / upload a hand-edited file.

Overview

Open the Editor tab. The layout is split:

  • Left — list of all YAML cases currently on the agent (under /app/tests/cases inside the container, mounted as the persistent test-cases volume). Search box filters by ID or title.

  • Right — the form. Three modes:

    • list — empty placeholder (no case selected, "+ New" not pressed).

    • new — empty form for authoring a fresh case.

    • edit — populated form for an existing case.

Left-panel header buttons:

  • TestRail — opens the import modal (Importing Tests).

  • ⬆ Import — upload a hand-edited .yaml file (POST /api/tests/upload-yaml).

  • + New — clear the form and switch to new mode.

  • 🗑 Delete — opens a dropdown with bulk-delete actions (see Bulk Delete Menu (🗑 Delete)).

Periodic TestRail refresh is handled by the background sync loop (see Auto-sync Interval).

Right-panel header buttons (visible only in edit mode):

  • ⬇ Export YAML — download the current YAML file.

  • Delete — remove the YAML file from disk (single test).

  • Regenerate — regenerate emulator_setup from the test text (see Regenerate Emulator Setup).

  • Save — persist changes to disk.

At the bottom of the form there is also ▶ Save & Run — saves the case and immediately starts a run with the model picked in the Tests tab.

There is no auto-save. After every change press Save (or Save & Run) at the bottom.

Bulk Delete Menu (🗑 Delete)

The red trash icon in the left-panel header opens a dropdown with two destructive actions:

  • Delete All Tests — calls DELETE /api/tests and wipes the entire /app/tests/cases directory. Use after a major converter rewrite when you intend to fully reimport from TestRail.

  • Delete by Section — lists every distinct section value in the current YAMLs, with a count next to each. Click a section to delete only those tests. Useful for cleaning up an obsolete TestRail section without losing the rest.

Both actions are irreversible. The persistent volume is overwritten on disk. Reimport from TestRail to restore.

Core Fields

These are the top-level keys that drive the runner before steps are even considered.

Field YAML key Notes

Test ID

id

Format C<TestRail-id>, e.g. C7677. Used as a stable identifier and as the report row title.

Portal

portal

support or management. Dictates which portal the runner logs into and which sidebar / widgets are available.

Section

section

Free-text grouping (e.g. TR-181, TR-181 Mesh). Pure metadata, used for list filtering.

Title

title

Human-readable test title, shown in the list and reports.

Project

project

Dropdown of the projects defined under Organizing Cases: Projects & Sections. Groups the case for list filtering and reporting alongside section.

Priority

priority

Low / Medium / High / Critical (free values also allowed). Rendered as a colored pill in the list and edit header (green / blue / amber / red).

Type

type

Test category — Acceptance, Functional, Performance, Regression, Security, Smoke & Sanity, Usability, Other (custom values preserved). Pure metadata.

Reference

reference

Jira ticket reference(s), e.g. DEV-1234. Rendered as a clickable link when Settings supplies a jira_base_url (Settings); otherwise shown as plain text.

Base URL

base_url

Defaults to the value from Settings if left empty. Override only when one specific case must hit a non-default portal.

Username / Password

credentials.username / .password

Same fallback rule — empty here means use the Settings defaults.

Preconditions

preconditions

Plain-text bullet list (one per line). Read by the LLM during conversion to deduce required emulator_setup.parameters, not parsed at runtime.

Emulator Serial

emulator_setup.device_serial

The CWMP serial number of the emulated CPE. Defaults to test-runner-001. Must match the device the agent navigates to in the portal.
In edit mode an attribution strip above the form shows Created by (created_by) and Last edited by (edited_by) with timestamps, plus a per-case edit history. These are written by the server from the logged-in account — they are read-only and not part of the editable form. See Organizing Cases: Projects & Sections for the project / section taxonomy these fields feed into.

Emulator Setup

Most tests need state on the emulator — TR-181 parameters, dynamic instances, diagnostic results — for the dashboard / portal to render the expected value. This is configured under Emulator Setup.

If the block is missing, click + Add Emulator Setup and an empty stub is created (tree_file, parameters, delete_objects, instance_limits, wifi_client_limit, ui_preconditions, phases). All sub-fields are optional.

Execution order (server-side, before the first step runs):

  1. Push tree_file to the emulator (factory reset → upload tree).

  2. Apply delete_objects (CWMP DeleteObject).

  3. Apply instance_limits (truncates dynamic tabular objects to N instances).

  4. Apply wifi_client_limit (special-case for Hosts.Host).

  5. Set parameters via SetParameterValues.

  6. Run ui_preconditions against the management backend (see UI Preconditions).

  7. Trigger ACS Inform so backend reads the new state.

  8. If phases[0] exists, apply phase 1 parameters and continue.

Tree File and "Generate Tree"

tree_file is the name of an XML file under /opt/mcp-shared/ on the test-runner host. It defines the initial TR-181 tree pushed into the emulator (factory reset baseline). Common templates:

File Purpose

181.xml

Default Fiber CPE: CPU, MEM, 4 radios, full diagnostics. Fallback for unrecognised KPI types.

181-cellular.xml

Cellular CPE.

181-mesh-de.xml / 181-mesh-multiap.xml

Mesh CPEs with DataElements / MultiAP data models.
Selecting a different tree_file does not re-push the tree. It only changes which file is used the next time the test runs. To verify what’s currently on the emulator, query its API directly (see Troubleshooting).

The Tree File row exposes three controls:

  • Tree dropdown — populated from GET /api/trees, which lists every *.xml in /opt/mcp-shared/. The currently selected file is always pre-rendered as the first option so the dropdown reflects the saved value immediately, even when the list is still loading or the file is no longer on disk.

  • Upload — opens a file picker. Selecting an .xml calls POST /api/trees/upload (multipart). The server validates the payload as well-formed XML, sanitises the basename (no path traversal), and writes it into /opt/mcp-shared/. On success the dropdown refreshes and the new name is auto-selected. Re-uploading the same name replaces the existing file — the response carries replaced: true.

  • Generate Tree — described below.

Generate Tree

The Generate Tree button calls the bbf-tree-mcp server with a natural- language description of the test (title + preconditions + first step). The MCP returns a fresh XML tree tailored to the case. Workflow:

  1. Edit Title and Preconditions so they describe the device state you want.

  2. Click Generate Tree.

  3. A modal appears with a live status pane (connecting → reading preconditions → calling MCP). Generation typically takes 10–30 s but can run minutes for large mesh topologies.

  4. On success the modal shows the new filename and the tree_file field in the form is updated. Save the test to persist.

  5. On failure the modal shows the MCP error verbatim — fix the preconditions text or fall back to a template under /opt/mcp-shared/ (use the dropdown).

Use this when none of the existing templates fit (unusual KPI mix, exotic data model, multi-radio mesh topology).

Parameters

Key/value list of TR-181 paths that override the tree after factory reset. Examples:

parameters:
  Device.WiFi.AccessPoint.1.AssociatedDevice.1.SignalStrength: "-55"
  Device.WiFi.AccessPoint.1.AssociatedDevice.1.Noise: "-95"
  Device.WiFi.AccessPoint.1.AssociatedDevice.1.OperatingStandard: "ax"

Editing in the UI:

  • + Add opens an inline row — paste the full Device.<…​> path and the value. Both fields are free-text monospace.

  • The big monospace block lists existing entries — click in the value field to edit in place; the x button removes.

  • Values are sent as strings to the emulator; numeric/boolean coercion happens on the emulator side per TR-181 type.

SNR is not a TR-181 parameter — it’s computed as SNR = SignalStrength - Noise. To get a target SNR set both SignalStrength and Noise accordingly.

Delete Objects and Instance Limits

Delete Objects

List of TR-181 instance paths to delete after the tree is pushed. Useful when the template tree contains a parameter the test wants absent (e.g. delete OperatingStandard to test the missing-param weight redistribution path).

Instance Limits

Map of parent path → N. Backend truncates the dynamic tabular collection at parent to keep only N instances. Example: Device.Ethernet.Interface.: 2 keeps only the first two Ethernet interfaces.

WiFi Client Limit

Special integer — caps Device.Hosts.Host.{i} to the given count. Equivalent to setting Device.Hosts.Host.: N in Instance Limits, but exposed separately because it is the most common case.

UI Preconditions

Pre-test actions executed against the management backend’s REST API (not against the portal browser). Each entry has an action and params. The server dispatches them through the _PREFLIGHT_ACTIONS table; the full set is below, grouped by purpose.

Table 1. Monitoring
Action Purpose

CREATE_MONITORING

Creates a monitoring rule on the device so the network-history widget has data points. Mostly auto-applied; rarely edited by hand. (Also the anti-flap trigger — its presence disables the autonomous Periodic Inform for the run.)

DELETE_MONITORING

Removes a monitoring rule (the inverse of CREATE_MONITORING) — for tests that assert monitoring is off.

TRIGGER_MONITORING_TASKS

Forces the backend’s monitoring tasks to run now instead of waiting for the next scheduled pull, so freshly set state is reflected immediately.

CREATE_GROUP_UPDATE

Creates a group update with the selected diagnostics list and optional reactivation flag (periodic re-run). Used for tests that exercise the group update widget.
Table 2. Diagnostics
Action Purpose

RUN_DIAGNOSTICS

Generic diagnostic runner. Supports IPPingDiagnostic, DownloadDiagnostic, UploadDiagnostic, NSLookupDiagnostic, TraceRouteDiagnostic, NeighboringWiFiDiagnostic. For IPPing the host field controls latency band (8.8.8.8 for "good", 203.50.2.71 for "bad").

RUN_INTERFERENCE_DIAGNOSTICS

Triggers NeighboringWiFiDiagnostic and (if INTERFERENCE_DASHBOARD_SEED is on) seeds the ClickHouse interference dashboard.

RUN_NEIGHBORING_WIFI_DIAGNOSTICS

Alias of RUN_INTERFERENCE_DIAGNOSTICS — runs the neighboring-Wi-Fi scan (same handler), named after the diagnostic for readability.

RUN_UPLOAD_DIAGNOSTIC

Runs an UploadDiagnostic so the upload-speed result is populated before the test reads it.

VERIFY_DEVICE_ONLINE

Polls the backend until the device reports online — a guard step so later actions don’t run against a still-booting CPE.
Table 3. ACS / parameters
Action Purpose

ACS_TASK

Drives the ACS via the management REST API (SPV / GPV / add / delete object / reboot / factory reset) as a precondition. Same surface as the acs_task step type (see Steps), run before the first step.

SET_ACS_PARAM

Sets an ACS configuration parameter (e.g. overwriteProvisionParameters) that must be in place before the run.

SET_CPE_LOGIN_GENERATE

Enables ACS Basic / Digest credential generation for the device (sets the CPE login + generateCredentials flags). Required precondition for the credential-length tests driven by cred_length_verify; restored on post-flight.

SET_TARIFF

Assigns a tariff plan to the device (static QoE tariff seed).

SET_TARIFF_DYNAMIC

Assigns a dynamic tariff (used by diagnostic / QoE tests that need a rate-limited plan).

SET_UI_SETTING

Writes a portal UI setting (e.g. an Advanced-View toggle) the test depends on.
Table 4. Files / devices
Action Purpose

ADD_DEPLOYMENT_FILE

Registers a deployment (firmware / config) file in the platform so a file-download / deployment test can select it.

ADD_DOWNLOAD_FILE

Registers a download file entry the device can be told to fetch.

CREATE_NBI_DEVICE

Creates a device via the North-Bound Interface (NBI) before the run — for tests that need a freshly provisioned device record.

device_serial and portal are auto-filled from the test’s top-level fields, so you only need to choose the action and (if relevant) the diagnostic type, host, or action-specific params.

Multi-Phase Tests

Some TestRail cases are written as a sequence of "set state A → verify; set state B → verify". The converter encodes them as a single YAML with multiple phases. Each phase has a name and its own parameters map.

Editing:

  1. Click + Add Phase to append. The button appears both in the populated Phases header and in the empty-state row when no phases exist yet — fresh tests can grow phases from zero without touching YAML.

  2. Each phase shows an editable name and a parameter map (same UI as the main Parameters block).

  3. Order matters — phases run in array order. Use the x to remove.

  4. The corresponding phase step in the steps list (see Steps) is what triggers the runtime to apply that phase.

Phases share the base parameters — phase entries are additive overrides on top of the initial tree + base parameters, not full replacements.
A phase with an empty parameters map is a scenario marker — the runtime treats the matching step as instant pass and only re-runs any diagnostic preconditions tied to the phase. Such empty phases are preserved through a load → save round trip so converter-emitted markers (common in multi-scenario tests) survive in-place edits.
Converter-emitted phases sometimes carry extra fields like preflight (e.g. SET_TARIFF_DYNAMIC for diagnostic tests) and diag_params. The editor preserves those fields verbatim — they are out-of-band of the visible inputs but round-trip cleanly on save.

Steps

Steps is a flat list. Each step has:

Field Meaning

Type

One of nine step types. The dropdown exposes all of them: nav, interact, phase, acs_task, emulator_action, spa_task_verify, db_query, log_check, cred_length_verify. Empty type defaults to interact. See the Step Type reference below for what each one does and its fields.

Action

Free-text. For nav steps it’s a hint for the report; for interact steps it’s the prompt the LLM sees ("Open Wi-Fi tab and verify the SSID is 'TestSSID'"). Used as a label for the deterministic verify types and ignored for phase / emulator_action.

Expected

What the verifier should look for. Programmatic verifier scans the DOM; LLM verifier reads the rendered page text.

nav_action

Only for nav type. ~25 actions grouped in the dropdown (Navigation, Dropdowns / dialog, File Download, File Upload, Diagnostics / Advanced View / misc). See the nav_action reference below for the full list and one-line purpose each.

nav_target

Label text for the action (sidebar item, tab name, button text, widget caption, dropdown trigger label, option text). E.g. nav_action: widget, nav_target: Network Score. (Some actions take an action-specific field instead — e.g. export_verify reads report_format = excel / csv.)

wait_for_content_change

Optional checkbox for click_button. The runner takes a content snapshot before the click and waits until it differs.

phase_index

Only for phase type. 1-based index into emulator_setup.phases[] selecting which delta to apply at this step. The editor renders this as a dropdown of existing phases by their name, e.g. 2: Scenario 2: The "LAN" KPI is displayed as "Good". The currently-bound phase is rendered as the first option so the dropdown shows the saved value even before the phase list finishes loading. If no phases are defined an inline hint points back to Emulator Setup → Phases.

emulator_action

Only for emulator_action type. The MCP tool name — typically set_parameters_batch or invoke_event.

params

Only for emulator_action type. Map of parameters / arguments to forward to the MCP tool.

Use the up / down arrows in each step row to reorder; ✕ removes; + Add Step appends.

Step Type Reference

The nine step types and what each does. For step phrasing (how to word action / expected so the converter and verifier agree), see Writing TestRail Cases.

Type Purpose & key fields

nav

Deterministic browser script (fast, never stalls). Drives the portal via a nav_action + nav_target. See the nav_action reference.

interact

Handed to Claude with a snapshot-action loop. action is the prompt, expected is what the verifier looks for. Use when the step needs reasoning.

phase

Apply a multi-phase emulator delta and refresh state. phase_index selects the 1-based entry in emulator_setup.phases[] (see Multi-Phase Tests).

acs_task

Drive the ACS through the management REST API. acs_action = spv / gpv_direct / add_object / delete_object / reboot / factory_reset, with params, push, reprovision, verify_tasks_created, and wait_seconds.

emulator_action

Push parameters / invoke Inform directly through the emulator MCP, no UI. emulator_action is the MCP tool name (set_parameters_batch / invoke_event); params are its arguments.

spa_task_verify

Verify a Device-Activity SPA monitoring task was emitted (e.g. a "Set parameter attributes / Notification" task) after a portal action.

db_query

Assert against the Platform DB (MySQL / Oracle). Requires the Platform DB connection to be configured in Settings.

log_check

Post-test SSH grep of the server logs. Warn-only — it attaches its findings to the report but never fails the test. Requires Log Server settings in Settings.

cred_length_verify

Validate generated-credential min/max length. Fields: set_min / set_max (optionally push credentialMinLength / credentialMaxLength first), min_len / max_len (expected random-length window), oui, and poll_seconds. Used by the credential-length feature (C91182) together with the SET_CPE_LOGIN_GENERATE precondition.
Of these, only nav, interact, phase, and acs_task get tailored inputs in the editor form. The remaining types are typically converter-emitted; edit their fields via the YAML download / upload path (see Download / Inspect YAML).

When Type is nav, the nav_action dropdown groups ~25 actions. One-line purpose each:

Table 5. Core (Navigation + Dropdowns / dialog)
nav_action Purpose

login

Log into the portal (always the first step).

sidebar

Click a left-sidebar item (nav_target = item label).

tab

Switch tab inside a widget / page.

click_device

Click the row of the test device (auto-targets the emulator serial).

click_button

Click any button by label; optional wait_for_content_change.

widget

Open a dashboard widget (nav_target = widget caption).

expand_dropdown

Open a Material mat-select dropdown.

select_option

Pick an option in an open mat-select dropdown.

hover_button

Hover a button (reveals tooltips / hover-only UI).

scroll_to_section

Scroll a named section into view.

press_escape

Send Escape (close an overlay / dropdown).

check_row

Tick the checkbox on a table row.

hover_in_dialog

Hover an element inside an open dialog.
Table 6. File Download (deterministic, self-verifying)
nav_action Purpose

download_dropdown_verify

Open the File-Download Add-form dropdown and assert a value is present / absent.

download_form_verify

Read the File-name dropdown placeholder and assert the default text.

download_fill_verify

Type into Description / Username / Password and confirm each is populated.

download_create_verify

Create a download task and verify a row renders in the list.
Table 7. File Upload (deterministic, self-verifying)
nav_action Purpose

upload_file

Upload a file (URL mode default or explicit URL).

upload_open_form

Open the Upload Add form.

upload_delete

Delete an uploaded file (optionally from FTP).

upload_get_file

Fetch / read back an uploaded file entry.
Table 8. Diagnostics / Advanced View / misc
nav_action Purpose

diagnostic_dropdown_verify

Collect an open diagnostic dropdown’s options and self-verify (min count / max numeric).

diagnostic_field_verify

Assert named fields are present / absent in the diagnostic Add form.

advanced_view_set_param

Fill named param value(s) in the Advanced-View tree edit mode.

advanced_tree_select

Click a child tree node and assert the selection cascade.

device_activity_method_verify

Open the Device-Activity "Task name" filter and assert the target method is listed.

export_verify

Click export and confirm a 2xx report request fired. Takes report_format (excel / csv).

network_map_show

Open the network-map modal and collect its topology text for the verifier.

Step Type Cheat Sheet

Goal Use type Example

Log into the portal

nav + nav_action: login

always the first step

Click an item in the left sidebar

nav + nav_action: sidebar

nav_target: Devices

Switch tab inside a widget / page

nav + nav_action: tab

nav_target: Wi-Fi

Click the row with the test device

nav + nav_action: click_device

auto-targets emulator_serial

Click any other button

nav + nav_action: click_button

nav_target: Save

Open a dashboard widget

nav + nav_action: widget

nav_target: Network Score

Open a Material Design select dropdown (mat-select rendered in cdk-overlay-container)

nav + nav_action: expand_dropdown

nav_target: Configuration

Pick an option in an open mat-select dropdown

nav + nav_action: select_option

nav_target: 5GHz

Apply the next phase from emulator_setup.phases[] (push params + refresh)

phase + phase_index

phase_index = 2

Push TR-181 params directly without UI (no portal click — emulator-only state change)

emulator_action + emulator_action: set_parameters_batch

params: { "Device.WiFi.Radio.1.Channel": "11" }

Verify a value or perform any UI action that needs reasoning

interact

action = "Verify the Health value is 'Good'"
Prefer nav whenever the action has a fixed label and no decision-making — it is roughly an order of magnitude faster than interact and never stalls.
expand_dropdown issues an Escape only when the next step is not select_option, so the open dropdown stays accessible to the option click. The selector logic walks cdk-overlay-container because Angular Material renders the option list at the end of the DOM, not inside the trigger element.

Regenerate Emulator Setup

The Regenerate button (top-right of the form, edit mode only) re-runs the converter pipeline using the current text fields (preconditions, steps, expected, title) and replaces the emulator_setup block while leaving steps and metadata untouched.

A modal appears with live status (saving edits → re-running the converter → score-simulator retry → reload). On success the modal shows the new tree_file, parameter count, and any delete_objects / wifi_client_limit that were derived. On failure the converter error is shown verbatim.

When to use:

  • You changed Preconditions or step Expected values and want the emulator state regenerated.

  • You’re investigating why a test was imported with the wrong tree / parameters.

  • The backend formula changed and you want the score-verifier-driven retry to pick a new parameter combination.

This overwrites manual edits inside emulator_setup. If you hand- tuned the parameters, save a copy first via ⬇ Export YAML.

Download / Inspect YAML

The ⬇ Export YAML button downloads the raw YAML file as it lives on disk. Useful for:

  • Diffing two cases when one passes and the other fails.

  • Pasting into a Jira ticket / chat for handover.

  • Manually editing fields the editor doesn’t expose (rare — usually only experimental converter output keys, e.g. kpi_type, which is read by the tree-coverage pre-flight, the score simulator, and the converter retry loop).

To upload a hand-edited YAML back, use the Editor tab header → ⬆ Import action, or POST /api/tests/upload-yaml directly.

Saving and Validation

  • Save — POST /api/tests (new) or PUT /api/tests/{filename} (existing). Server normalises the YAML, validates the tree file path, and writes to the persistent volume.

  • On error, the toast shows the server message — typically a missing required field (id, title, portal) or an invalid tree_file reference.

After You Save

Saved cases are picked up immediately by:

  • The Tests tab listing.

  • The Dashboard regression-pass-rate calculation.

  • Any subsequent batch run that includes this file.

You do not need to redeploy the container; the volume is shared.