Importing Tests

There are two UI flows for landing a YAML test case onto the agent:

  1. Pull from TestRail by Project / Suite / Section (the TestRail modal).

  2. Reimport a single case by case-ID after a converter change.

How the case is worded in TestRail decides whether each step converts to a deterministic action or a slower LLM fallback. Before importing, see Writing TestRail Cases for the step-phrasing grammar.
At a glance

  • One case — TestRail modal → Reimport by Case ID, paste the ID, click Reimport.

  • Many cases — same modal → pick project/suite/section, tick rows, Import.

  • After a converter change — reimport (the persistent volume keeps the old YAML). Tick Force refresh to bypass the converter cache.

  • No UI / scripted — POST /api/parse-xml, /api/import-xml-batch, /api/testrail/reimport. See XML Import API (no UI).

A third path — pasting raw TestRail XML — is API-only (no UI) and is covered in XML Import API.

All three feed the same Smart Converter pipeline.

TestRail Modal

Opened by the TestRail button on the Editor tab header.

Connection status pill at the top:

  • 🟢 Connected — credentials in Settings are valid.

  • 🔴 Disconnected — fix in Settings → TestRail.

Step 1: Pick Project / Suite / Section

Three cascading dropdowns: Project, Suite, Section. Selecting a section loads its cases into the preview list below.

The same step also has a Reimport by Case ID field at the bottom — paste a single case ID (with or without the C prefix) and click Reimport to re-convert one case in isolation. This shares the page with the section selector — there are no separate sub-tabs.

Conversion options (collapsed by default):

  • Emulator serial — overrides the default emulator_setup.device_serial.

  • Model — opus (default) or sonnet for the LLM enrichment stage of the converter. Haiku was dropped from the converter UI in favour of Opus, which produces noticeably less jitter on multi-step enrichment.

  • Knowledge MCP — when on (default), the converter pulls TR-181 spec snippets and matching QA Validation Notes from knowledge-mcp to ground LLM enrichment for widget-scoring tests (CPU / MEM / LAN / WAN / WiFi-client). Notes that match the test’s scope (test_id, kpi_type, wifi_health_topo) are injected as a "QA Validation Notes (team-authored, prefer over priors)" section in the Stage 2 prompt. Disable to skip both kinds of enrichment.

  • Force refresh — bypass the converter’s idempotency cache. By default a case re-converts only when the TestRail input or relevant tree changes — same input → same output skipped. Tick this to force Stage 2 LLM enrichment to re-run even when nothing upstream changed (useful after converter logic changes).

  • Overwrite existing — when a YAML already exists for a case ID, replace it instead of skipping.

  • Assign template trees — fall back to a built-in template under /app/templates/setups/ when the TestRail case has no attached tree.

Step 2: Preview

The modal lists candidate cases — ID, Title, Steps count, and 📎 if the case has an attached XML tree.

  • Tick the rows to import. Select All for the entire section.

  • Cases without a 📎 tree attachment will use the converter’s tree fallback (template based on detected kpi_type, or bbf-tree-mcp if Assign template trees is unchecked).

Step 3: Import

Click Import. The agent:

  1. Fetches each case via TestRail REST.

  2. Downloads any attached XML tree (TR-181 dump).

  3. Validates and falls back to a template if the attachment is invalid.

  4. Runs Smart Converter.

  5. Writes the YAML to /app/tests/cases/.

  6. Persists the conversion timestamp in /app/tests/.testrail_sync.json.

  7. Maps the case’s TestRail Priority, Type, and References (Jira) and resolves its Project from the suite — assigning and registering the project automatically (re-applied on reimport and on cache hits). See Organizing Cases.

A toast confirms the count of converted / skipped cases. Failures are listed with the offending case ID and the converter exception — usually a missing tree attachment or an unparseable step.

XML Import API (no UI)

The previous Import XML / Batch Import XML modals were removed from the Editor in favour of the TestRail-API path. The same converter is still exposed over HTTP for air-gapped or read-only TestRail accounts.

Endpoint Purpose

POST /api/parse-xml

Convert a single <case> XML to a YAML preview (no disk write).

POST /api/import-xml-batch

Convert and persist a full TestRail XML export (<sections><cases>…​).

POST /api/run-xml

Convert + run a <case> in one shot (query params model, dry_run).

Body schema for import-xml-batch (all non-xml fields optional):

# Single case preview (no write)
curl -X POST http://ai.friendly-tech.com:8083/api/parse-xml \
  -H 'Content-Type: application/json' \
  -d '{"xml":"<case>...</case>"}'

# Batch import (writes YAMLs).
curl -X POST http://ai.friendly-tech.com:8083/api/import-xml-batch \
  -H 'Content-Type: application/json' \
  -d '{
    "xml":"<sections>...</sections>",
    "emulator_serial":"test-runner-001",
    "overwrite":true,
    "llm_model":"opus",
    "backend_rag":false,
    "section_filter":"TR-181 Mesh"
  }'
backend_rag defaults to false server-side. The flag is preserved for backward compatibility but is not recommended — ft_code Qdrant retrieval caused over-engineered, hallucinated TR-181 paths in earlier versions. TR-181 spec context is now pulled from knowledge-mcp and bbf-tree-mcp during LLM enrichment instead.
llm_model defaults to opus server-side. The earlier sonnet default was dropped along with the Haiku option in the UI to reduce multi-step enrichment jitter.

Reimport After Converter Changes

When the converter is modified (a new template, a fixed formula, a new preflight action), existing YAMLs become stale — the persistent test-cases volume keeps the old converted output. You must reimport to pick up the fix.

  • For one case — TestRail modal → Reimport by Case ID field, paste the ID, click Reimport.

  • For many — Settings has an Auto-sync interval (minutes). When > 0 a background loop calls TestRail once per interval and re-converts cases modified upstream. Set to 0 to disable.

The reimport endpoint:

curl -X POST http://ai.friendly-tech.com:8083/api/testrail/reimport \
  -H 'Content-Type: application/json' \
  -d '{"case_id":7677}'
Reimport overwrites the YAML on disk. Manual edits in the Editor are lost. If you have hand-tuned edits, save the YAML first via ⬇ Export YAML, or make the converter fix permanent so reimport produces the right output.

After Import

After the toast confirms a successful import, the case appears immediately in:

  • The Tests tab list (refresh if not auto-updated).

  • The Editor tab list — open it to verify emulator_setup and steps.

  • /api/tests — programmatic listing.

Then either run it (Running Tests) or jump into the Editor to fine-tune.