Importing Tests

There are three UI flows for landing a YAML test case onto the agent:

  1. Pull from TestRail by Project / Suite / Section (the TestRail modal).

  2. Import a whole TestRail run by its run id — the run pins its own suite and case set, and every case is labelled with the run.

  3. Import or reimport a single case by case-ID.

How the case is worded in TestRail decides whether each step converts to a deterministic action or a slower LLM fallback. Before importing, see Writing TestRail Cases for the step-phrasing grammar.
At a glance

  • One case — TestRail modal → Import single test case, paste the ID, click Import.

  • Many cases — same modal → pick project/suite (section optional) → Preview → tick rows → Import N cases.

  • A whole run — same modal → Import TestRail run → run id.

  • After a converter change — reimport (the persistent volume keeps the old YAML). Tick Force refresh to bypass the converter cache.

  • In a hurry — set LLM to No LLM: deterministic re-conversion in seconds, with no Stage-2 drift.

  • No UI / scripted — POST /api/parse-xml, /api/import-xml-batch, /api/testrail/reimport. See XML Import API.

A TestRail XML export is a fourth path: Test cases → Library → Import XML…, or the API in XML Import API. The export only names the cases; their content comes from TestRail. The Import test cases dialog has an LLM mode list. The default is No LLM, cached only.

All three feed the same Smart Converter pipeline.

The Converter tab starts the same imports. The tab also keeps a journal of every import. See Converter Tab.

TestRail Modal

Opened by the TestRail button in the Editor header.

Connection status pill at the top:

  • 🟢 Connected — credentials in Settings are valid.

  • 🔴 Disconnected — fix in Settings → TestRail.

Step 1: Pick What to Import

Three cascading dropdowns: Project, Suite, Section. The section is optional — All sections imports the whole suite. Picking them loads nothing on its own: click Preview to reach step 2.

The same screen carries two shortcuts that bypass the section selector:

  • Import TestRail run — a run id, plus an optional project label. The run decides the suite and the case set; each imported case is tagged with the label, which defaults to Run <id> — <run name>. This is how a wave is pulled in, and it is also the only membership that is trustworthy: a case’s project records the import path, not the run.

  • Import single test case — one case id (with or without the C prefix).

Conversion options, inline on the same screen:

  • Emulator serial — overrides the default emulator_setup.device_serial.

  • Model — opus (default) or sonnet for the LLM enrichment stage of the converter. Haiku was dropped from the converter UI in favour of Opus, which produces noticeably less jitter on multi-step enrichment.

  • LLM — how much LLM to spend on this import. The default is No LLM, cached only:

    • With LLM (full) — Stage 2 enrichment runs and spends LLM quota.

    • No LLM — deterministic stages only. Seconds instead of minutes, and byte-stable: the right choice when the converter change you are testing is deterministic.

    • No LLM, cached only — reuse a cached conversion, and skip any case that has none rather than building it. Force refresh is disabled in this mode. The two options ask for opposite things. A note under the options says which cases the import skips. A case never converted before has no cached conversion.

  • Knowledge MCP — when on (default), the converter pulls TR-181 spec snippets from knowledge-mcp to ground LLM enrichment for widget-scoring tests (CPU / MEM / LAN / WAN / WiFi-client). Disable to skip this enrichment.

  • Force refresh — bypass the converter’s idempotency cache. By default a case re-converts only when the TestRail input or relevant tree changes — same input → same output skipped. Tick this to force Stage 2 LLM enrichment to re-run even when nothing upstream changed (useful after converter logic changes). No effect in No LLM, cached only, the default mode.

  • Overwrite existing — when a YAML already exists for a case ID, replace it instead of skipping.

  • Assign template trees — fall back to a built-in template under /app/templates/setups/ when the TestRail case has no attached tree.

Step 2: Preview

The modal lists candidate cases — ID, Title, Steps count, and 📎 if the case has an attached XML tree.

On a large selection the preview stops probing attachments and says so (attachments_checked: false). "0 with device trees" then means not checked, not none attached — do not conclude anything about trees from a big preview.

  • Tick the rows to import. Select All for the entire section.

  • Cases without a 📎 tree attachment will use the converter’s tree fallback (template based on detected kpi_type, or bbf-tree-mcp if Assign template trees is unchecked).

Step 3: Import

Click Import. The agent:

  1. Fetches each case via TestRail REST.

  2. Downloads any attached XML tree (TR-181 dump).

  3. Validates and falls back to a template if the attachment is invalid.

  4. Runs Smart Converter.

  5. Writes the YAML to /app/tests/cases/.

  6. Persists the conversion timestamp in /app/tests/.testrail_sync.json.

  7. Maps the case’s TestRail Priority, Type, and References (Jira) and resolves its Project from the suite — assigning and registering the project automatically (re-applied on reimport and on cache hits). See Organizing Cases.

The import is asynchronous: the request returns a job_id and the modal polls it, then shows a results screen with Created / Skipped / Errors and a per-case error list. Failures name the offending case ID and the converter exception — usually a missing tree attachment or an unparseable step. The results screen also lists each skipped case with its reason. In No LLM, cached only mode, the reason is often "no cached conversion".

Only one import may run at a time per worker: a second one is refused with 409 import already running (job …).

The job record also carries llm_usage — how many CLI spawns the import cost, broken down by stage. Worth a glance after a large wave import.

XML Import API

The previous Import XML / Batch Import XML modals were removed from the Editor in favour of the TestRail-API path. The endpoints below stay for scripts.

A TestRail XML export only names the cases. The import reads each named case from TestRail by its ID. It converts the case in the same background job as the TestRail modal. The text of the XML file is not converted.

  • The import needs a working TestRail connection.

  • A case in the file and missing from its TestRail suite is reported as an error.

  • The answer is a job: {job_id, status, total}. Poll GET /api/testrail/import/jobs/<job_id> for the result.

Endpoint Purpose

POST /api/parse-xml

Convert a single <case> XML to a YAML preview (no disk write).

POST /api/import-xml-batch

Start an import job for the cases a TestRail XML export names (<suite><sections>…​).

POST /api/run-xml

Convert + run a <case> in one shot (query params model, dry_run).

Body schema for import-xml-batch (all non-xml fields optional):

# Single case preview (no write)
curl -X POST http://ai.friendly-tech.com:8083/api/parse-xml \
  -H 'Content-Type: application/json' \
  -d '{"xml":"<case>...</case>"}'

# Batch import: starts a job. Poll /api/testrail/import/jobs/<job_id>.
curl -X POST http://ai.friendly-tech.com:8083/api/import-xml-batch \
  -H 'Content-Type: application/json' \
  -d '{
    "xml":"<sections>...</sections>",
    "emulator_serial":"test-runner-001",
    "overwrite":true,
    "llm_model":"opus",
    "backend_rag":false,
    "section_filter":"TR-181 Mesh"
  }'
backend_rag defaults to false server-side. The flag is preserved for backward compatibility but is not recommended — ft_code Qdrant retrieval caused over-engineered, hallucinated TR-181 paths in earlier versions. TR-181 spec context is now pulled from knowledge-mcp and bbf-tree-mcp during LLM enrichment instead.
llm_model defaults to opus server-side. The earlier sonnet default was dropped along with the Haiku option in the UI to reduce multi-step enrichment jitter.

Reimport After Converter Changes

When the converter is modified (a new template, a fixed formula, a new preflight action), existing YAMLs become stale — the persistent test-cases volume keeps the old converted output. You must reimport to pick up the fix.

  • For one case — TestRail modal → Reimport by Case ID field, paste the ID, click Reimport.

  • For many cases, set Auto-sync Interval (minutes, 0 = off) in Settings.

    A value above 0 starts an automatic sync once per interval. The automatic sync converts at most 10 changed cases again. For more changed cases, use Sync with TestRail now. See Automatic Sync. To stop the automatic sync, set the value to 0.

  • For every stale case, click Reimport stale… on the Converter tab. See Re-import Stale Cases.

The reimport endpoint:

curl -X POST http://ai.friendly-tech.com:8083/api/testrail/reimport \
  -H 'Content-Type: application/json' \
  -d '{"case_id":7677}'
Reimport overwrites the YAML on disk. Manual edits in the Editor are lost. If you have hand-tuned edits, save the YAML first via ⬇ Export YAML, or make the converter fix permanent so reimport produces the right output.

After Import

After the toast confirms a successful import, the case appears immediately in:

  • The Tests tab list (refresh if not auto-updated).

  • The Editor tab list — open it to verify emulator_setup and steps.

  • /api/tests — programmatic listing.

Then either run it (Running Tests) or jump into the Editor to fine-tune.