Test Readiness

Each test case on the stand has one readiness label. The label tells you if the test agent can run the case and check all of it.

The test agent reads the case file and its TestRail text before any run. This reading is the analysis. The last run of the current case file refines the result. A run counts from any environment (env) that fits the case. A run on an env that does not fit does not change the label.

A gap is a TestRail line with no matching step or check. Each gap has a code, a source and the exact TestRail line.

Readiness Labels

A case has exactly one label. If several labels apply, the first label in the table wins.

Label Meaning

Manual only

A person marked the case. The test agent does not automate it.

Postponed

A person marked the case for later work.

Reimport first

The case file is older than its TestRail text or the converter. Re-import the case before you read its gaps.

Cannot run

The stand, the test agent, the portal or a product bug stops the case. A need for an Oracle database does not stop a case.

Coverage gap

The case runs. A precondition, a step or an expected result has no check.

Ready

The analysis finds no gap. No run of the current case file confirms the result yet.

Ready ✓ confirmed

The analysis finds no gap. The last run of the current case file passed the audit.

Unknown

The test agent has no label for the case yet. Pending means a later list load computes the label.

The source of a gap decides the label:

Source Meaning

stand

The stand lacks a device, a service or a setting.

agent

The test agent has no action or check for the line.

portal

The portal has no such control or page.

product_bug

A known product bug stops the line.

converter

The converter left the line out of the case file.

adapter

The adapted TestRail text lacks a step.

seed

The test agent did not create or confirm a precondition.

probe

No check measures the expected result.

qa

The expected result is too vague for a measurement.

A gap with the source stand, agent, portal or product_bug gives Cannot run. A gap with any other source gives Coverage gap.

A timeout or an unknown error in a run does not change the label. The readiness detail lists these runs.

Read the Label of a Case

  1. Open the Test cases tab.

    Each row shows the readiness label as a pill.

  2. Point at the pill.

    The tooltip shows the first gap and the number of gaps.

  3. Click the pill.

    The readiness detail of the case opens.

The readiness detail has these parts:

  • The label and the reason for the label.

  • The mark and its reason, if a person marked the case.

  • The last run of the current case file that counts.

  • The env requirements and the env notes of the case.

  • The gaps by axis: Case, Preconditions, Steps and Expected.

  • The flags. A flag is a risk, not a gap.

  • The runs that did not count, for example a timeout.

Each gap row shows the Code, the Source, the Line, the TestRail Text and the Evidence. If the evidence names a run, click Open run to open the run report.

Filter by Readiness

  1. In the Test cases tab, click Readiness: any in the filter row.

  2. Select one or more labels.

    The list shows only the cases with these labels.

  3. To filter by a gap, click More filters.

  4. In the Any gap code list, select a code.

    The list shows only the cases with this gap code.

  5. In the Any source list, select a source.

    The list shows only the cases with a gap from this source.

A filter changes the list only. Selected cases stay selected, even when the filter hides them.

Mark a Case Manual Only or Postponed

You need the editor or the admin role to mark a case. Each mark needs a reason from a fixed list:

Mark Reason Use

Manual only

Unsafe on the shared stand

The case changes data for every user of the stand.

Manual only

Needs human judgment

Only a person can judge the result.

Manual only

Other (note required)

Any other reason. Type the reason in Note.

Postponed

Awaiting a QA or owner decision

The case waits for a decision.

Postponed

Feature missing in the new portal

The new portal lacks the feature.

Postponed

Too costly now

The automation costs too much now.

Postponed

Other (note required)

Any other reason. Type the reason in Note.

To mark one case:

  1. Click the readiness pill of the case.

  2. In the readiness detail, click Mark….

  3. In Mark, select Manual only or Postponed.

  4. In Reason, select a reason.

  5. If the reason is Other (note required), type a note in Note.

  6. Click Save mark.

    The case shows the new label.

Save mark stays disabled while the reason is Other (note required) and Note is empty. Note holds up to 500 characters. A mark belongs to the test identifier (ID), so one mark covers every scenario file of that case.

To mark several cases:

  1. In the Test cases tab, select the cases.

  2. In the bar above the list, click Mark….

  3. In Mark, select the mark.

  4. In Reason, select the reason.

  5. If the reason is Other (note required), type a note in Note.

  6. Click Save mark.

    Each selected case shows the new label.

To clear a mark:

  1. Click the readiness pill of the case.

  2. Click Clear mark.

  3. In the dialog, click Clear mark.

    The case gets its computed label back.

To clear the marks of several cases:

  1. In the Test cases tab, select the cases.

  2. In the bar above the list, click Clear mark.

    The button label ends with the number of selected marked cases.

  3. In the dialog, click Clear mark.

    Each selected case gets its computed label back.

A re-import keeps the mark. The analysis never removes a mark. The mark history keeps the 20 latest entries of a case. The current mark and each clear count as entries. Older entries drop out.

Start a Batch

The batch exclusion leaves cases out of a batch. An administrator switches the batch exclusion on or off in the Readiness tab. The switch applies to every account. The switch is off by default.

If the batch exclusion is off, a batch runs every selected case.

If the batch exclusion is on, a batch leaves out Manual only, Postponed and Cannot run cases by default:

  1. In the Test cases tab, select the cases.

  2. Click Run Selected.

    If the selection holds a case to leave out, a dialog opens. The dialog shows the number of cases per label.

    The re-import choice rebuilds the case files without the large language model (LLM). The LLM enriched some case files. These files lose the enrichment. No batch starts with this choice.
    Include anyway also runs the Manual only cases. Some of them are unsafe on the shared stand.

  3. In What to run, select a choice.

  4. Click Continue.

Choice Result

Leave them out

The batch runs the other cases. This choice is the default. A message shows how many cases the batch left out.

Leave out only Manual only and Postponed

The batch runs the Cannot run cases and leaves out the marked cases. The dialog offers this choice only if the selection holds marked cases and other cases to leave out.

Include anyway

The batch runs every selected case.

Re-import the Reimport first cases

The re-import dialog opens for the Reimport first cases. No batch starts.

The dialog applies these rules:

  • Some stop reasons can change on the next run: product_bug.precondition, portal.control_not_found, agent.covered_click, agent.wrong_screen, agent.not_performed and agent.refused.

  • A Cannot run case runs if all its stop reasons are of this kind.

  • A Cannot run case with any other stop reason is left out. Examples: a stand gap, a known product bug.

  • A Cannot run case with no gap code is left out.

  • If the dialog opens for other cases, it names the Cannot run cases that still run.

  • The dialog counts the cases that need another env. Example: "2 need another env (Oracle)".

  • The batch leaves out a case with the tag Needs Oracle on an env without Oracle.

  • The case runs if the test agent does not know the database of the env.

  • On an env other than Default, Cannot run cases run.

  • Only then the dialog adds this line:

    Cannot run cases still run on <env>: that env may have what they lack.

  • On every env, the batch leaves out Manual only and Postponed cases. The dialog offers the re-import for Reimport first cases.

  • If no case is left out and no case is Reimport first, no dialog opens and every case runs.

  • A Manual only or Postponed case is left out, even while its label is still Pending.

  • Other Pending cases and Unknown cases run when the batch starts.

If the selection holds Reimport first cases and no case to leave out, the dialog offers Include anyway and the re-import. The default choice is Include anyway. The Reimport first cases run unless you select the re-import.

If the test agent cannot read the switch state, the Readiness tab shows "unknown" for the switch. The batch leaves out no case.

The dialog opens when a batch starts from these controls, with any number of cases:

  • Run Selected with two or more cases. A large selection first asks you to confirm with Run all.

  • The Rerun… list in the batch report.

  • Duplicate run in the batch report.

  • Run deferred in the batch report.

These controls run one case and open no dialog:

  • Run Selected with one case.

  • Rerun in a run report.

  • Run on a draft row.

  • Save & Run in the editor.

Env Requirements

A case can need a feature of the env, for example an Oracle database. This need is not a gap. The case keeps its other labels.

  • The tag Needs Oracle marks the case next to its readiness pill.

    Point at the tag to read "Runs on an env with an Oracle database".

  • The readiness detail shows "Needs an Oracle env".

  • In the Test cases tab, the Needs: any list filters the cases by requirement.

  • In the Readiness tab, a count shows the cases per requirement.

A run counts when its env fits the case. A case without a requirement fits every env. The test agent ignores a run on an env without the required database.

A run can fail because of one env setting, for example the Internet service provider (ISP) mode. The test agent records this refusal as an env note on the case. An env note is not a gap and does not change the label. The readiness detail lists the env notes.

Passed and Partial

A pass of a case with a gap checks only part of the case.

In the Test cases tab, the last run of such a case shows partial next to PASS. Point at partial to see the first unchecked line.

The batch report splits the passes into three groups:

Group Meaning

confirmed

The pass confirms the label of the case.

partial

The run passed, but it is not the run that confirmed the label. The label is Coverage gap or Cannot run.

not judged

The run passed, but it is not the run that confirmed the label. The label is any other label.

Readiness Tab

To open the tab, click Readiness in the sidebar.

The tab has these parts:

  • The number of cases per label. Click a label to list its cases in the Test cases tab.

  • The number of cases per portal and label.

  • The Gap codes table.

  • The Manual only and Postponed table, with the reason, the note and the author of each mark.

  • The count QA verified. Click it to list the frozen cases in the Test cases tab. See QA-Verified Cases.

To narrow the tab to one portal, pick it in the Portal list. The label counts, the Gap codes table and the marks show that portal only. The Sample column always shows the samples of the whole stand. The test agent keeps your choice in the browser. The default is All portals.

When you click a label or a gap code, the test agent reads the case list again. Then it opens the Test cases tab with the search and project filters cleared.

The Gap codes table has these columns:

Column Meaning

Code

The gap code.

Source

The source of the gap.

Cases

The number of cases with this code.

Only gap

The number of cases with no other gap. A fix of this code makes these cases ready.

Sample

One TestRail line with this code.

The table starts with the largest Only gap value. Click a row to list its cases in the Test cases tab.

To save the data, click one of these buttons:

  • Export listed cases (CSV): one row per case, with the label, the gap codes and the first gap.

    The export holds every case except drafts and explore cases. Only the Portal list changes it.

  • Export summary (Markdown): the label counts, the gap codes and the marks. The first line names the portal.

An administrator sees two more controls:

  • Recompute labels rebuilds every label. The tab header shows the progress.

  • Switch on and Switch off control the batch exclusion. See Start a Batch.

Use Recompute labels after an update of the test agent. The recompute reads no TestRail case and starts no run.

After a Change of the Coverage Rules

A case file stores a coverage map. The coverage rules of the import day made this map. The rules can change later. The stored map can then show a gap that is not real.

The analysis checks each case file again with the current rules. This check runs in memory. The case file does not change. A QA-verified case stays frozen.

If the new check fails, the analysis keeps the stored result. The readiness detail then shows the flag coverage_relink_failed: <ExceptionClass>. The class name names the error.

A passed run can still show the gap probe.audit_incomplete. The audit of that run used the old rules. Run the case again on the updated test agent. The new run uses the current rules for its audit. The gap clears when the new run passes the audit.

A QA-verified case can run again. The test agent does not import the case again. The case file stays as it is.

The run takes the re-checked map when it loads the case file. The pre-flight and the audit of that run use this map. A later change of the case file does not change the map of that run.

The audit block of the run report shows which map the audit used. The key coverage_relinked is true when the audit used the re-checked map. The key coverage_rules_hash names the rules of that map. If the check fails, the key coverage_relink_failed names the reason. The audit then uses the stored map.

A plain load of the Test cases list does not judge a stored result again. After a release, an administrator clicks Recompute labels in the Readiness tab. The recompute applies the current rules to every case.

QA-Verified Cases

A QA-verified case is frozen. Nothing in the test agent rewrites the case without your confirmation. A change can make the QA verdict wrong.

Five pills show how a case is QA verified:

Pill Meaning

QA ✓

A QA account signed off the case and cited a passed run.

QA ✓ test agent

QA verified the case in the test agent. The QA-verified record names the run. The test agent did not audit that run. The record is therefore not a sign-off.

QA ✓ (run not cited)

QA verified the case in the test agent. The QA-verified record names no run.

unreadable QA record

The test agent cannot read the QA-verified record. The test agent keeps the case frozen to be safe.

QA ✓ case

Another file of the same case carries the sign-off. The whole case is frozen.

Point at a pill to see who verified the case and when. The Test cases tab shows the pill next to the readiness pill. The QA verified (any) option of the sign-off filter lists all kinds.

Sometimes the test agent cannot read the QA state. Then the Test cases and Readiness tabs show a banner. The banner says "The QA-verified state could not be read; QA marks may be missing". The Run QA-verified regression button stays disabled until the state is readable.

A QA-verified record is not a sign-off. A sign-off needs a run that the test agent audited.

To get a sign-off:

  1. Run the case again.

  2. Open the run report.

  3. Click Mark QA-verified.

Ask an administrator to record or clear a QA-verified record.

What the Freeze Blocks

Action Result

Adaptation

QA holds the case. The adapter never rewrites it in TestRail.

Bulk re-import, import, sync, bulk delete, Find / replace… on the Title field and Empty trash

The action skips the frozen cases and names them. Example: "2 QA-verified cases skipped: C1, C2". It still handles the other cases. Find / replace… on other fields changes frozen cases too. Those fields hold no step text.

The editor Reimport button

The button skips a frozen case. It never overrides the freeze.

Single re-import in the TestRail dialog, regenerate, generate tree, upload, delete and the permanent purge of the trash

A dialog asks you to confirm. Only an administrator can confirm.

Create, duplicate and restore from the trash

A dialog asks you to confirm. Every editor account can confirm.

Editor Save and the restore of an earlier version

A dialog asks you to confirm. Every editor account can confirm.

Override the Freeze

A save on a frozen case makes the QA mark stale. A re-import replaces the case file QA verified.

  1. Start the action on the frozen case.

    The dialog QA verified case names the account and the time of the verification.

  2. Read the consequence in the dialog.

  3. Click Continue.

    The action runs.

If your account cannot confirm the action, the test agent shows "Only an admin can do this". Ask an administrator.

An administrator sees every override in the Activity tab. The entry names the account that confirmed the override.

An overridden delete keeps a QA-verified record. The case stays frozen. A re-import stays blocked until an administrator clears the record. A run-backed sign-off lives in the case file, so it goes to the trash with the file.

If the test agent cannot read the QA state, it shows an error and changes nothing. Try again later.

Unfreeze a Case

To remove a sign-off:

  1. Open the run report of the case.

  2. Click Remove next to the QA pill.

  3. Confirm in the dialog.

    The dialog has its own Remove button.

To clear a QA-verified record or an unreadable QA record, ask an administrator.

The case is no longer frozen after the record is gone.

Adaptation Tab

In the Adaptation tab, a frozen case shows the pill Held by QA and the reason. The row has no Release hold button. A bulk release names the cases that stay held by QA. A job result lists the skipped cases. If the test agent cannot read the QA state, a banner says "The QA-verified state could not be read; holds may be missing".

Regression Set

The frozen cases form the regression set of the stand.

  1. In the Test cases tab, select an env.

  2. Set the portal filter.

A batch of more than 10 cases asks you to confirm. A large set holds the env for hours. Run it at night.

  1. Click Run QA-verified regression.

    A batch named "QA-verified regression" with the date starts for the frozen cases under the filter. The readiness gate can leave cases out of the batch. See Start a Batch.

The button shows only for accounts with the right to run tests. It stays disabled until you select an env.

In the batch report, a failed or inconclusive frozen case shows the red tag REGRESSION. The tag means that QA verified the case and the run now fails. The batch summary counts the regressions.