Harbor QA-image cleanup (Cleanup-Images job)

QA builds push per-branch image tags to Harbor (bugfix_automationtests_0.0.1 .. n, feature_x_0.0.1 .. _n). Nothing ever removed them — the registry grew unbounded, and a plain age-based retention would kill the builds of long-living feature* branches mid-testing.

cleanupImagesStep deletes a branch’s tags when the branch has finished its life, signalled by JIRA: a merged branch does NOT mean its builds are obsolete — closing the last bug/task on the branch does.

Guard 1 — the branch allowlist (protects release tags)

QA branch builds and release/GA images are pushed to the same Harbor repositories (api/northbound-api holds both). There is no separate QA project, so the tag prefix is the only boundary between a throwaway build and a shipped image — and the JIRA guard below cannot see that difference: it answers "is the work finished", not "is this branch disposable".

Therefore the step refuses, before any API call, every branch outside QAConfig.Registry.CLEANUP_BRANCH_PREFIXES (default bugfix/, feature/). main, master, develop, release/ and hotfix/ are never touched, whatever JIRA sends. It is an allowlist, not a denylist: an unfamiliar branch shape is refused rather than guessed.

Override with the CLEANUP_BRANCH_PREFIXES env var (comma-separated, each prefix must keep its / boundary — bugfix without it would also match bugfixed/…​).

Guard 2 — "the last closer cleans" (stateless)

Tags of a branch are deleted iff no JIRA issue living on that branch is still open.

  • An early closer sees an open neighbour and skips.

  • The last closer sees everyone Done and cleans.

  • Nothing is stored anywhere, so a reopened issue never leaves stale state behind — the decision is recomputed from the sources of truth on every event.

JIRA cannot invert branch → issues natively. The issue set of a branch is derived as:

  1. Commit subjects of the branch → issue keys, from GitHub compare default_branch…​branch.

  2. Each key is expanded with its parent and subtasks (BFS closure) — this covers QA subtasks that never made a commit but still test the branch’s builds.

  3. Every key’s statusCategory is checked live in JIRA.

Merged and deleted branches — why the pull request is the second source

Step 1 returns nothing in the situation this job runs in most often: the issue is closed after the branch was merged, so the branch has no commits of its own (a deleted branch 404s outright). Treating that as "no other issues exist" would silently invert the invariant into the first closer cleans — the neighbours would never be seen.

So an empty or missing compare falls back to the commits of the branch’s pull request, which outlives both the merge and the deletion of its head ref. No compare commits and no pull request → the issue set is undecidable → tags are kept and the build says so.

Failing safe

Every undecidable state keeps tags: unreadable GitHub repo, failed compare, a branch whose history cannot be recovered, unreadable JIRA issue, unmapped Harbor repo, a cleanup account that can browse no JIRA project. Deleting is idempotent — a re-run or a race just finds zero tags.

Stray key-shaped tokens in commit subjects are real: UTF-8 matches the issue-key pattern. They are filtered by project key, not by a bare 404 — JIRA answers 404 both for "no such issue" and for "you may not see it", so a 404 inside a project the account CAN see counts as open. This is why the account needs Browse on every project that appears in commit subjects: without it, a missing permission would quietly turn open issues into non-blockers.

Deletion is per tag, never per digest (sibling tags share a digest). Deleting a tag frees no disk space by itself — the scheduled Harbor GC with "delete untagged" does.

Job

Cleanup-Images — a pipeline job calling the step. The agent line is not decoration: the step shells out (curl) and archives an artifact, both of which need a workspace, so a bare cleanupImagesStep() with no node would die on the first request with MissingContextVariableException.

@Library('qa-shared') _
pipeline {
    agent { label 'built-in' }
    options { ansiColor('xterm'); disableConcurrentBuilds() }
    stages {
        stage('Cleanup Images') {
            steps { script { cleanupImagesStep() } }
        }
    }
}

Parameters (created on first run, standard first-run pattern):

Parameter Smart value sent by JIRA Example

REPO

{{branch.repository.name}}

northbound-api

BRANCH

{{branch.name}}

bugfix/automationtests

ISSUE_KEY

{{issue.key}}

DEV-2612

APPLY

literal true

false = dry-run (rollout mode)

AUDIT_DELETE

never sent by JIRA

audit mode only, see below

JIRA Automation rule (one global rule, all projects)

  1. Trigger: Issue transitioned → To: any status of category Done. All issue types (tasks and subtasks — the guard decides who is last).

  2. Condition (optional, saves no-op builds): {{issue.branches.size|0}} greater than 0. No former guard "all subtasks Done" is needed — the step’s guard subsumes it and also covers independent tasks sharing the branch.

  3. For each → smart value {{issue.branches}} → Send web request: POST https://jenkins.friendly-tech.com/job/<folder>/job/Cleanup-Images/buildWithParameters with Authorization: Basic <jenkins-user:api-token> and the query parameters from the table above.

If {{issue.branches}} does not resolve on this JIRA edition inside a transition trigger, fall back to sending only ISSUE_KEY and resolving branches in the step via GET <jira>/rest/dev-status/1.0/issue/detail?issueId=<id>&applicationType=github&dataType=branch (the API behind the dev panel). Verify with one curl before wiring the rule.

Audit mode — what is actually sitting in there

The event-driven path answers "may I delete the tags of this branch". It structurally cannot answer "how much is in there and what of it is safe to remove": when a branch turns out to be undecidable, the step returns before it ever asks Harbor for a tag list.

Run the same job with BRANCH and ISSUE_KEY both empty and only REPO filled in, and it audits every tag of that repository’s images instead.

The direction is branches → tags, never tags → branches: a tag cannot be parsed back into a branch name, because _ is ambiguous (a replaced / in feature/x, a real underscore in oneiot_backend). So the step lists every branch that exists plus the head ref of every pull request — that second half is what keeps merged-and-deleted branches out of the orphan bucket — turns each into a prefix, and asks which branches could own each tag.

A tag is <prefix><version>, and both halves of that shape are load-bearing:

  • The remainder after the prefix must look like a version. Otherwise bugfix/DEV-12 would claim bugfix_DEV-12_extra_0.0.7, which belongs to the different branch bugfix/DEV-12_extra and carries its own open issues. The same rule guards the event path.

  • Exactly one branch may claim a tag. Zero claimants means orphan — and note that the branch list is knowably incomplete (a branch deleted without a pull request, or one past the paging caps), so the true owner may simply be absent; without the version rule its tags would be inherited by a shorter namesake whose verdict says nothing about them. Two claimants is equally decisive in the other direction: feature/x/y and feature/x_y collapse onto one prefix, and a tie is evidence that ownership is unknowable. Resolving it by sort order would let a cleanable branch decide the fate of a protected one’s builds.

Every tag lands in exactly one bucket:

Bucket Meaning Who removes it

DELETABLE

every issue living on the branch is Done

this job

KEEP

the branch still has open issues

nobody yet

UNDECIDABLE

no commits of its own and no pull request, or no issue key in any commit

retention

RELEASE

a shipped artifact: v1.0.0-b0.0.12, 1.1.1.0.0.0, 6.5.2, release*, hotfix_*, latest, stable

nobody — ever

DEVELOPMENT

a moving pointer to work in progress: dev, main, snapshot

nobody — ever

PROTECTED

built from a branch outside CLEANUP_BRANCH_PREFIXES (main, release/*, …)

nobody — by design

ORPHANS

no branch and no pull request claims the tag

retention

RELEASE and DEVELOPMENT are a second, independent barrier, decided by the shape of the tag alone — no branch list, no JIRA answer, no operator setting can override them. The branch allowlist already excludes these tags, but only because it happens to be narrow: widen CLEANUP_BRANCH_PREFIXES once and shipped images would be exposed. Deletion is irreversible, so the tag gets its own veto. A QA build is always <branch>_<version> and therefore begins with a letter from a branch name; a tag starting with a digit (or v + digit) is a version in its own right. The two kinds are reported separately because "shipped" and "current development build" are not the same answer to an operator staring at a delete list.

The retention safety net below is not bound by any of this — it deletes by age and pattern. Its rules must exclude these shapes explicitly, or it will remove exactly what this job refuses to touch.

The full list is printed and archived as harbor-audit.txt, so a build from three weeks ago still tells you what it saw.

Deleting the whole DELETABLE bucket needs two switches: APPLY and AUDIT_DELETE. The second one exists because audit mode is selected by the absence of BRANCH/ISSUE_KEY, and a JIRA rule whose smart values stopped resolving (an anticipated state — see the NOTE above) would send exactly that: REPO plus APPLY=true. One switch would have turned a per-branch deletion into a repo-wide one with no other signal. AUDIT_DELETE is never sent by the rule, so only a human at the job form can start a mass deletion; the count of tags and branches is warned about first. Read the dry-run before setting it.

Note that an audit walks dozens of branches that share issues, so JIRA issue lookups and the project list are cached for the run; the guard itself is exactly the same code as the event path.

Configuration

All knobs live in QAConfig (env-var overridable, same pattern as everything else):

Knob Default Override

Registry.HARBOR_CLEANUP_CREDENTIAL_ID

harbor-cleanup-robot

HARBOR_CLEANUP_CREDENTIAL_ID

Registry.CLEANUP_BRANCH_PREFIXES

bugfix/,feature/

CLEANUP_BRANCH_PREFIXES

Registry.GIT_TO_HARBOR_REPOS

map in QAConfig.groovy

edit the map

Tracker.JIRA_URL

https://friendly-tech.atlassian.net

JIRA_URL

Tracker.JIRA_CREDENTIAL_ID

jira-cleanup

JIRA_CREDENTIAL_ID

Tracker.GITHUB_API_URL

https://api.github.com

GITHUB_API_URL

Tracker.GITHUB_ORG

Friendly-Technologies

GITHUB_ORG

Tracker.GIT_CREDENTIAL_ID

git-cred (existing)

GIT_CREDENTIAL_ID

GIT_TO_HARBOR_REPOS maps a git repository (as the JIRA dev panel names it, lowercase) to the Harbor repositories holding its QA builds — the git and Harbor names diverge (northbound-api → api/northbound-api, ft-device-network → ftacs/ft-device-network-service), so the map is explicit. An unmapped repo fails the build loudly instead of guessing; extend the map and re-run. One git repo may publish several images (provision-portal).

Credentials to create:

  • harbor-cleanup-robot — Harbor robot account, DELETE artifact + list, scoped to the repositories in GIT_TO_HARBOR_REPOS and nothing else. Do NOT reuse harbor-cred (the read-only tag puller). Since QA and release tags share those repositories, this robot can technically delete a GA tag — the branch allowlist is what stops it, so keep the scope narrow.

  • jira-cleanup — read-only JIRA account. This is JIRA Cloud, so the credential is the Atlassian account email as username and an API token (id.atlassian.com → Security → API tokens) as password — a login password is rejected by the REST API. It needs Browse on every project whose keys appear in commit subjects; a project it cannot see makes its issues indistinguishable from typos (see "Failing safe" above).

  • git-cred — already exists (GitHub user+token, used by syncNodesConfigStep). Needs read access to the repositories and their pull requests.

Rollout

  1. Sync the shared lib (syncSharedLibStep), create the job, run once manually — first run creates the parameters.

  2. Run manually on a really-closed issue with APPLY=false — read the dry-run report.

  3. Wire the JIRA Automation rule with APPLY=false for a few days; check build descriptions.

    Every build ends with a one-line description, so the whole rollout is readable from the job’s build history without opening logs:

    • dry-run: N tags of <branch> — the guards passed, N tags would be deleted. This is the line to audit: is <branch> really disposable, is N plausible?

    • skipped: '<branch>' is not a QA branch — the allowlist refused it. Seeing main or release/* here is expected and healthy; it means JIRA does hand out those branches and the allowlist is load-bearing.

    • skipped: N open issue(s) on <branch> — the "last closer" guard held.

    • skipped: issue set of <branch> undecidable — merged/deleted branch with no pull request. Frequent occurrences mean the fallback cannot see PRs; check `git-cred’s scope.

    • no tags matched <prefix>* — nothing was built from that branch, or the prefix convention differs from the assumption.

  4. Flip the rule to APPLY=true.

  5. Make sure Harbor GC runs on schedule with delete untagged, and configure the retention safety net below.

Safety net — Harbor Tag Retention (mandatory companion)

The event-driven job has blind spots that retention (declarative, self-healing) covers:

  • a branch whose commits never mention an issue key — no dev-panel link, no event;

  • a lost event (Jenkins down, webhook failed, Automation throttled).

Configure per QA project: retain release_ and pinned tags, delete bugfix_ / feature_** older than ~120 days. The wide window is what makes long-living feature branches safe; the event-driven job provides the precise, immediate cleanup.