SDD: Version Control Service (Release Manifest) — ft-internal-tools

  • Status: DRAFT

  • Task grade: architecture-led

  • Ticket: DEV-2755 — CI/CD manager > Finalize sprint > Implement version control service

  • Deadline: 12 August 2026

  • Long-form companion: Release Manifest — design reference

Baseline: main, as it stands today. The feature stores its data in JSON files through the existing storage service, the same way release history, QA history and release-build history already do. It takes no dependency on the Postgres work that is under way elsewhere, because that work is not close to release and this feature has a date.

Section 5 defines the storage so the later move to Postgres is additive: the stored form is already a single self-contained document per manifest, which is exactly the shape a jsonb column takes.

No shared-library transport work is needed either: ft-github-client.get_file_content already reads any repository file at any ref (client.py:419 calls repos/{repo}/contents/{path}?ref=), which is exactly what the guide reader needs.

0. Requirements and assumptions

Elicited answers

  • Where the service lives → a module inside CI/CD Manager, not a fourth application.

  • How the deployment checklist is filled → generated draft from the deployment guides on the documentation portal and a manual editor, because the generated part cannot be trusted for every case. The release manager is the single author of the prose; per-component assignment to service teams is out.

  • Which repository the deployment facts come from → docs-portal, not ft-deployments. The latter describes a development cluster and carries generic versions, so it is not a source of truth and is not read at all.

  • How support receives the page → a link in the finalize-sprint e-mail; the page itself published on the documentation portal, with the authenticated UI reserved for the release manager.

  • Scope against the deadline → keep all phases in, reduce depth per phase, and state which items may not make the date.

Assumptions taken

  • The feature is built on main and stores manifests as JSON, deliberately not waiting for the Postgres work — see the baseline note above and the migration path in section 5.

  • Manifest volume stays in the order of tens per year. At roughly 50 KB per snapshot that is about 1 MB after a year in one file; if it grows past that, section 5 names the split.

  • The manifest reproduces the prototype release-manifest-v11.html and the reference guide v1.0 plus its Docker Pull addendum. Two documented deviations are listed in section 10.

  • Rendering is a port of the working prototype, but not a free one: the prototype renders from module-level COMPONENTS / LAYERS / DEPS constants while a manifest carries its own catalogue, so every render function is rewired to read from the snapshot. Measured against the prototype — 484 lines of CSS, 750 of rendering, 230 of Upgrade Path — that is 2.5 to 3 days, not 2.

  • The documentation portal is an acceptable reading surface for support engineers, i.e. no per-customer access control is required. Confirmed with the customer names in place: the portal is an internal support resource and the manifest is an internal feature, so bug items render the customer name on the published page as well.

  • The deployment guides stay alive and hand-maintained after this ships. The manifest indexes them; it does not replace them. If they stop being updated the generated part of the checklist goes quiet, which is why an unchanged guide is reported explicitly rather than as "no changes".

1. Problem and goal

The Release Manifest replaces the legacy VersionReleaseControl spreadsheet and the deployment guides on the shared disk: one page that tells a support engineer which version and build of every component is current and compatible, what changed this sprint, and what a customer on an older version must apply. Today it exists only as a prototype whose data arrays are hand-edited. Goal: the manifest is produced automatically when a sprint is finalized, curated by the release manager, and published where support can read it.

2. Scope

In

  • A manifest snapshot per sprint release and per hot-fix build, generated from the release record, the Jira items and the deployment guides.

  • An admin-only Version Control tab in CI/CD Manager for reviewing the draft and editing the deployment checklist.

  • A static HTML document PER RELEASE published to the documentation portal, written once and never rewritten, beside a regenerated index, Upgrade Path view and the shared renderer.

  • Both views from the guide: Releases and Upgrade Path.

Out

  • Editing releases themselves. The manifest is a read model; corrections happen upstream.

  • Per-customer access control or customer-facing distribution.

  • Kubernetes deployments. The checklist is Docker Compose only.

3. Affected components

CI/CD Manager backend — new routes/manifest/ (_crud.py, _publish.py) and services/manifest/ (storage.py, catalog.py, solution.py, builder.py, guide_reader.py, guide_diff.py, renderer.py) plus scripts/build_baseline.py. No schema change: storage.py is a storage service in the shape of services/release_storage.py.

Touched — routes/workflow/release/_changelog.py (persist the resolved Jira items), routes/workflow/release_build.py (hot-fix snapshot), routes/workflow/email.py (CTA button), frontend/js/sprint-release.js (one call after the docs-portal phase).

CI/CD Manager frontend — new manifest-render.js, manifest-upgrade.js, manifest-admin.js, manifest-editor.js, css/manifest.css.

Shared libraries — ft-jira-client gains an additive customers field alongside the existing hasCustomer flag, resolved through the Assets API (see deviation 2 in section 10). ft-github-client and ft-harbor-client are used as they are; neither changes.

Documentation portal — release-process.adoc and sprint-end-checklist.adoc cover the release and sprint-end procedure this feature changes, so they are updated in the same change. The published artifact goes into the documentation portal’s own repository, docs-portal/modules/ (decided 2026-08-17; originally this repository’s docs/modules/), which the portal already consumes as the ft-internal-tools component (antora-playbook.yml); nothing is committed to the docs-portal repository.

4. Design and approach

A manifest is an immutable snapshot written when a release is finalized. It embeds the component catalogue, the dependency edges, the versions, the Jira items and the checklist as they were at that moment, so re-reading an old release is unaffected by later renames. The catalogue is composed rather than re-entered: FT components come from data/projects.json (adding a layer attribute), third-party components from a static map in catalog.py, refined by the pinned tags in the deployment guide where the guide states a version. latest is never rendered as a version, and MySQL — built from a Dockerfile in every source — is always a static entry. Emulators and Superset are out of the catalogue. The Solution version is derived from the FTACS major.minor version shipped in the release; every other component keeps its own version line, which is why the version table is deliberately mixed-version.

The deployment checklist is a generated draft with a human override: a semantic diff of the deployment guides produces the mechanical part, the release manager corrects it, and regeneration never overwrites an edited field. The parsed structure is diffed, never the prose, so a rewritten paragraph produces no rows. Diffs are computed between manifest snapshots — each manifest stores the guide commit sha and the parsed fingerprint — so nothing depends on the documentation portal being versioned, which it is not. The diff answers "what changed in the deployment guide since the previous manifest", not "what changed in this release"; the checklist is captioned accordingly, and the commits between the two shas are stored with the snapshot so the release manager can see what produced each row.

One renderer serves both consumers: it is a pure function from the snapshot bundle to DOM, fed from the API inside the application and inlined into the published artifact.

Alternatives considered

  • A standalone version-control-manager application — rejected: every input already lives in CI/CD Manager, so it would fetch all of its data back over HTTP and pay for its own auth, database, chart and CI first.

  • A module in Deployment Manager — rejected: it is a separate application on its own database, it owns service contracts but none of the release, version or Jira data, and it deliberately dropped its platform_version concept in migration 20260520_0003.

5. Data model and DB changes

No DB changes. Manifests live in data/release_manifests.json, written through the same storage service as the rest of the application, with services/manifest/storage.py in the shape of services/release_storage.py — a read-modify-write under a module-level lock so concurrent draft edits cannot clobber each other.

{ "manifests": [
  { "id": "sprint-7",              // stable, also the DOM anchor and deep-link fragment
    "type": "sprint",              // 'sprint' | 'hotfix' | 'baseline'
    "releaseId": "4390807f-...",   // sprint manifests: the release record
    "buildRunId": null,            // hot-fix manifests: the release-build run
    "solutionId": "sol-7",
    "solutionVersion": "7.0",
    "targetBranch": null,          // hot-fix only, e.g. '7.0.x'
    "name": "Sprint 7",
    "date": "2026-06-18",
    "status": "draft",             // 'draft' | 'published'
    "publishedAt": null,
    "publishedBy": null,
    "createdAt": "...", "updatedAt": "...",
    "snapshot": { }                // the self-contained document, see below
  }
] }

The snapshot document carries components, layers, solutions, deps, items, versions, checklist, guide and source; the full shape is in the design reference. Everything a published manifest renders from is inside it — the record above is only the index the sidebar and the publish flow need.

JsonStorage.write truncates and rewrites in place, and JsonStorage.read swallows a parse error and returns the default. Together that turns an interrupted write into a silently empty history, and the next write makes the loss permanent. Manifest storage writes to a temporary file in the same directory and os.replace-s it into position, and treats an unreadable file as an error rather than as an empty list. This is release history — losing it quietly is the one failure mode that is not recoverable.

Two additive changes elsewhere: an optional layer attribute on each project, and the resolved Jira items persisted on the release record. They used to be computed during changelog generation and discarded; preview-changelog now writes them onto the project entry as jiraItems — key, url, the verbatim summary, type and customers — for the refs Jira confirmed. The split into Tasks/Features and Bug fixes is made from type when the manifest is assembled, not here.

The 7.1.1 baseline

The first manifest is generated for release 7.1.2 and has to show the delta against 7.1.1, which never existed as a manifest. There is no backfill and no importer: one seed record of type baseline is written instead, carrying versions and the guide fingerprint but no items and no checklist. Versions are assembled from Harbor — for each project with a harborProject, the newest tag matching the release pattern v{version}-b{build}, ordered by version. Neither the newest tag by push time nor a fixed 7.1.* filter works: the newest artifact is usually a branch build (feature-DEV-1709, bugfix-dev-2741, dev, a bare sha), and several components are not on the 7.x line at all — Northbound API and Device Network are on 1.0.2, AI Agent on 1.0.4, Provision Portal on 4.0.0. Two tag formats exist in the registry (v1.0.2-b0.0.1 and the legacy 5.0.1_0.0.6), and a tag may carry a free-text suffix (v7.1.1-b0.0.2-fix415), so the parser tolerates both and the result stays editable.

The baseline is frozen when it is written, not computed at 7.1.2 finalize time. Between now and that release there are weeks of pushes; resolving it lazily would place them before the baseline and report changed components as stable.

python -m scripts.build_baseline --name <version> --date <YYYY-MM-DD> is the one way to write it. It is deliberately not idempotent: an existing baseline is refused and left alone, before Harbor is read. Re-running it a month later would otherwise resolve the tags of that day into a record whose entire purpose is to state the tags of this one. It is a script and not an endpoint precisely because of that — it runs once per installation, has no UI, and an operation that must never be repeated does not belong behind a URL anyone can POST to twice. It reports unversioned — the components Harbor could not answer for — because the record is written once and completed by hand afterwards. The deployment-guide fingerprint is frozen with it: without one the first real manifest has nothing to diff and its checklist comes out empty.

Migration path to Postgres

The storage choice is deliberately the only thing that has to change when the database work lands. The record above maps one-to-one onto a table whose snapshot is a jsonb column:

CREATE TABLE cicd.release_manifests (
    id               text        PRIMARY KEY,
    release_id       uuid        NULL,   -- no FK, per the DB contract
    build_run_id     uuid        NULL,
    type             text        NOT NULL,
    solution_id      text        NOT NULL,
    solution_version text        NOT NULL,
    target_branch    text        NULL,
    name             text        NOT NULL,
    release_date     date        NOT NULL,
    status           text        NOT NULL,
    snapshot         jsonb       NOT NULL,
    published_at     timestamptz NULL,
    published_by     text        NULL,
    created_at       timestamptz NOT NULL,
    updated_at       timestamptz NOT NULL
);
CREATE UNIQUE INDEX release_manifests_release_id_idx
    ON cicd.release_manifests (release_id) WHERE release_id IS NOT NULL;
CREATE UNIQUE INDEX release_manifests_build_run_id_idx
    ON cicd.release_manifests (build_run_id) WHERE build_run_id IS NOT NULL;

Three rules keep that migration additive, and they constrain the code written now:

  • Callers never touch the file. Everything goes through services/manifest/storage.py; no route, builder or renderer reads data/ directly. The migration then replaces one module’s internals, following the dual-read/write facade already used for release history.

  • Field names are the column names. The JSON record uses the names above so the mapping is mechanical, and snapshot stays one opaque document rather than being spread across columns.

  • No JSON-only affordances. Nothing relies on ordering within the file, on rewriting unrelated records in the same write, or on reading the whole list to answer a single-id lookup.

When the move happens, snapshot becomes a JSONB column, and SQLAlchemy does not track in-place mutation of JSON column contents — a nested list appended to a shallow copy compares equal and the UPDATE is skipped. The repo layer must deep-copy before mutating, the same trap already handled in the release and cleanup-schedule repos.

6. FE-BE contract changes

New admin-only routes under /api/manifest, all returning the app’s existing JSON envelope.

GET  /api/manifest/releases                     -> sidebar list
GET  /api/manifest/<id>                         -> one snapshot
GET  /api/manifest/bundle                       -> all published snapshots (renderer input)
POST /api/manifest/draft                        -> build or rebuild a draft
PUT  /api/manifest/<id>                         -> release-level edits
PUT  /api/manifest/<id>/checklist/<compId>      -> one checklist entry, marks it edited
POST /api/manifest/<id>/publish                 -> render, commit to the portal, publish
GET  /api/manifest/export/<name>                -> one file of a publication, uncommitted

No change to any existing endpoint.

7. Protocol

N/A — no TR-069 / TR-181 / TR-369 / LwM2M parameter paths are touched.

8. Logging and observability

Draft generation and publication log against the release id at INFO with releaseId=<id> manifestId=<id> action=<draft|publish> result=<ok|failed>; a generation failure logs at WARNING and is recorded as a release-level log entry, visible in the release history, because generation must never fail the finalize flow. The guide reader logs the repository, path and sha it resolved, so a stale read can be traced; a guide that has not changed since the previous manifest is logged as guideUnchanged sha=<…​> rather than producing a silently empty checklist.

9. Test plan

pytest — test_manifest_catalog.py (projects and the guide compose; the database role expands to mysql and oracle; an unknown alias is reported, not dropped); test_manifest_solution.py (FTACS 7.0.0 gives sol-7 / 7.0; a release without FTACS falls back instead of crashing); test_manifest_builder.py (status new / updated / stable / hotfix / breaking; unreleased components inherit from the previous snapshot; the pull command echoes the recorded imageUrl verbatim — the regression guard for deviation 1); test_manifest_guide_diff.py (variable added, removed, default changed; image tag changed; prose rewritten with no semantic delta gives no row; no change at all gives a quiet row); test_build_baseline.py (a branch tag, dev, latest and a bare sha are never taken as a version; a suffixed tag parses; the legacy underscore format parses; a project with no release tag is reported for manual entry, not silently skipped); test_manifest_editor_precedence.py (regeneration preserves every edited field); test_routes_manifest.py (auth on every route, draft and publish transitions); test_manifest_storage.py (round-trip of a full snapshot; concurrent draft edits serialise instead of clobbering; a write interrupted before os.replace leaves the previous file intact; an unreadable file raises instead of reporting an empty history).

vitest — manifest-upgrade.test.js (chronological order, the from-exclusive to-inclusive window, later supersedes earlier, no-change entries skipped, same-version and downgrade validation); manifest-render.test.js (pull cell, status badge class, diff row builder).

Playwright — draft, edit a checklist entry, publish; sidebar groups hot-fixes above sprints; a breaking entry shows its banner and the diff tabs switch; Upgrade Path builds a combined checklist; the pull command copies to the clipboard; a hot-fix release shows no version table.

A Playwright quality mark on the finalize flow must be green before sprint-release.js is touched.

10. Risks, rollout and follow-ups

Two deviations from the source documents. Implementing them verbatim would ship something wrong.

  1. The addendum specifies a Harbor tag built as <version>_<build>, e.g. 7.0.6_0.0.2. The pipeline actually pushes v<version>-b<build> — a real release record carries hub.friendly-tech.com/configs/ft-configs-service:v1.0.1-b0.0.1. The manifest therefore renders the recorded imageUrl verbatim and never recomputes a tag, so a support engineer cannot be handed a pull command the registry will reject.

  2. The guide shows a customer name on bug items. The Jira integration reduces the "Project (assets)" field to a boolean, discarding the identity. Resolved by measurement, not by a spike: the issue payload carries [{workspaceId, id, objectId}] with no label, so a second call is required — GET https://api.atlassian.com/jsm/assets/workspace/{workspaceId}/v1/object/{objectId} with the same credentials returns label. DEV bugs do carry the field, so it applies to sprint releases and not only to PROD tickets. The client caches objectId → label; a release touches a handful of distinct ids. The label is rendered verbatim, never cleaned up in code — it is a "Projects" object and may read SCG England (via Provu); corrections belong in Jira.

Schedule. The window from 31 July to 12 August is 9 working days against roughly 19 days of design, and every phase ships at reduced depth. But 12 August is the delivery date, not the operational one: the first real manifest is produced when release 7.1.2 is finalized, which is expected at the end of August or the beginning of September. Work that misses the 12th can therefore still be in place for the first manifest — what cannot slip is what has to exist at that finalize: storage with atomic writes, draft generation, and a frozen 7.1.1 baseline, because a release state that has passed cannot be reassembled afterwards.

Items drop in this order: the Upgrade Path view, then hot-fix manifests, then the generated part of the checklist (the manual editor stays). Not droppable: generation at finalize, publication, customer names — now cheap, see deviation 2 — and atomic writes. Backfill of previous releases is not planned at all and no importer is written; historic manifests, if ever needed, are entered by hand through the same editor.

Storage. JSON is chosen against the deadline, not on merit: the Postgres work is not close to release, and coupling this feature to it would hand the date to another project. The cost is a file rewritten in full on every save and a history that is one file rather than one row per manifest. Section 5 constrains the code so the later move is a change to one module; the risk is that those three rules erode under time pressure, at which point the migration stops being additive. The test_manifest_storage.py cases are what keep them honest.

Decay risk. Hot-fix manifests are the one deferral that loses data: release_build_history prunes after 10 days, so hot-fixes shipped while it is outstanding cannot be reconstructed and must be entered by hand.

External dependencies — closed before day 1 rather than waited on. The layer assignment comes from the prototype (databases, infrastructure, core, API, frontend) with Hazelcast moved into layer 1 to match the guide’s startup order; the third-party list is the seven entries the prototype carries and projects.json does not; emulators and Superset are excluded. The breaking-change prose is written by the release manager, with an AI assist that stays outside the application: a button copies the diff, the release items and the relevant guide section to the clipboard, and the answer is pasted back into the editor. No LLM integration is added — settings.json has no provider and this is the one place in the document where a confident hallucination would be executed against a customer’s production.

11. Lens exemptions

  • Protocol — no device-facing parameter paths are touched.

  • Performance — generation runs once per release on an admin path; the published artifact is a static file.

  • Migration and back-compat — a new data file and two additive fields; no schema change and no existing reader changes.