Database

ftfcc Schema

ftfcc is this service’s own schema: read-write, owned and migrated exclusively by this service’s Liquibase (db.changelog/ftfcc/db.changelog-master.yaml). M1 ships six tables, M2 adds four more — fcc_test_setting_latency, fcc_device_state, fcc_audit, fcc_result_dead_letter — plus two indexes on the existing fcc_activation, and M3 adds two more again: [fcc-model-capability] and [fcc-domain-setting], plus one column on the existing fcc_test_device. That is twelve tables, and M4 adds the thirteenth: [fcc-legacy-import], the cursor of the legacy ETL.

Every M2 addition is a separate changelog file appended to the master under an explicit # ===== M2 ===== marker, and the two fcc_activation indexes sit in changelog_add_fcc_activation_indexes_m2.xml rather than in the table changelog: that file has already run at every installation and its changesets must keep their checksums. M3 follows the identical convention under a # ===== M3 ===== marker, and fcc_test_device.protocol is added by changelog_add_fcc_test_device_protocol.xml for exactly that reason — an executed changeset is never edited, because its checksum is what tells a live installation that nothing has moved under it.

Schema-Wide Rules

These rules are visible in every changelog and are not incidental:

Rule Why

Zero foreign keys

Integrity is app-enforced (FT house convention). A no-FK schema also makes the M4 legacy ETL and any future re-partitioning simpler: nothing to drop and recreate around a bulk load.

VARCHAR(32) with the enum constant name, never TINYINT/ordinal

An ordinal breaks silently when a value is inserted in the middle of the Java enum (existing rows now mean something else) and is unreadable in a WHERE state = ? written by hand or in an ad-hoc SQL query during an incident.

BIGINT for byte counters (total_bytes, test_bytes, file_size_ul)

The legacy schema used INT, which overflows: a gigabit connection running a 30-second upload test moves roughly 3.75 GB, well past the signed INT range.

Audit columns (created, creator, updated, updator) on every table

Filled by Spring Data JPA auditing (FccAuditableEntity, JpaAuditingConfig). The auditor is X-FT-User-Id as a string during an HTTP request, or the literal system for scheduler/outbox/watchdog writes — see Component View. This is what makes "who did this" answerable in a regulatory export.

Paired <property dbms="mysql|oracle"> types

Every table changelog is written only in terms of the ${number}, ${big_number}, ${tiny_number}, ${date}, ${clob.type} and ${now} properties declared once in db.changelog-properties.xml. A raw MySQL type in a table changelog would break Oracle silently at customer-install time, which is exactly the failure this file exists to prevent.

Deliberate, named indexes

Every index in the schema exists to serve one named query (a poller, the watchdog, a specific screen). The comment above each createIndex changeset in the source states which query it is for — see the table below.

fcc_test

The user-authored campaign. It is not a run: every concrete execution is a row in fcc_activation, and a test with reactivation_days > 0 produces many of them over time.

Column Type Meaning

id

BIGINT PK

domain_id

INT NOT NULL

The ftacs domain (cpe.location_id) the test belongs to. Every read is filtered by it via the identity filter — see Identity Contract.

name

VARCHAR(255) NOT NULL

legacy_result_kind

VARCHAR(32)

SPEED | LATENCY, and NULL for every campaign this service created. It says WHICH HALF OF A LEGACY update_group an imported anchor is, and nothing else: the ETL makes one archived campaign per (update_group, result kind), and fcc_test_legacy_ug_id_result_kind_unique_idx is what makes that idempotent across passes — see Legacy Import.

IT REPLACED type, WHICH WAS THE CAMPAIGN’S DIAGNOSTICS TYPE AND IS GONE. A campaign now holds several configurations, of possibly different kinds, so there is no single type for it to carry; the type lives on fcc_test_config, which is the level that actually has one. The index needed a discriminator on this table and type was the only column that could serve, which is why the column was renamed to what it actually meant rather than kept: a native single-configuration campaign and a legacy anchor carried indistinguishable values in type, and they do not here.

state

VARCHAR(32) NOT NULL

NOT_ACTIVE | SCHEDULED | RUNNING | PAUSED | STOPPED | ERROR | SCHEDULED_FOR_REACTIVATION | COMPLETED | ARCHIVED. ARCHIVED is unreachable through the ordinary state machine; only the M4 legacy ETL sets it, on the anchor row it creates for an imported legacy test.

source

VARCHAR(32) NOT NULL, default ACS_STREAM

ACS_STREAM | LEGACY. LEGACY marks a test imported by the M4 ETL: it has no outbox history and can never be reactivated.

legacy_ug_id

INT

Legacy update_group id; populated only for source=LEGACY.

scheduled_at

DATETIME(3)

reactivation_days

INT NOT NULL, default 0

A COUNT OF RUNS, and never a duration — 0 = single run, N = N runs after the first. The name was wrong on day one. On the wire it is reactivation.reactivationCount (Reactivation Is a Rule); the column kept its name because renaming it would move a migration, the state machine, the planner and the legacy ETL to change a string no caller sees — the same mapping url_dl / downloadUrl already has.

reactivation_repeat_every

INT, default 1

Days between two runs — reactivation.repeatEvery. The default is the 24-hour constant FccReactivationService used to hard-code, so every campaign stored before this column keeps its cadence without a back-fill. Nullable, and it was NOT NULL for one commit of this branch — which broke every creator of a campaign that did not know about it. The entity field has no initializer, Hibernate puts every mapped column into the INSERT, and an explicit NULL does not let a DEFAULT apply, so FccLegacyCampaignHandler.createAnchor failed outright and took the whole legacy import with it. Both readers already answer 1 for a null on purpose — FccReactivationService.repeatEveryOf because a planner cannot refuse, and FccTestMapper.toReactivation because a form cannot render a null cadence — so the constraint protected nothing it was not already protected against. The changeset was edited in place rather than followed by a drop-constraint one, which is allowed only because nothing is released; see the comment in changelog_add_fcc_configs.xml. A development database that migrated between the two versions keeps the NOT NULL — the changeset’s columnExists precondition marks it ran and changes nothing — and needs one ALTER by hand: the troubleshooting row carries the statement for both vendors.

reactivation_starts_on

DATETIME(3)

The ANCHOR the slots are counted from — reactivation.startsOn. Run n opens at starts_on + n * repeat_every days. Null means the planner measures from the run that just closed, as it always has. It is not a second schedule: when the campaign first runs is scheduled_at, and the two are separate columns so that a recurrence rule cannot silently move a booking.

command_seq

BIGINT NOT NULL, default 0

Monotonic per-test command counter, incremented in the same transaction as the fcc_acs_outbox row it accompanies. The ACS rejects a command whose commandSeq is not strictly greater than the last one it accepted for that test (REJECTED(STALE_SEQ)), which is what stops a replayed or reordered command from restarting a finished test.

activation_count

INT NOT NULL, default 0

Number of activations created so far. > 0 is exactly the condition that freezes fcc_test_device — see [fleet-immutability].

version

BIGINT NOT NULL, default 0

Backs the JPA @Version optimistic lock. Needed because a concurrent user edit and a scheduler-driven activation of the same test must not silently overwrite one another.

Indexes: fcc_test(domain_id, state) serves POST /FCC/items, which is always narrowed to the caller’s domain and in practice almost always also to a state ("show me what is running"). fcc_test(state, scheduled_at) serves the SCHEDULED / SCHEDULED_FOR_REACTIVATION poller, which scans across all domains every poll cycle and must not scan the whole table.

fcc_test_config

ONE CONFIGURATION OF A CAMPAIGN: which diagnostic to run, on which devices, with which timing and retry knobs. A campaign holds SEVERAL of these — one per element of configs[] on PUT /FCC/item/details — and each one has its own settings row, its own frozen fleet and its own ACS activation on every run.

It is Group Update’s update_group_child, and that correspondence is the whole point of the level:

Group Update Here What it holds

update_group

fcc_test

The campaign: name, domain, state, schedule, recurrence rule

update_group_child

fcc_test_config

The CONFIGURATION: what to measure, on which devices, with which knobs

ug_task / ug_fcc

fcc_test_setting_speed / fcc_test_setting_latency

The per-configuration parameters

the child’s device selection

fcc_test_device

The per-configuration frozen fleet

MIXED KINDS IN ONE CAMPAIGN ARE ALLOWED — a speed configuration beside a latency one, exactly as one legacy update_group carries a 103 task and a 104 task. That is what removed fcc_test.type: a campaign no longer HAS one type, and a column answering the question anyway would have to pick a configuration and call its type the campaign’s.

The timing columns themselves map to FccScheduleDto + FccRetryPolicyDto in the command, and are shared by both kinds: a latency configuration has the same window, the same iteration length and the same WAN sampling policy as a speed one. The contract states them ONCE per campaign, in activation, so the save writes them identically to every configuration row of a campaign and the details read answers them from the first.

Column Type Meaning

test_id

BIGINT NOT NULL

Plain id column, no FK (see Schema-Wide Rules). The campaign this configuration belongs to.

seq

INT NOT NULL, default 1

The 1-based POSITION in configs[], and the configuration’s IDENTITY within its campaign. It is not a cosmetic ordering: the array a client PUTs is the array it GETs back, an element sent without an id means "the nth configuration", and a save that re-wrote the set in a different order would move a fleet of twenty thousand devices from one configuration to another with nothing in the answer saying so.

type

VARCHAR(32) NOT NULL

SPEED | LATENCY — WHICH DIAGNOSTIC THIS CONFIGURATION RUNS, and therefore which settings table carries its parameters, which half of the fcc_activation snapshot is filled and which half is written null, and which targets block its ACTIVATE command carries. See fcc_test_setting_latency and The Settings Block Must Match the Type.

only_online

TINYINT(1)

Skip devices the ACS reports as offline instead of queueing the diagnostic.

push

TINYINT(1)

Use a connection request instead of waiting for the next Inform.

total_duration_h, single_duration_min, batch_interval_min

INT

FccScheduleDto. batch_interval_min = 0 means one single batch.

retry_failed, retry_wan

INT

FccRetryPolicyDto.

wan_sample_gap_sec, wan_retry_interval_sec, wan_step_timeout_sec, diag_timeout_sec

INT

Rest of FccRetryPolicyDto. wan_step_timeout_sec is the timeout waiting for the WAN GetParameterValues answer; diag_timeout_sec is the timeout waiting for the diagnostic result itself.

manufacturer, model

VARCHAR(255)

WHAT THE OPERATOR CHOSE, beside the fleet it resolved to — DEV-1381’s configs[] identity. Nullable: a campaign saved before these columns existed has no recorded choice, and a back-fill would be today’s facts written into yesterday’s snapshot.

source_type

VARCHAR(32)

FccConfigSourceTypeEnum constant NAME (All, Condition, Individual, File), never an ordinal — the convention of every enum-valued column here.

condition_id

INT

The id of a saved condition in the iotw schema — iotw_view.id, with no foreign key for the same reason as everywhere else here. Stored, not re-resolved: the condition is evaluated once at save time and the fleet it selected is in fcc_test_device, so this column is what lets the details read answer the condition’s name ([iotw-grant]) and a later edit show which condition the frozen fleet came from. A condition deleted in the portal afterwards leaves the id pointing at nothing, which the read answers as a null conditionName rather than as an error — see the subset this service translates.

fcc_test_device is the fleet; these four columns are the CHOICE that produced it. Until DEV-1381 #10 the selection was resolved and forgotten, so a saved campaign could not be re-opened showing "all devices of this model" — only the 4000 rows it came to. They are never re-resolved: the fleet stays a frozen snapshot (fcc_test_device), and these say what was asked for, which is what a details screen renders and what an export has to carry.

Index: fcc_test_config(test_id, seq) UNIQUE serves SELECT * FROM fcc_test_config WHERE test_id = ? ORDER BY seq (test detail, activation snapshot build) and enforces "one configuration per (campaign, position)" in the database rather than only in the service — a campaign cannot hold two configurations claiming the same slot in configs[]. It replaced fcc_test_config(test_id) UNIQUE, which said "one configuration per campaign" and was the schema half of a limit the service no longer has.

fcc_test_setting_speed

The SPEED parameters of ONE CONFIGURATION (FccSpeedTargetsDto, FccTiersDto, FccWanThresholdsDto) — one row per fcc_test_config whose type is SPEED. LATENCY has its own sibling table, fcc_test_setting_latency — nothing here is shared with it, which is why the two are not squeezed into one wide table.

It hangs off the CONFIGURATION and not off the campaign, which is what lets one campaign carry two speed configurations measuring different URLs or different tiers, and a speed configuration beside a latency one. The row’s owner is config_id and there is deliberately no campaign pointer beside it — see the config_id row of fcc_test_setting_latency for the reasoning, which is the same here.

Column Type Meaning

config_id

BIGINT NOT NULL

The fcc_test_config these parameters belong to; the unique index enforces one row per configuration.

time_based_duration_sec

INT

TR-143 TimeBasedTestDuration in seconds — how long the device transfers for. Read only when test_mode is TIME_BASED, and the save REFUSES it on a FILE_BASED configuration rather than storing a value nothing would apply (TIME_BASED). Nullable and never back-filled: every configuration that existed before contract 1.6 was file-based.

time_based_interval_sec

INT

TR-143 TimeBasedTestMeasurementInterval in seconds — the STEP of the IncrementalResult table the device returns. Null lets the device choose, and the ACS then omits the parameter from the SetParameterValues entirely. It may not exceed time_based_duration_sec: a step longer than the window measured yields a table with no rows.

threshold_dl, threshold_ul

INT

WAN cross-talk thresholds in kbps. Above these the WAN is considered busy and the iteration is recorded as CROSS_TALK instead of a speed measurement — see The Pass/Fail Formula.

tier_dl, tier_ul

INT

The subscriber’s provisioned tier in kbps; speed_success_percent is measured against it.

url_dl, url_ul

VARCHAR(512)

Two columns, not one: DownloadDiagnostics.DownloadURL and UploadDiagnostics.UploadURL are independent parameters of two different TR-181 objects and routinely point at different hosts. The ACS appends the device serial to url_ul.

connections_dl, connections_ul

INT

Two columns for the same reason as the URLs: DownloadDiagnostics.NumberOfConnections and UploadDiagnostics.NumberOfConnections are separate parameters, and devices advertise different DownloadDiagnosticMaxConnections / UploadDiagnosticsMaxConnections limits — one shared column would be silently clamped on half the fleet.

file_size_ul

BIGINT

UploadDiagnostics.TestFileLength, in bytes, upload only (the download direction has no counterpart parameter; the file size is whatever the server serves). BIGINT for the same reason as total_bytes.

test_mode

VARCHAR(32), default FILE_BASED

FILE_BASED | TIME_BASED. v1 accepts FILE_BASED only — the validator rejects TIME_BASED — but the column and enum constant exist now so the future TR-143 TimeBasedTestDuration work is a code change, not a migration on a live schema.

wan_sample_mode

VARCHAR(32), default TWO_SESSIONS

TWO_SESSIONS | PREVIOUS_ITERATION; v1 is TWO_SESSIONS only.

speed_tier_mode

VARCHAR(16)

Where the tier came from — the constant name of FccSpeedTierModeEnum (Default, Manual); the wire values are lower case. Provenance and not a rule: the tier value is what the ACS is given either way, and a tier is never checked against the preset list. Nullable and never back-filled — a campaign saved before the column records no provenance rather than claiming a preset nobody can name. See speedTierMode Is Provenance, Not a Rule.

speed_success_percent

INT

Percentage of the tier that still counts as a pass (default 80, see Configuration Reference).

hubb_id

VARCHAR(64)

Operator-side correlation id, carried through to the result rows.

Index: fcc_test_setting_speed(test_id) UNIQUE, same reasoning as fcc_test_config.

fcc_test_setting_latency

The LATENCY-specific half of a test, the exact sibling of fcc_test_setting_speed. Two narrow tables and not one wide one, for the reason the speed table already gives: nothing here is shared with it. A LATENCY test has no URL, no connection count and no upload file size; a SPEED test has no ping host, no repetition count and no per-packet timeout. Merged into one table every row would be half NULL, and the "which columns must be present" rule would live only in Java.

Four columns map one to one onto the TR-181 IPPing object the ACS drives:

Device.IP.Diagnostics.IPPing.Host
Device.IP.Diagnostics.IPPing.NumberOfRepetitions
Device.IP.Diagnostics.IPPing.Timeout
Device.IP.Diagnostics.IPPing.DataBlockSize
Column Type Meaning

config_id

BIGINT NOT NULL

Plain id column, no FK. The fcc_test_config these parameters belong to; the unique index enforces one row per configuration. There is no campaign pointer on the row on purpose: a settings row is only ever reached through its configuration, so a second pointer at the campaign would be an unread column that could disagree with the one that is read.

threshold_dl

INT

WAN cross-talk threshold in kbps: above it the WAN is busy and the iteration is recorded as CROSS_TALK instead of a latency measurement. Download only — a ping has no direction of its own to sample, and the contract’s FccWanThresholdsDto.ulKbps is explicitly null for a latency test.

tier_dl

INT

The subscriber’s provisioned downstream tier in kbps, carried through to the result rows so a report can state what the latency was measured against. Same single-direction reasoning as threshold_dl: FccTiersDto.ulKbps is null for latency.

host

VARCHAR(255)

IPPing.Host, the ping target. One host, not a list: v1 pings a single target. The one value with no default anywhere — the ACS cannot ping nothing and answers REJECTED(INVALID_CONFIG) without it, which is why the validator refuses a latency test that omits it (PUT /FCC/item/details).

repetitions

INT

IPPing.NumberOfRepetitions — what the diagnostic was configured with. SuccessCount + FailureCount adds back up to it — see [fcc-result-latency] for why packets_sent is reconstructed from those two counters rather than read from this column. Since contract 1.5, fcc_result_latency.repetitions carries the sibling figure for what the device actually ran with, reported per iteration on FccResultDto.pingRepetitions — this column is the campaign’s configured value, that one is the per-measurement actual; see [fcc-result-latency].

timeout_ms

INT

IPPing.Timeout, per packet, in milliseconds.

data_block_size

INT

IPPing.DataBlockSize, the payload of one echo request in bytes.

speed_tier_mode

VARCHAR(16)

Where the tier came from — the constant name of FccSpeedTierModeEnum (Default, Manual); the wire values are lower case. Provenance and not a rule: the tier value is what the ACS is given either way, and a tier is never checked against the preset list. Nullable and never back-filled — a campaign saved before the column records no provenance rather than claiming a preset nobody can name. See speedTierMode Is Provenance, Not a Rule.

latency_success_ms

INT

The average round trip that still counts as a pass. Not a TR-181 parameter — it is the pass/fail bound this service applies, because the ACS publishes raw measurements and never a verdict, so the criterion has to be stored next to the test that defined it rather than derived at read time from whatever the default happens to be then. Falls back to FccRuntimeDefaults.LATENCY_SUCCESS_MS (100) when the snapshot carries none.

hubb_id

VARCHAR(64)

Operator-side correlation id, carried through to the result rows.

threshold_dl and tier_dl keep the SPEED spelling on purpose: a latency run still samples the WAN for cross-talk and still reports against the subscriber’s provisioned tier, and using the same column names keeps the fcc_activation snapshot a straight copy.

Index: fcc_test_setting_latency(test_id) UNIQUE, same reasoning as fcc_test_config — it serves the test-detail read, the delete-by-test of an edit and the re-take of the activation snapshot at dispatch time, and enforces the one-row rule in the database rather than only in the service.

One type, one settings row. FccTestService.saveSettings writes the row the type calls for and deletes the one of the other kind in the same transaction. The delete is not housekeeping: a test whose type was changed from SPEED to LATENCY would otherwise keep its fcc_test_setting_speed row, and although FccActivationMapper.applySnapshot picks the half it needs by the type, the leftover row is still what POST /FCC/item/details reads back and what a later type change would revive — a test carrying two contradictory sets of parameters, of which only one was ever measured.

fcc_test_device

The frozen device set of a test — a snapshot, not a live view. serial, model_id, domain_id and — since M3 — protocol are resolved once from ftacs.cpe when the test is saved and are never refreshed: ftacs is read-only and may be a different database instance, so a join at read time is not available, and more importantly a regulatory report has to show the fleet the test actually ran against, not the fleet as it looks today.

How a selection becomes these rows is the last section of Device Resolution; this section is what the rows hold.

Column Type Meaning

test_id

BIGINT NOT NULL

The campaign the device was frozen for. It is kept beside config_id deliberately — see the index note below.

config_id

BIGINT NOT NULL

The fcc_test_config whose fleet this device is in, and the row’s real owner: the fleet belongs to the configuration, and one ACS activation carries one configuration’s devices and no others.

cpe_id

INT NOT NULL

ftacs.cpe.id — the id the ACS addresses the device by in FccDeviceDto.

serial

VARCHAR(64)

Required in practice, enforced by the validator rather than by a column constraint: FccDeviceDto.serial is mandatory in wire contract 1.3, so a device with no serial is refused with 400 when it is added to the fleet, and the whole stored set is checked again when the test is activated — the stored set is a snapshot, can predate the rule, and is frozen after the first activation. See A Device Without a Serial Is Refused — Twice.

model_id

INT

ftacs.cpe.product_class_id. This is what this product calls a model id, and it is the same value fcc_model_capability.model_id is keyed by and the catalog reports as FccCatalogModelItem.modelId — see The ACS Schema This Service Reads.

domain_id

INT

Copied from the cpe row, not from the test: a device may sit in a sub-domain of the test’s own domain.

protocol

VARCHAR(32)

M3, changelog_add_fcc_test_device_protocol.xml. The FccCpeProtocolEnum constant name the device carried when the fleet was frozen. See Why a Snapshot Carries the Protocol.

Index: fcc_test_device(config_id, cpe_id) UNIQUE does two jobs at once — it rejects a duplicate device in the same CONFIGURATION at insert time (a fleet is assembled from overlapping groups and up to ft.fcc.max-devices-per-test rows can be inserted in one batch, so de-duplicating only in memory is not enough), and it serves the read that builds the device list of every ACTIVATE command.

It replaced fcc_test_device(test_id, cpe_id) UNIQUE, and that is the visible consequence of allowing overlapping fleets. The old index was the statement "a device appears once in a campaign", which is no longer true: a device named by two configurations of one campaign is measured by both, and it then sits in two concurrent ACS activations — which fcc_sa_activation permits, being unique on (activation_id, cpe_id), and which is the operator’s decision to make rather than this schema’s. The replacement says what is still true: a device appears once in a CONFIGURATION. A campaign-level count therefore counts a shared device once per configuration, which is how many measurements of it the run actually performs.

fcc_test_device(test_id, cpe_id) is re-created NON-UNIQUE in the same changeset, because eight reads ask this table a question about the CAMPAIGN — how many devices a campaign would touch (the activation impact dialog), which campaigns hold a device of this model, which serials a report scope covers — and none of them is about configurations. test_id and config_id are written by the one statement that inserts the fleet, from the configuration they belong to, so the pair cannot drift.

Once fcc_test.activation_count > 0, the set is immutable: FccTestValidator.validateDeviceSetEditable throws FccConflictException (HTTP 409) if the submitted fleet differs from the stored one. The comparison is PER CONFIGURATION, and the SET OF CONFIGURATIONS is frozen with it. Each configuration is its own activation measuring its own fleet, so moving a device from the first configuration to the second changes what both of them measured without changing the campaign’s device list at all — which a campaign-wide comparison would have accepted; and adding or removing a configuration is the same thing in a different shape, because a run would then dispatch a number of activations the finished runs did not have. This is a product rule, not a technicality — see Endpoints Implemented in M1 for the exact response, and PUT /FCC/item/export + PUT /FCC/item/import for the escape hatch it points at: the export writes the campaign to a file and the import answers it as a new unsaved one, whose fleet the operator changes before saving.

Since M3 the comparison is made after resolution, on the resulting set of cpeId`s, so a `configs[] selection whose manufacturer, model or conditionId was reworded but which resolves to exactly the stored devices is a legal no-op edit. What a compliance report has to describe is the devices that were measured, and neither the wording of a condition nor the order of a list changes those.

The set of models is frozen with the devices, and there is deliberately no fcc_test_model table.

A model is fcc_test_device.model_id, so an immutable device set is already an immutable model set. A second table could only be a copy of a SELECT DISTINCT over this one, and a copy that can disagree with its source is worse than the join it saves: the per-model figures a report needs are a GROUP BY over this snapshot, and they are then guaranteed to describe the same devices the run measured.

Why a Snapshot Carries the Protocol

protocol is not a display field. The fleet is frozen precisely so that a regulatory report describes the devices the campaign actually ran against, and the protocol is part of that description.

The ACS of v1 answers SKIPPED(UNSUPPORTED) for a device it cannot drive (USP and Other Non-CWMP Devices Are Included and Flagged), so "how many of these devices were never going to produce a measurement at all" is a question asked of a finished run — weeks later, when a report is assembled and a device count does not match a measurement count. At that point ftacs.cpe may perfectly well say something different, because the CPE was re-provisioned, or migrated from TR-098 to TR-181, or removed. Without this column the only way to answer the question is to re-resolve the fleet against a live table, which is exactly what a snapshot exists to avoid.

Three properties of the column are load-bearing:

  • It holds the constant NAME, never an ordinal, like every other enum-valued column in ftfcc (see FccDbConstants). The ACS’s own CpeProtocolEnum is persisted by ordinal on its side, which is a recorded known issue there — inserting a constant ahead of LWM2M silently re-maps every stored device. A name cannot do that.

  • It is a plain String on the entity and not @Enumerated, unlike the enum columns of fcc_test. Those hold this service’s own vocabulary; this one holds a foreign one. A constant that a later build of this service no longer declares must still be readable in a report years from now, and a mapped enum would instead fail the whole read of that campaign’s fleet on an unknown name. The protocol is already a name on the wire (FccResolvedDeviceItem.protocol), so nothing downstream needs it typed.

  • The mapping from cpe.protocol_id happens in exactly one place, FccTestDeviceDao.bind, through FccCpeProtocolEnum.fromProtocolId. The raw id travels out of the ACS schema untouched — a null there means TR-069 rather than "unknown", and only that method knows it — so the coercion is applied on the way into the snapshot and never repeated by a caller.

Nullable, and never back-filled. A fleet frozen before this column existed has no protocol, and filling it in from a live table would be a snapshot rewritten with today’s facts, which is the one thing this table must never contain.

Two Indexes for the M4 Report Filters

changelog_add_fcc_test_device_model_index.xml (M4) adds two more, in a changelog of its own so changelog_create_fcc_test_device.xml keeps its checksums.

Index Query it serves

fcc_test_device(model_id, test_id)

The model chain of FccReportScopeResolver (Endpoints Implemented in M4): SELECT DISTINCT test_id FROM fcc_test_device WHERE model_id IN (…​), and the redundancy check SELECT test_id, COUNT(id) FROM fcc_test_device WHERE test_id IN (…​) AND model_id IN (…​) GROUP BY test_id. model_id leads because it is the predicate both statements start from; test_id follows so the second is answered from the index alone rather than by visiting the rows.

fcc_test_device(cpe_id)

The device filter of the same resolver: SELECT DISTINCT serial FROM fcc_test_device WHERE cpe_id = ? AND serial IS NOT NULL. A report names a cpeId; the result tables know only serials, so this lookup runs once per filtered request.

Neither predicate is served by the existing (test_id, cpe_id) UNIQUE index above: a composite index can only be used from its leading column, and both new reads start from model_id or from cpe_id, never from test_id.

fcc_activation

ONE ACTIVATION IS ONE CONFIGURATION OF ONE RUN. A campaign with two configurations and reactivation_days = 7 produces fourteen of these over its lifetime: seven runs, two activations each.

A RUN IS (test_id, seq) AND NOT A ROW. The ACS command carries one configuration and one device list, so a campaign of N configurations is N commands — each with its own idempotency_key and its own command_seq, which is what keeps the ACS from taking the second for a retry of the first and what stops it discarding all but one as stale. They share the seq, because the RUN is one slot of the recurrence rule: two configurations must not consume two of the campaign’s daily runs. fcc_activation(test_id, seq) was already non-unique — it was written so because a failed dispatch may be retried — and it serves this for a different reason.

That N-rows-per-run shape is why the campaign’s own state machine is fed a run-ended event only by the last activation of the run to reach a terminal state: letting the first configuration to finish move the campaign would take it out of RUNNING and, worse, PLAN_REACTIVATION would create the next run while the rest of this one was still measuring. The predicate is a query over the siblings' state and deliberately not a counter — see FccActivationTransitionService.isLastOfRunToEnd.

Column Type Meaning

test_id

BIGINT NOT NULL

config_id

BIGINT

The fcc_test_config this run measures. NULLABLE, AND NEVER BACK-FILLED: a run dispatched before configurations were addressable had exactly one, but attaching it to the configuration row that happens to exist NOW would claim the run measured a configuration that may have been edited, re-ordered or replaced since — and this table’s whole purpose is to record what the command actually carried, independently of the test rows it was built from. NULL says "a single-configuration run from before configurations were addressable", which is the truth.

seq

INT NOT NULL

1 for the first run, +1 for every reactivation. Shared by every activation of one run — see [a-run-is-not-a-row].

idempotency_key

VARCHAR(64) UNIQUE NOT NULL

UUID, generated once when the row is created and reused verbatim on every republish of the command. The ACS answers DUPLICATE for a key it has already processed — accepted or rejected; which of the two it was is carried in the ack’s reason and applied by this service, see DUPLICATE Carries the Original Outcome. The unique index is the single defence against a double start: if a command is republished (outbox retry, a service restart between offer() and the state update, ACS failover), inserting a second activation with the same key fails in the database — the ack-based DUPLICATE check alone would arrive too late.

command_seq

BIGINT NOT NULL

Value of fcc_test.command_seq at the moment this activation was created. It is not re-used: a republish of the same command keeps it, a rejected command still consumes it, and a corrected command gets a new one — see Hazelcast Queues. Since wire contract 1.3 the ack carries it back, which is what lets a stale ack be told apart from a fresh one (Matching an Ack).

state

VARCHAR(32) NOT NULL

PLANNED | DISPATCHING | RUNNING | STOPPING | STOPPED | FINISHED | FAILED.

stop_reason

VARCHAR(32)

USER_STOP | USER_PAUSE | WATCHDOG | ACS_UNAVAILABLE | ERROR.

failure_reason

VARCHAR(255)

Free text from the ACS ack or the local failure. Deliberately not an enum — it is diagnostic text, not a value other code branches on.

source

VARCHAR(32), default ACS_STREAM

Same dictionary as fcc_test.source (ACS_STREAM | LEGACY) — not the same dictionary as the ClickHouse fcc_result_*.source, see [two-source-dictionaries].

window_start_at, window_end_at

DATETIME(3)

The window the ACS is allowed to run this activation in. window_end_at is recalculated from the actual ACTIVATION_STARTED timestamp when the start is late, and is what the watchdog compares against.

started_at

DATETIME(3)

Set when the ACS reports ACTIVATION_STARTED, not when the command is sent.

stopped_at, finished_at

DATETIME(3)

in_flight_count

INT, default 0

Devices the ACS still had in flight when it acknowledged a STOP.

device_count

INT, default 0

Size of the device set this run was dispatched with.

settings snapshot (30 columns: test_type …​ hubb_id)

mixed

See [fcc-activation-snapshot] below.

Indexes: fcc_activation(idempotency_key) UNIQUE (see above); fcc_activation(state, window_end_at) serves the window watchdog (WHERE state = 'RUNNING' AND window_end_at IS NOT NULL AND window_end_at ⇐ ?); fcc_activation(test_id, seq), deliberately not unique, serves the per-test run history behind the test detail screen and the regulatory export — forbidding a repeated seq here would turn a recoverable dispatch retry into an insert error.

M2 adds two more, in changelog_add_fcc_activation_indexes_m2.xml. Neither is served by the existing (state, window_end_at) index: a composite index can only answer a range on its first non-equality column, so a predicate on window_start_at or on stopped_at behind an equality on state falls back to scanning every row in that state.

Index Query it exists for

fcc_activation(state, window_start_at)

The PLANNED-reactivation poller, every 30 seconds: WHERE state = 'PLANNED' AND window_start_at ⇐ ? ORDER BY window_start_at, id, then FOR UPDATE SKIP LOCKED on the claimed ids. Without it every tick scans the PLANNED rows of the whole table — and a campaign with reactivation_days = 7 plans its next run the moment the previous one finishes, so PLANNED is never a small set. The same index also backs the stalled-activation claim, which is state = 'DISPATCHING' AND window_start_at ⇐ ? — PLANNED is deliberately not a stalled state, see Ack Timeout and the Republication Window.

fcc_activation(state, stopped_at)

The stop-grace watchdog: WHERE state = 'STOPPING' AND stopped_at IS NOT NULL AND stopped_at ⇐ ? ORDER BY stopped_at, id. A run sits in STOPPING only until its in-flight devices drain or ft.fcc.stop-grace runs out, so the set is normally tiny — but the poll is unconditional and would otherwise pay for the whole state partition on every tick.

The IS NOT NULL in both predicates is part of the statement and not an assumption about the data: window_end_at and stopped_at are nullable — a run whose test carries no total_duration_h has no computed end — and NULL ⇐ ? is unknown rather than false, so leaving it out would make the statement’s meaning depend on the optimizer rather than on the schema.

M4 adds one more index, in the same changelog_add_fcc_test_device_model_index.xml (Two Indexes for the M4 Report Filters): fcc_activation(started_at), serving the period filter of POST /FCC/reports/list (WHERE started_at >= ? AND started_at ⇐ ? AND test_id IN (SELECT id FROM fcc_test …​)). All three existing indexes on this table lead with state, which a report never constrains — it reports on runs in every state — so a report over a date range alone had no index to use at all before this one. started_at and not created, because the filter is about when the measurements were taken, not when the row was inserted — see [report-period-predicate].

fcc_acs_outbox

The transactional outbox toward fccCommandQueue. A Hazelcast offer() is not part of a database transaction, so the command is written here in the same transaction that creates the activation and bumps fcc_test.command_seq; a separate dispatcher performs the offer(). Nothing is ever sent that is not committed, and nothing committed is silently lost.

Column Type Meaning

activation_id

BIGINT NOT NULL

idempotency_key

VARCHAR(64) NOT NULL

Copied from fcc_activation: the ack carries the key, not the row id.

command_seq

BIGINT NOT NULL

type

VARCHAR(32) NOT NULL

ACTIVATE | STOP. PAUSE and UPDATE_CONFIG exist in FccCommandTypeEnum but the ACS rejects them in v1 and this service never writes them here.

payload

CLOB / LONGTEXT NOT NULL

The FccCommandDto as JSON, not Java serialization. These rows must survive a deploy that changes the DTO shape, and only JSON degrades gracefully when a field is added; a serialized Java graph would fail to read back after a serialVersionUID or shape change and strand a committed command. The DTO contract’s "only ever add fields" rule (DTO Contract Rules) applies to this stored JSON exactly as it does to the queue payload — the Java serialization on the Hazelcast queue itself is a separate, short-lived hop.

state

VARCHAR(32), default PENDING

PENDING | SENT | ACKED | ABANDONED. Mapping to fcc_activation.state: row written (PENDING) moves the activation PLANNED → DISPATCHING; a successful offer() sets SENT (activation stays DISPATCHING); an applied ack sets ACKED, and the activation moves DISPATCHING → RUNNING for ACCEPTED (or a DUPLICATE whose reason unwraps to an acceptance) and DISPATCHING → FAILED for a rejection, including a DUPLICATE carrying REJECTED:<reason> — see DUPLICATE Carries the Original Outcome. An ack whose commandSeq is older than the row’s is ignored outright and changes neither state (Matching an Ack). ABANDONED (attempts exhausted) fails the activation in the same transaction: FccOutboxPublisher calls FccStalledActivationService.failAbandoned right after FccOutboxDao.markAbandoned, so the run goes FAILED with a failure_reason naming the exhausted outbox row and the owning fcc_test moves to ERROR. That is a second, independent route to the same outcome — the separate, age-based FccStalledActivationService.failStalled check (window_start_at older than ft.fcc.activation-max-delay) still exists alongside it and, with the shipped defaults, normally fires first; neither one is the only way a stuck activation gets failed — see The Transactional Outbox.

attempts

INT, default 0

sent_at

DATETIME(3)

next_attempt_at

DATETIME(3) NOT NULL

NOT NULL so the dispatcher predicate never special-cases a null; a row due immediately is written with "now".

last_error

VARCHAR(512)

Since M2 the outbox carries STOP rows written by three different paths, all through the single FccStopCommandPublisher: the operator stopping or pausing a campaign, the operator deleting one whose run is still live, and FccStalledActivationService giving up on a run the ACS never acknowledged. A STOP is always a new command — its own idempotency key, command_seq = fcc_test.command_seq + 1 — and never a republish, because the dispatcher re-sends a stored payload verbatim and reusing the ACTIVATE’s key would make the two commands indistinguishable to the ACS. See An Effect Is a Declaration, Not an Action.

Indexes: fcc_acs_outbox(state, next_attempt_at) serves the 5-second dispatcher (WHERE state = 'PENDING' AND next_attempt_at ⇐ ? ORDER BY next_attempt_at, id); fcc_acs_outbox(state, sent_at) serves the 30-second ack-timeout claim (WHERE state = 'SENT' AND sent_at ⇐ ? ORDER BY sent_at, id), which the first index cannot answer; a unique index on idempotency_key serves the lookup FccAckService does for every ack and, being unique, also rules out the duplicate that would make that Optional lookup throw. fcc_acs_outbox(activation_id) serves ack correlation. Of these four, (state, next_attempt_at) and (activation_id) ship with the table itself in changelog_create_fcc_acs_outbox.xml; only (state, sent_at) and the unique idempotency_key index were missing and were added afterwards, in their own changelog_add_fcc_acs_outbox_indexes.xml, a separate changelog so the table changelog keeps its checksums.

A fifth index, fcc_acs_outbox_state_created_idx (state, created), was added in M5 (changelog_add_fcc_acs_outbox_metrics_index.xml, again a separate changelog so the earlier two keep their checksums), to serve FccMetricsSampler’s outbox sample — see Outbox Metrics and the sampler’s latency isolation. The M5 code review found the sampler’s own javadoc had been claiming an access path that did not exist: the query it described as index-served was `SELECT state, COUNT(*), MIN(created), MIN(sent_at) …​ GROUP BY state over the three live states, and no index on this table carries created — (state, next_attempt_at) and (state, sent_at) both answer the state predicate but neither can serve MIN(created) without a row lookup per matching entry, which is exactly the access path both database optimisers abandon in favour of a full scan once that cost is priced in. ABANDONED was also one of the three states counted, and it should never have been on a 15-second tick at all: markAbandoned is terminal and nothing in this service purges the row afterward, so ABANDONED accumulates for the life of the installation exactly like ACKED does — it is not bounded by work in flight the way PENDING and SENT are. The fix is now one statement per state rather than one grouped query: PENDING and SENT are counted from (state, created) and the pre-existing (state, sent_at) respectively, both index-only reads with MIN(…​) answered as the head of the range in a single descent on both vendors, and ABANDONED is counted separately, on its own five-minute cadence rather than every sample tick, since a range over a set that only grows is not something to put on a 15-second poll.

fcc_device_state

Where one device of one run currently is, as last reported by the ACS on fccDeviceStateQueue.

This is current state, not history: one row per (activation_id, cpe_id), overwritten in place. The history of a run is its measurements in ClickHouse; this table answers the single question the per-device status screen asks — "what is this CPE doing right now, and how many times has it retried".

Column Type Meaning

activation_id

BIGINT NOT NULL

The run. Keyed on the run and not on the test: a device that took part in three daily runs of one campaign has three rows.

cpe_id

INT NOT NULL

ftacs.cpe.id — the id the ACS addresses the device by in FccDeviceStateDto.

serial

VARCHAR(64)

state

VARCHAR(32) NOT NULL

INITIAL | OFFLINE | SKIPPED | WAN_CHECK | CROSS_TALK | SENT | FINISHED | ERROR | STOPPED. The wire dictionary com.friendly.fcc.dto.FccDeviceStateEnum, stored and rendered verbatim — the ACS owns this vocabulary end to end and a second local copy would only add a mapping that can drift. The precedent is fcc_acs_outbox.type, which stores FccCommandTypeEnum the same way. STOPPED (contract 1.5) is a device halted by a STOP between steps — not a failure, its own category exactly like CROSS_TALK, never folded into error — see the wire dictionary. It needed no schema migration: the column is already VARCHAR(32) and the longest existing value already fits. Only the changelog’s XML comment naming the dictionary was updated (changelog_create_fcc_device_state.xml), which is checksum-neutral because Liquibase computes a changeset’s checksum from the parsed change objects and never from an XML comment.

state_reason

VARCHAR(32)

UNSUPPORTED | BUSY | TIMEOUT | NO_WAN_IFACE | DEVICE_FAULT | ACTIVATION_GONE | USER_STOP | UNKNOWN — why the device is in that state when the state alone does not say. The opposite case from state: the ACS sends it as a free-form String, so it is normalized through FccDeviceStateReasonEnum.fromWire and an unrecognised value lands as UNKNOWN rather than poisoning the column. See Device State Reasons.

iteration, wan_retries, failed_retries, skip_count

INT NOT NULL, default 0

Monotonic counters, merged with GREATEST(stored, incoming). See [device-state-counters-not-null] for why all four are NOT NULL with a default.

observed_at

DATETIME(3) NOT NULL

The instant the ACS observed the state, not the instant this row was written — created and updated record the latter. The two differ by the queue latency, and the upsert compares the observed instant, never the local clock, to decide whether an incoming message is newer than the stored one.

Six of these columns are written and never read back, and they stay that way on purpose.

serial, state_reason, iteration, wan_retries, failed_retries and skip_count are kept current by the fccDeviceStateQueue listener, but the only readers of this table are two aggregate queries — countByActivationIdAndState and countGroupedByActivationId — and both look at state alone. DEV-1381 #9 is why: POST /FCC/devices/status answers six counters for a campaign and the per-device grid that would have shown these columns is not in the contract.

They are not dropped, because the data is the only record of why a device ended where it did and it cannot be reconstructed afterwards: the ACS sends each state once, and a run that finished with 4 000 devices SKIPPED has nothing else to say which were busy, which had no WAN interface and which were stopped by the operator. Writing them costs one already-existing UPSERT; re-deriving them later would cost the measurements that were never taken. If a device grid comes back — the obvious form being a page of fcc_device_state for one activation_id — it needs no migration and no back-fill, which is the whole point of keeping them.

Why the counters are NOT NULL with default 0: GREATEST returns NULL for a NULL argument.

The row is written by an atomic upsert — INSERT …​ ON DUPLICATE KEY UPDATE on MySQL, MERGE on Oracle (FccDeviceStateDao) — whose whole point is that iteration, wan_retries, failed_retries and skip_count only ever move forward, via GREATEST(stored, incoming). GREATEST returns NULL as soon as one argument is NULL, on both MySQL and Oracle, so a single nullable counter would silently erase the stored value the first time a message arrived without it — not lose to it, erase it.

Three halves of the same rule, and all three are required:

  1. NOT NULL DEFAULT 0 in the schema;

  2. primitive int components in FccDeviceStateRow, bound with setInt and never setObject, so the type system makes it impossible to forget;

  3. FccDeviceStateService.orZero substituting 0 for an absent wire value, where "the ACS did not say" still means "nothing new to raise it by".

Why GREATEST at all: Hazelcast guarantees delivery, not order, so a message reporting iteration 4 can arrive after one reporting iteration 5, and both are true statements about the past. Taking the maximum makes the four columns immune to that reordering — the stored value is the furthest the device is known to have got, and no arrival order can walk it backwards.

serial, state and state_reason are not monotonic — a device legitimately goes SENT, then ERROR, then WAN_CHECK, and there is no "greater" state — so the only correct merge for them is newest-wins, decided by observed_at with >= rather than >: two messages can carry the same observation instant, and the later arrival is then at least as informed as the stored one. observed_at itself takes the maximum, so a reordered older message updates the counters it can only raise and leaves both the state and the watermark alone.

Index Why

fcc_device_state(activation_id, cpe_id) UNIQUE

The index the atomic upsert is built on. Without a unique key on exactly these two columns, MySQL’s ON DUPLICATE KEY has nothing to match and Oracle’s MERGE …​ ON (…​) degrades into a full scan per row; the only remaining option would be a read-modify-write in Java, which loses a concurrent update from a second service instance consuming the same queue. It also serves the per-device lookup WHERE activation_id = ? AND cpe_id = ?.

fcc_device_state(activation_id, state)

The per-device status screen and the grouped progress counters behind it (WHERE activation_id = ? AND state = ?), and the STOPPING drain recount SELECT COUNT(*) …​ AND state = 'SENT'. The unique index above leads with activation_id too, but cannot answer a state predicate without visiting every device of the run — and a run holds up to ft.fcc.max-devices-per-test (20000) of them.

=== fcc_audit

Who did what to a campaign — an append-only trail, never updated and never deleted.

The four audit columns every table already carries answer "when was this row last touched and by whom"; they do not answer "who stopped this campaign on the 14th", because an update overwrites its predecessor. A compliance campaign disturbs subscriber lines and its results end up in a regulatory filing, so the lifecycle actions have to be a trail of their own. And because the product has no role model, this row is the only accountability there is — see No Authentication by Design.

Column Type Meaning

entity_type

VARCHAR(32) NOT NULL

TEST | MODEL_CAPABILITY | DOMAIN_SETTING (FccDbConstants.AUDIT_ENTITY_TYPE_*). Deliberately generic — entity_type + entity_id rather than a test_id column — and M3 is where that paid off: the two administration tables got their trail with no second table and no ALTER. It is a set of String constants and not an enum, unlike action, for exactly that reason. See [audit-entity-types-m3].

entity_id

BIGINT

The business key of the row the action was taken on, never a surrogate id: fcc_test.id for TEST, the model id for MODEL_CAPABILITY, the domain id for DOMAIN_SETTING — because those are what an auditor reading the trail after a delete still holds. Nullable: an action refused before the row existed still belongs in the trail. It is not an FK and the row it names may already be gone — DELETE /FCC/items removes fcc_test and keeps this row, which is the one case where the trail is the only record.

action

VARCHAR(32) NOT NULL

CREATE | UPDATE | ACTIVATE | STOP | PAUSE | DELETE (FccAuditActionEnum). Lifecycle actions only — the operator decisions a regulatory audit asks about. A scheduled activation is recorded as ACTIVATE with the booked instant in the details rather than as a vocabulary of its own: the decision being recorded is "this operator committed the fleet to a run". CLONE is gone, with POST /FCC/item/clone: nothing can write that action any more, and a development database that already holds such rows keeps the string — nothing reads this column back into the enum, so they are history rather than a mapping problem.

user_id

INT

X-FT-User-Id. A null means the engine did it, not "unknown user" — the scheduled-campaign poller and the reactivation path create runs nobody asked for at that moment, and an auditor reading a daily campaign has to be able to tell the operator’s first activation from the six the service performed by itself.

domain_id

INT

The domain the test belonged to at the time, so the trail stays readable after a test is moved or removed.

details

VARCHAR(1024)

Free-text summary — "mode=IMMEDIATELY activationId=1001 seq=1 devices=120 reactivationDays=7", "reason=USER_PAUSE activationId=1001 stopCommandSent=true". Deliberately not structured JSON: it is read by a human during an audit, never queried by the service, and a CLOB would make the trail dramatically more expensive to scan for the one screen that does read it. Truncated at 1024 rather than rejected — an over-long detail string is cosmetic, while an exception here would roll back the operator action the row describes.

Index: fcc_audit(entity_type, entity_id, created) is built for the read behind the history tab of a test, WHERE entity_type = ? AND entity_id = ? ORDER BY created DESC — but FccAuditRepository declares no query method at all today, and nothing else reads fcc_audit either. The index anticipates a read that does not exist yet; that is a defensible thing to have done, since the append-only rule and the column shapes already have to be right before a reader can be built on top of them, but it means the index is currently unexercised rather than serving anything. created is the third column so that, once a reader is added, the sort is answered from the index instead of a filesort over a trail that only grows. The leading entity_type is what lets the same index serve the M3 entity types without a second one.

==== Three Entity Types, One Action Vocabulary

FccAuditService gained an overload in M3 (record(entityType, action, entityId, domainId, userId, details)); the original five-argument method is now a thin delegate that passes AUDIT_ENTITY_TYPE_TEST.

action stays the closed FccAuditActionEnum while entity_type is a set of String constants, and the split is deliberate: CREATE / UPDATE / DELETE mean the same thing on all three entities, so per-entity constants such as MODEL_CAPABILITY_UPDATE would make the trail impossible to read across them — "what changed on the 14th" would become three queries instead of one — for exactly no gain.

entity_type entity_id domain_id Written by

TEST

fcc_test.id

The test’s domain

CREATE, UPDATE, ACTIVATE, STOP, PAUSE, DELETE

MODEL_CAPABILITY

The model id

null

CREATE, UPDATE, DELETE on PUT/DELETE /FCC/admin/model-capability

DOMAIN_SETTING

The domain id

The same domain id

CREATE, UPDATE, DELETE on PUT/DELETE /FCC/admin/domain-setting

The null domain_id on MODEL_CAPABILITY is a statement, not a gap. The change belongs to no domain — the table is installation-global ([fcc-model-capability]) — and recording the actor’s own domain would suggest the change was scoped to it, which is precisely the misreading the X-FT-Domain-Scope: ALL requirement exists to prevent.

Both new paths write in the same transaction as the change they describe, enforced by Propagation.MANDATORY exactly as the campaign paths are.

=== fcc_result_dead_letter

The last resort for a measurement that could not be stored.

A measurement is the product of a compliance campaign: the ACS disturbed a subscriber line to produce it and it may end up in a regulatory filing, so it must never be silently lost. Through M1 the only place a result could live between arriving on the Hazelcast queue and reaching ClickHouse was an in-memory buffer, and exhausting that buffer — or restarting the service while it was full — destroyed the rows. This table is where they go instead; it replaces both TODO(M2) drop paths and the shutdown drop.

Column Type Meaning

activation_id, test_id, serial, direction, iteration

BIGINT / BIGINT / VARCHAR(64) / VARCHAR(32) / INT

All five are denormalised out of the payload purely so an operator answering "what did we lose for run X" can use an index and a WHERE clause instead of parsing JSON. The payload stays the source of truth on resend. direction is DOWNLOAD | UPLOAD | PING.

reason

VARCHAR(32) NOT NULL

CLICKHOUSE_UNAVAILABLE | LATE. Two different failures, handled differently — which is why this is a dictionary and not free text: one is retried, the other never is. See [dead-letter-reasons].

state

VARCHAR(32), default PENDING

PENDING | RESENT | DISCARDED. PENDING means "waiting to be resent". RESENT means "handed back to the ingest path", not "already in ClickHouse" — that write is asynchronous. DISCARDED is a row that will never be resent: a payload that cannot be read, and every LATE arrival, which is written straight into DISCARDED and never passes through PENDING at all — see [late-is-parked-discarded]. Terminal rows are kept, never deleted: the trail of what could not be stored, and of what was eventually recovered, is itself part of what a compliance audit asks for.

attempts

INT NOT NULL, default 0

Bounds nothing in v1. There is no resend poller and no attempt ladder yet, so this column is 0 on every PENDING row and FccResultDeadLetterDao.findPending carries no predicate on it at all — the only resend path is the operator calling POST /FCC/admin/result-dead-letter/resend. The column is reserved for a future automated resend poller, and is deliberately not read back into the row this service maps: selecting it into a component nobody consults is how a reader comes to believe a ladder exists. This is not an outbox: there is no idempotency key and no backoff column, because the destination de-duplicates on the natural key (activation_id, serial, direction, iteration).

last_error

VARCHAR(512)

received_at

DATETIME(3) NOT NULL

When the result reached this service — not when it was parked. A row written after a long outage still has to record the arrival it describes, and it is this instant, never Instant.now() at flush time, that the after-stop comparison uses.

payload

CLOB / LONGTEXT NOT NULL

The FccResultDto as JSON, never Java serialization — the identical rule as fcc_acs_outbox.payload and for the identical reason: the row has to survive a deploy that adds a field to the DTO. It is the wire form and not a mapped row, so a resend runs the same ingest path a live message runs and cannot drift from it.

reason What it means and what happens next

CLICKHOUSE_UNAVAILABLE

The result is valid, the store is not reachable — or the in-memory retry buffer was exhausted, or the JVM was shutting down with rows still buffered. Written as PENDING, and it is meant to be resent, through POST /FCC/admin/result-dead-letter/resend.

LATE

The result arrived more than ft.fcc.stop-grace after its run’s stopped_at. Written straight as DISCARDED and never resent: inserting a measurement into a window a report has already closed would change a number somebody has already filed. See [late-is-parked-discarded] and The After-Stop Window.

A LATE row is parked straight into DISCARDED, never into PENDING.

PENDING means "waiting to be resent", and a LATE row is never resent by design. It is also what keeps the recovery path working at all: LATE rows are written in bursts, when a stopped run’s grace expires, so they are both numerous and the oldest rows in the table. Leaving them PENDING would fill every oldest-first read with rows the resend can only skip, and no CLICKHOUSE_UNAVAILABLE measurement would ever be recovered again.

The read guards against this from the other side too: FccResultDeadLetterDao.findPending carries AND reason <> 'LATE' in both vendor statements, so rows written by an earlier build — when LATE did land in PENDING — cannot block the head of the read either. A resend that still meets one counts it (fcc.result.resend.skipped.late) — there is no fcc.result.deadletter.* meter — and there is deliberately no per-row log line for it: FccResultDeadLetterService.resend sits in a loop over up to ft.fcc.dead-letter.resend-batch-size rows, so a per-row line would print as many times as the skip does, and what an operator acts on is the aggregate fcc dead letter resend read=…​ skippedLate=…​ line logged once per call plus the counter itself, which is the one signal that the table holds rows the recovery path cannot use.

The reason column is kept either way: DISCARDED says what happened to the row, LATE says why, and an operator asking "what did this run measure after it was stopped" needs the second one.

Index: fcc_result_dead_letter(state, created) serves the only reader, WHERE state = 'PENDING' AND reason <> 'LATE' ORDER BY created, id. The state prefix is what keeps the read proportional to the backlog rather than to the whole table, since RESENT and DISCARDED rows are kept forever and never removed by the service.

FccResultDeadLetterService.store runs in its own transaction (REQUIRES_NEW) and never throws. The ingest path that calls it is already in the middle of a failure; a record written in the caller’s transaction could be rolled back with it, and the one thing this table must never do is vanish together with the problem it documents. If even that insert fails, the measurement is genuinely gone — not stored, not parked, nothing will retry it — and only then is fcc.result.lost incremented, separately from fcc.result.parked.

=== fcc_model_capability

What one CPE model can be asked to do, as far as this installation has been told. At most one row per model.

Column Type Meaning

id

BIGINT PK

Surrogate. PUT /FCC/admin/model-capability returns it, but no caller ever keys on it — the model id is the business key.

model_id

INT NOT NULL

ftacs.product_class.id — the very value fcc_test_device.model_id snapshots for every device of a fleet, and the value FccCatalogService hands to the wizard as FccCatalogModelItem.modelId. There is deliberately no foreign key: this schema has none at all (Schema-Wide Rules), and the referenced table lives in a schema this service holds a read-only account on, so a constraint could not be created from here and a migration that tried would fail at customer-install time. Referential integrity is a write-time product rule instead — FccModelCapabilityAdminService refuses a model_id that ftacs.product_class does not know (FtacsCatalogDao.modelExists).

connections_dl_max

INT

Ceiling for TR-143 DownloadDiagnostics.NumberOfConnections. Nullable, and null is UNCONSTRAINED rather than zero.

connections_ul_max

INT

The same for UploadDiagnostics.NumberOfConnections. Two columns for the same reason fcc_test_setting_speed carries two: the devices advertise DownloadDiagnosticMaxConnections and UploadDiagnosticsMaxConnections separately and routinely differ between the directions, so a row may record a known download ceiling while saying nothing at all about upload.

supports_time_based

TINYINT(1) NOT NULL, default 0

Whether the model can run a TR-143 time-based test (TimeBasedTestDuration). Recorded for v2 and consulted by nothing: FccTestValidator refuses testMode=TIME_BASED for every model regardless of this flag, because the ACS of v1 cannot parse a time-based result. The column exists now so that the v2 work is a code change and not a migration on a live schema, and so that an operator can start populating it from device datasheets before the code catches up. NOT NULL with a default, unlike the two ceilings, because this is a two-valued fact: "unknown" and "not supported" lead to the same v1 behaviour, so a third state would be recorded and never read.

A model with no row here is UNCONSTRAINED, which is why a null ceiling is not zero.

The table is an allow-list of known limits, not a registry of every model. The vast majority of an installation’s models will never have a row — nobody has read their datasheet — and reading an absent row as "zero connections" would refuse every test on every one of them, with a 400 naming a ceiling nobody entered.

Null therefore means "nothing is known about this model, so neither this service nor the UI may narrow the operator’s choice". Three places implement that one reading, and all three must keep it:

  1. FccModelCapabilityProvider.capabilitiesOf leaves an absent model out of the returned map entirely, and callers fall back to FccModelCapability.unknown() — (null, null, false).

  2. FccCatalogService renders those nulls straight onto FccCatalogModelItem.connectionsDlMax / connectionsUlMax, so the UI leaves the input unconstrained.

  3. FccTestValidator.validateConnectionCeilings takes the minimum non-null ceiling across the fleet’s models, so a null neither lowers nor raises it — see The Capability Ceiling Is Refused, Not Clamped.

The one place a missing row is legitimately a 404 is POST /FCC/admin/model-capability/details, and that is not a contradiction — see 404 on Read, 403 on Write.

Installation-global, not per-domain. There is no domain_id column, because a connection ceiling is a property of the hardware: the same model behaves the same way in every domain. That is also why the admin write path requires X-FT-Domain-Scope: ALL — a row here constrains every operator of the installation, and this product has no role model that could express "installation administrator" any other way (The Two Administration Tables Are Not Authorized the Same Way).

Index: fcc_model_capability(model_id) UNIQUE. It enforces one row per model in the database rather than only in the admin service — the PUT is an upsert keyed on model_id, and two rows for one model would make the ceiling that is applied depend on which one the read happened to return first. It also serves the only two reads there are: WHERE model_id = ? (the admin details screen) and WHERE model_id IN (…​) (FccModelCapabilityProvider, which resolves the whole model set of a catalog page or of a resolved fleet in one statement, chunked at 1000 for Oracle).

Oracle additionally gets a FCC_MODEL_CAPABILITY_0 sequence, the same convention every other table of this schema uses.

=== fcc_domain_setting

The middle level of the three-level settings chain — see [settings-resolution-chain] for what the chain does and when it runs. At most one row per domain.

The per-domain level exists for multi-ISP installations: one deployment serves several ISPs, each is a domain, and each has its own test servers. A single global URL is wrong for all but one of them, while making every operator retype the same two URLs into every campaign is how they end up mistyped.

Column Type Meaning

id

BIGINT PK

Surrogate; the domain id is the business key.

domain_id

INT NOT NULL

ftacs.cpe.location_id — what everything outside the ACS calls the domain. No foreign key, for the same two reasons as fcc_model_capability: this schema has none at all, and the domain table lives in a schema this service may only read.

url_dl, url_ul

VARCHAR(512)

This ISP’s own test servers: DownloadDiagnostics.DownloadURL and UploadDiagnostics.UploadURL. Two columns for the reason fcc_test_setting_speed already gives — independent parameters of two different TR-181 objects, routinely different hosts — and the same VARCHAR(512), so that a value copied from here into a test can never be truncated on the way. These are two of the six fields of the chain with no global level below them — see [which-fields-have-which-levels].

threshold_dl, threshold_ul

INT

WAN cross-talk thresholds in kbps, per direction: above these the link is considered busy with somebody else’s traffic and the iteration is recorded as CROSS_TALK rather than as a speed measurement. Per-domain because the figure depends on the access technology the ISP runs, not on the campaign. This column feeds three of the six fields of the chain with no global level below them: speed.wanDownloadThreshold and speed.wanUploadThreshold read it directly, and latency.wanUsageThreshold reads the same threshold_dl — FccTestService.resolveSpeedDefaults / resolveLatencyDefaults pass an explicit global = null for all three, since a latency run has no upload direction to sample and therefore no domain-level threshold_ul counterpart of its own — see [which-fields-have-which-levels].

speed_success_percent

INT

Percentage of the provisioned tier that still counts as a pass. Unlike the URLs this one does have a global level below it (FT Configs, then FccRuntimeDefaults.SPEED_SUCCESS_PERCENT = 80), because it is a policy figure that an installation can state once and most ISPs share.

latency_success_ms

INT

The pass/fail bound this service applies to a measured average round trip, in milliseconds; global fallback FccRuntimeDefaults.LATENCY_SUCCESS_MS = 100. The ACS publishes raw measurements and never a verdict, so the bound is ours to hold.

latency_host

VARCHAR(255)

The ISP’s default ping target, written into TR-181 IPPing.Host. The third of the six fields with no global level below the domain (with url_dl, url_ul above and the three cross-talk thresholds below). A default here does not make the field optional on a test — FccTestValidator.validateLatencySetting still refuses a latency campaign that has no host from either level. What it buys the operator is not retyping the ISP’s own target; what it must not buy them is a campaign that pings nothing.

Every field is nullable, and a null is indistinguishable from the absence of the row: both mean "this domain publishes no opinion on this value". FccDomainSettingResolver.forDomain returns an Optional and FccTestService.chain treats an empty one and a null field identically, on purpose — a caller that had to tell them apart would be encoding a distinction that has no meaning.

Index: fcc_domain_setting(domain_id) UNIQUE, enforced in the database and not only in the admin service. The PUT is an upsert keyed on domain_id, and two rows for one domain would make the URL a campaign is frozen with depend on which row the read happened to return first — the exact silent divergence the save-time freeze exists to prevent. It serves the read that runs on every save of every test (WHERE domain_id = ?, FccDomainSettingResolver) and the scoped admin list (WHERE domain_id IN (…​), the caller’s visible domains).

Oracle additionally gets a FCC_DOMAIN_SETTING_0 sequence.

=== fcc_legacy_import

The cursor and the progress of the legacy ETL, which reads the FCC results the ACS wrote before this service existed and replays them into ClickHouse — see Legacy FCC Import for what the import reconstructs and why.

Exactly two rows, one per result_kind, and that is the whole cardinality of the table. The two legacy sources are imported by independent cursors because they are not the same shape: ug_fcc_speed_result carries end_date, type and the bom/eom counters, and ug_fcc_latency_result has none of those and carries the round-trip aggregates instead. A single shared cursor would have to be the union of both column sets — and, worse, a stall on one kind (an unreadable row, a ClickHouse outage on one table) would hold the other kind’s import hostage. Two rows let one kind sit FAILED while the other keeps making progress, and let an operator restart one of them alone.

Column Type Meaning

result_kind

VARCHAR(32) NOT NULL

SPEED | LATENCY (FccLegacyResultKindEnum). The identity of the row; see the UNIQUE index below.

state

VARCHAR(32) NOT NULL, default IDLE

IDLE | RUNNING | STOPPING | COMPLETED | FAILED. COMPLETED is not terminal — the legacy flow keeps writing while this service runs, so it only means "everything whose start_date is below watermark_started_before is in ClickHouse" and the next start opens a fresh pass from that date. IDLE and FAILED differ in one thing: IDLE is interrupted on a committed cursor, FAILED is broken. A clean stop is therefore IDLE even when it carries a last_error.

run_id

VARCHAR(64)

Identifies one import attempt, and fences every mutating statement of the table: each one carries WHERE result_kind = ? AND run_id = ?, and a zero-row update means this run was evicted by a stale-heartbeat takeover and must stop writing. Changes on every start. See The Claim Is Enforced on Every Write.

zone_id

VARCHAR(64)

The zone this run interpreted the naive legacy timestamps in. It is a run parameter and not a property of the data: start_date, end_date, bom and eom are local wall clock with no offset stored anywhere, so a re-import under a different zone yields different instants for the same source rows, and that has to be visible afterwards rather than inferred. A resumed pass reads the zone back from here rather than from configuration — see Time Zone Is a Run Parameter.

cursor_ug_id, cursor_serial, cursor_direction, cursor_started_at, cursor_result_id, cursor_iteration

INT / VARCHAR(64) / VARCHAR(32) / DATETIME(3) / BIGINT / INT

One position, six columns, all five position values taken from one source row. cursor_direction holds the raw ug_fcc_speed_result.type as text and never a direction name; null for latency, whose rows have no direction column. See [legacy-import-composite-cursor].

pending_from_id, pending_result_id, pending_ug_id

BIGINT / BIGINT / INT

The legacy id range of the batch handed to ClickHouse and the campaign it was read for, committed before the insert and nulled after it. Non-null at startup means a previous run died inside exactly that window. See [legacy-import-pending-range].

watermark_started_before

DATETIME(3)

The completeness mark: every row of this kind whose start_date is below it has been imported, and the next pass covers [this, cutoff). A local wall clock, because the legacy column it is compared against stores no offset. Written only when a pass completes, never mid-pass. Nullable — a kind that has never completed a pass has no lower bound. See [legacy-import-watermark].

watermark_result_id

BIGINT

The highest legacy id a completed pass has covered. A diagnostic high-water mark that bounds no read; its one use is recognising a late arrival. See [legacy-import-watermark].

rows_read, rows_written, rows_skipped, rows_late_skipped, rows_invalid

BIGINT NOT NULL, default 0

BIGINT for anything that counts source rows: a legacy installation can hold tens of millions of results and an INT would wrap mid-import. rows_skipped is rows the crash recovery found already present in ClickHouse. rows_late_skipped is rows that reached ftacs after their day-window had been imported — excluded by the watermark, counted here and deliberately not back-filled (Late Arrivals Are Counted). rows_invalid is rows that cannot be expressed as a cursor position at all; one of them stops the pass where it stands (A Null Cursor Component Halts the Pass).

bytes_overflow_count, tier_parse_failed, status_unknown_count, properties_missing_count

BIGINT NOT NULL, default 0

Data-quality tallies, kept apart rather than summed because they need different answers: an overflowed total_bytes is a legacy schema limit, an unreadable tier is a bad row, a missing fcc_properties row is a missing configuration. Each is defined in Legacy FCC Import.

anchors_created, activations_created, devices_snapshotted, devices_unresolved

INT NOT NULL, default 0

Rows created in this schema, bounded by the number of legacy groups and devices rather than by the number of results — hence INT and not BIGINT. devices_unresolved counts serials that no live ftacs.cpe row matches; their measurements are still imported (A Serial With No Live cpe Row).

total_row_estimate

BIGINT

Counted once at the start, only so progress can be reported as a fraction. It is an estimate and nothing ever decides that the import is finished from it — the legacy flow keeps writing, so the number is stale the moment it is taken.

stop_requested

TINYINT(1) NOT NULL, default 0

The cooperative stop: the operator sets it, the running import notices it between batches and leaves a clean resumable cursor instead of being killed mid-batch.

heartbeat_at

DATETIME(3)

What distinguishes a slow import from a dead one, and what the claim’s staleness arm compares against. Refreshed between batches and over the slow parts of a pass that are not batches: the per-campaign MAX(id) prologue, the anchor resolve and the fleet top-up. Each of those can on its own outlast stale-after on a ten-million-row campaign, and an unbounded gap there is what makes a live worker evictable.

started_at, finished_at

DATETIME(3)

started_at is a pass parameter, not decoration: the pass-wide window cutoff is started_at - window-settle - 24h and is never derived from now, so a pass interrupted at campaign 50 and resumed an hour later still covers exactly the interval it set out to.

last_error

VARCHAR(1024)

Truncated, not a CLOB: this is the operator-facing reason the import stopped, and the full stack trace belongs in the log rather than in a status row.

Index: fcc_legacy_import(result_kind) UNIQUE — the two-rows-only rule enforced in the database and not only in the service. Every access is "the row for this kind": the ETL loads it by kind at start, updates it by kind after each batch, and /status reads both. A second row for one kind would hand a resuming import a stale cursor and re-import a range that was already written, which on ClickHouse cannot be undone. The unique index is also what makes the get-or-create at startup safe under two processes racing to begin the same import. Oracle additionally gets a FCC_LEGACY_IMPORT_0 sequence, the same convention every other table of this schema uses.

M4 also adds one index to the existing fcc_test, in changelog_add_fcc_test_legacy_ug_index.xml so the table changelog keeps its checksums: fcc_test(legacy_ug_id, type) UNIQUE, the key the import finds its own anchor campaign by — see One Update Group Can Become Two Anchors for why the type column is in it.

Four of the columns above — watermark_started_before, pending_ug_id, rows_late_skipped and rows_invalid — arrived with the M4 review, in changelog_add_fcc_legacy_import_watermark.xml and not in changelog_create_fcc_legacy_import.xml, for the same reason: that file has already run at every installation and its changesets have to keep their checksums. Each changeset is guarded by a not columnExists precondition with onFail="MARK_RAN", so an installation that already carries the column records the changeset as run rather than failing on it.

==== Why the Cursor Is Composite and Not a Last-Id

Rows are read in the order

(ug_id, serial, direction, start_date truncated to the second, id)

because that is exactly the order the per-group iteration rank is defined over: iteration numbers the measurements of one (window, serial, direction) group in the order they were taken, and the truncation to the second is what makes rows written within the same second rank deterministically instead of by insertion accident.

A plain "resume after id N" cursor cannot express a position in that order — ids are not monotonic along it — so a batch that ended in the middle of a group could not resume the group’s ranking. Hence the five position columns, plus cursor_iteration, which carries the running rank of the group the cursor currently sits in: a group split across a batch boundary continues at iteration N+1 on the next batch instead of restarting at 1 and producing two measurements that both claim to be the first.

cursor_result_id on its own therefore says nothing about how far the import has got. Only the tuple does.

Two rules the tuple has to obey, both of which the M4 review found broken:

  • cursor_direction stores the RAW legacy type as text, never a direction name. type 0, 1, 3 and null all map to the name DOWNLOAD, so the mapping is many-to-one and a name cannot be inverted into the number the legacy table is compared on. A cursor restored from a name would either re-admit rows that are already imported or drop the direction term from the traversal order half-way through a campaign.

  • All five position values come from ONE source row, and a null in any of them is refused rather than stored. In SQL both x > NULL and x = NULL are UNKNOWN, so such a cursor matches no row at all — the next batch comes back empty and the runner reads that as the end of the campaign. The pass halts instead; see A Null Cursor Component Halts the Pass.

==== Why pending_from_id / pending_result_id / pending_ug_id Exist

ClickHouse MergeTree has no unique constraint and no upsert: a row inserted twice stays in the table twice, forever, and there is no statement that can remove it. In a regulatory report that is not a cosmetic defect — it silently inflates the sample.

So the id range of a batch is written here and committed before the ClickHouse insert is attempted, and cleared to null only after the insert is acknowledged. A non-null pending_result_id found at startup therefore means the previous run died inside exactly that window and the batch’s fate is unknown; the recovery re-checks that bounded id range against ClickHouse — and only that range — before re-importing it, instead of having to reconcile the whole import.

The range is MIN..MAX of the batch and not first..last, because the traversal order above is deliberately not id order and the ids of one batch are scattered.

pending_ug_id ties the range to the campaign it was read for, and the recovery set is offered to that campaign and to no other. A run that died in the first batch of a campaign committed no cursor for it, so the stored cursor still names the previous campaign — which has nothing left to skip. Without the column the recovery set is handed to that previous campaign and cleared there, and the batch that genuinely is in doubt is re-imported into a table with no unique constraint. For the same crash, a non-null pending_result_id makes the row resumable on its own, with no cursor behind it: a fresh claim would otherwise erase the only record of which ids are in doubt before the probe could read it.

FccLegacyImportDao is hand-written JDBC and no method on it is transactional, which is what this contract rests on: each statement runs on autocommit, so the pending marker is durable the instant the call returns. Wrapping the import in a transaction would defer every cursor commit to the end of a run that may take hours and lose all of them to the crash they exist to survive. The counters are written as column = column + ? for the same reason — a commit adds what the batch actually did rather than overwriting the row with a stale absolute.

==== Why the Watermark Is a Date, and Why watermark_result_id Is Only a Diagnostic

A pass needs a completeness mark separately from the cursor at all because the cursor columns cannot answer the question: they name a position in a traversal order that is deliberately not id order, so a position in it says nothing about which rows are complete. The cursor resumes an interrupted pass; the watermark is what makes a new pass cheap and non-duplicating — without one a second run would re-read every row, and on a MergeTree with no unique constraint the duplicates could never be removed.

watermark_started_before is that mark. A pass covers start_date >= watermark_started_before AND start_date < cutoff, and on completion sets watermark_started_before = cutoff. The mark and the inclusion predicate are the same column, so the intervals of successive passes tile the timeline with no gap and no overlap; completeness is true by construction rather than by argument. It is a local wall clock because the legacy column it is compared against is one, and nullable because a kind that has never completed a pass has no lower bound — null reads as "start from the beginning".

An id watermark was unsound, and the argument that it was sound is the one this section used to make.

That argument was: only closed day-windows are imported, closed windows lie wholly in the past, and the legacy id is auto-increment in insertion order — so "all ids at or below the watermark are done" must hold.

It does not hold, because the id is global across every campaign and device of the installation while start_date is whenever the measurement was taken, and the two order independently. A device that buffers a result and flushes it late gets a high id with an old start_date; a device reporting in real time gets a lower id with a start date inside a window that is still open. Import the first, write its id as the watermark, and the second is below the watermark forever — above its own cutoff, never importable again, in a store where nothing can be back-filled into a ranked window.

The per-campaign cutoff compounded it: two campaigns whose ug_reactivate fires at different hours had cutoffs up to 24 hours apart, so one campaign’s watermark excluded another campaign’s importable rows. The cutoff is now pass-wide — see the 25-hour lag it costs.

watermark_result_id is kept, and is now only a diagnostic high-water mark. It records the highest legacy id any completed pass has covered, it bounds no read, and its one remaining use is the late-arrival predicate: a row matching start_date < watermark_started_before AND id > watermark_result_id reached ftacs after its day-window had already been imported, and is counted as rows_late_skipped rather than back-filled into a closed and already ranked window (Late Arrivals Are Counted). It only ever moves up: a later pass covers a later date interval whose highest id is not necessarily higher, and letting the mark fall would over-report rows_late_skipped.

The comment inside changelog_create_fcc_legacy_import.xml that calls it the completeness mark describes the superseded design, and says so in place.

== The Settings Resolution Chain

the request's own value  ->  fcc_domain_setting (the test's domain)  ->  global FT Configs
first non-null wins, reading left to right

FccTestService.chain is the one implementation, and resolveConfigDefaults / resolveSpeedDefaults / resolveLatencyDefaults are its three callers.

Why this is a section of its own and not a subsection of [fcc-activation-snapshot].

There are two freezes in this system, at two different moments, for two different reasons, and they are routinely confused:

Freeze When What it protects against

The chain, into fcc_test_config / fcc_test_setting_*

Save time, once

An administrator changing a global or per-domain default between two runs of one campaign

The snapshot, into `fcc_activation’s 30 columns

Every dispatch

An operator editing the campaign after a run happened

Nesting the chain under the snapshot section would put it under a heading about activation, and the single most important fact about it is that it does not run at activation time. The two sections cross-reference each other instead.

=== Which Fields Have Which Levels

Not every field has all three, and the gaps are deliberate.

Field Domain level Global level Note

speed.downloadUrl, speed.uploadUrl

Yes

No

There is no sensible installation-wide default for a URL. A domain with no row and a test with no URL simply stays unactivatable until somebody supplies one, which is what FccTestValidator.validateRunnable reports — and that has not changed and must not.

latency.testHost

Yes

No

Same reasoning. The requirement survives the domain default; see [fcc-domain-setting].

speed.wanDownloadThreshold, speed.wanUploadThreshold, latency.wanUsageThreshold

Yes

No (global = null is passed explicitly)

A cross-talk threshold with no value anywhere is legitimately absent; the WAN sampling then simply produces no CROSS_TALK verdict of its own.

speed.speedSuccessPercent

Yes

Yes — FT Configs, then FccRuntimeDefaults.SPEED_SUCCESS_PERCENT (80)

A policy figure most ISPs share.

latency.latencySuccessMs

Yes

Yes — FT Configs, then FccRuntimeDefaults.LATENCY_SUCCESS_MS (100)

Same.

config.wanSampleGapSec, wanRetryIntervalSec, wanStepTimeoutSec, diagTimeoutSec

No

Yes — FccRuntimeSettings

Timing knobs of the WAN sampling protocol, not an operator’s policy. resolveConfigDefaults walks only two levels.

latency.repetitions, timeoutMs, dataBlockSize

No

Yes — FccRuntimeDefaults (4, 1000 ms, 64 bytes)

Protocol defaults rather than policy: there is no published FT Configs setting for them and nothing ISP-specific about how many times one pings.

speed.tierDl/tierUl, latency.speedTier, connectionsDl/Ul, fileSizeUl, hubbId

No

No

The campaign’s own subject matter. A tier is the subscriber’s provisioned plan; defaulting it would be defaulting the thing being measured against.

testMode is not a chain at all: it is FILE_BASED when absent, full stop.

=== Blank Text Is Absent, Not a Value

chain and chainText differ in exactly one respect, and it is not cosmetic.

chainText treats a blank string as absent. A form that posts "urlDl": "" for a field the operator left empty is the normal shape of an FT UI request, and a strict first-non-null chain would let that empty string beat the domain default — so the per-domain URL would never apply for exactly the caller it exists for, and the campaign would fail validateRunnable later with no indication that a default was available.

Blank is also what validateRunnable already reads as missing, so this keeps the two halves agreeing on what "absent" means.

And "absent" is stored as NULL, never as "". Having decided that a blank is not a value, returning one would write an empty string into a nullable column whose entire vocabulary for "no value" is IS NULL — and every report, every migration and every hand-written query over fcc_test_setting_speed.url_dl would then have to know about a second spelling of nothing. So when neither the request nor the domain has anything, the result is null.

=== It Runs Once, at Save Time, and Never at Activation

This is the whole point of the design and the easiest thing to get wrong.

A campaign with reactivation_days = 7 runs for a week and is activated daily. If the chain were walked on each activation, an administrator who changed a global or per-domain URL on day three would silently make days 1-3 and days 4-7 measure different servers — and nothing in the report would show it: the test row, the activation snapshots and the measurements would all look internally consistent, each describing the configuration that was in force when it was written.

A regulatory filing has to state one set of parameters for the whole campaign. So FccTestService freezes the resolved values into fcc_test_config / fcc_test_setting_speed / fcc_test_setting_latency when the user saves, and each activation additionally snapshots them into fcc_activation ([fcc-activation-snapshot]).

The consequences, stated plainly because they are what an operator will ask about:

  • Editing fcc_domain_setting affects tests saved afterwards and nothing already stored. A campaign that already froze the old URL keeps measuring against it for the rest of its reactivations.

  • Deleting a fcc_domain_setting row makes future tests of that domain fall through to the global defaults, and changes nothing about the campaigns that already froze their values.

  • POST /FCC/item/details reads back the resolved values, not what was sent. A body that omitted speed.downloadUrl comes back carrying the domain’s URL, because that is the stored campaign.

  • PUT /FCC/item/export writes the settings as stored, and re-importing them does not re-resolve the chain. The file carries resolved values, so on import they are explicit request values and win over the target installation’s own defaults — an operator who wants the target domain’s test servers deletes downloadUrl / uploadUrl / testHost from the file before importing, because the chain only fills fields that are absent. See The Settings Chain, and How to Opt Out of It.

  • The audit trail matters more here than for a campaign, not less. A campaign saved after a domain edit looks exactly as though the operator had typed those numbers themselves; the entity_type = 'DOMAIN_SETTING' row in [fcc-audit] is the only record that they did not.

=== There Is No Walk Up the Domain Tree, and There Cannot Honestly Be One

FccDomainSettingResolver.forDomain is an exact domain_id match. A domain with no row of its own does not inherit its parent’s; it falls straight through to the global default.

That is a deliberate limitation rather than an omission, and it is not a limitation this service can lift by trying harder. Two independent facts stand in the way:

  1. The ACS encodes its domain hierarchy inside the dotted isp.name string and has no parent column. There is no parent_id, no closure table, no adjacency list — the shape of the tree lives in a naming convention. The only inheritance available from here would be a guess by string prefix, which breaks the first time a domain name contains a dot for some other reason.

  2. This service is never handed the tree at all. ui-backend sends a flat list of visible domain ids in X-FT-Domain-Ids (Identity Contract) and nothing else — no names, no parents, no ordering that means anything. Even if the ACS had a parent column, this service does not read isp, and adding that read would put the hierarchy of an entire product into a table this service holds a read-only account on and does not own.

And a guess is the wrong kind of wrong here: a wrongly inherited URL is a campaign that measures the wrong server while looking perfectly configured. It produces numbers, they go into a report, and nothing anywhere says which host they came from.

An installation that wants one URL for a subtree writes a row per domain of that subtree. That is more rows, and they are rows somebody can read.

== The fcc_activation Settings Snapshot

The 32-column block on fcc_activation (test_type, test_mode, wan_sample_mode, threshold_dl/ul, tier_dl/ul, url_dl/ul, connections_dl/ul, file_size_ul, time_based_duration_sec, time_based_interval_sec, ping_host, ping_repetitions, ping_timeout_ms, ping_data_block_size, total_duration_h, single_duration_min, batch_interval_min, retry_failed, retry_wan, wan_sample_gap_sec, wan_retry_interval_sec, wan_step_timeout_sec, diag_timeout_sec, speed_success_percent, latency_success_ms, only_online, push, hubb_id — 28 names above, four of which are the dl/ul pairs counted as one each, for 32 columns in the table itself) duplicates fcc_test_config / fcc_test_setting_speed on purpose.

test_mode AND THE TIME_BASED PAIR ARE THE SNAPSHOT COLUMNS THAT SELECT A FORMULA, which makes them load-bearing in a way the rest of the block is not. Everything else here is reported; these three are computed with. FccSpeedStatusCalculator reads test_mode to choose between the file-based rate (total_bytes over bom..eom) and the time-based one (test_bytes_under_full_loading over period_of_full_loading_micros) — see The Pass/Fail Formula. Taking the mode from the CAMPAIGN instead would re-judge a finished run the moment somebody switched the campaign to the other mode, which is the exact failure this whole snapshot exists to prevent, except that it would change a verdict rather than a displayed figure.

An absent test_mode is read as FILE_BASED, because that is what every run dispatched before contract 1.6 actually was.

The test is editable; the run is not. A regulatory report has to show the parameters the ACS actually applied at the time a run happened, not the ones somebody typed into the test afterwards. Reading the settings back off fcc_test at report time would silently rewrite history every time an operator edits a test.

The consequence: editing a test’s settings takes effect only from the next reactivation. FccTestService.resolveConfigDefaults / resolveSpeedDefaults / resolveLatencyDefaults resolve absent values once, at save time — never at activation time — and since M3 they walk the full three-level chain (request, then fcc_domain_setting, then FT Configs) rather than only the global one: see [settings-resolution-chain]. A test with reactivationDays=7 runs for a week and is activated daily; if the defaults were re-read on every activation, an administrator changing a global or per-domain default mid-run would silently make days 1-3 and days 4-7 measure different things. The defaults are frozen into the test when the user saves it, and each activation additionally snapshots them into fcc_activation — two freezes, for two different reasons (the test freeze protects against a default drifting between two runs of one campaign; the activation freeze protects the report against a later edit of the test).

The ping_* and latency_success_ms columns were created in M1, before any latency campaign could exist, precisely so the M2 latency milestone became a code change and not an ALTER TABLE on a table that by then held live activations. They are in use since M2.

=== The Snapshot Is Type-Driven, and the Other Half Is Written NULL

FccActivationMapper.applySnapshot switches on fcc_test_config.type — the CONFIGURATION’s type and not the campaign’s, which no longer exists: a SPEED configuration’s activation gets the fcc_test_setting_speed columns, a LATENCY one gets the fcc_test_setting_latency ones. The half that does not belong to the type is written null, explicitly, rather than left as it was.

This row is what a regulatory report renders, and a leftover URL next to a ping result is a report stating parameters the run never used. Leaving the other half alone would do exactly that on the two paths that re-snapshot an existing row: a test whose type was changed between two runs, and FccActivationHandler.dispatch, which re-takes the snapshot of a row that was planned under the previous type. For a LATENCY run the upload columns (threshold_ul, tier_ul) stay null as well — a ping is symmetric and the contract states FccWanThresholdsDto.ulKbps / FccTiersDto.ulKbps are null for a latency test — and so do test_mode and wan_sample_mode, which describe how a throughput measurement is taken and have no meaning for a ping.

=== A Planned Reactivation Re-Takes Its Snapshot at Dispatch

FccActivationHandler.planReactivation writes a provisional snapshot when it creates tomorrow’s run as PLANNED, so that a planned row is a complete, readable run rather than a stub the UI has to special-case. FccActivationHandler.dispatch then re-takes the snapshot from the current fcc_test_config / fcc_test_setting_* rows at the moment the run is actually sent.

That re-take is what implements the product rule stated above: the run already on the ACS keeps the configuration its command carried — a republish re-sends the stored fcc_acs_outbox.payload and never re-runs the mapper — while the next one picks up whatever the operator changed in the meantime. Snapshotting once at planning time instead would freeze a configuration hours or days before it is used, and an operator who fixed a wrong URL would watch the wrong one run anyway.

command_seq is the one column planReactivation deliberately leaves unset, storing the placeholder 0 until dispatch assigns the real sequence in the same transaction as the outbox row. The sequence is consumed by the command, not by the activation row: the ACS discards a command whose sequence is not greater than the last one it accepted, so a reactivation planned today and sent tomorrow would otherwise burn a sequence now and make every command written in between look stale. activation_count is not bumped by planReactivation either — it counts the activations an operator asked for, which is what freezes the device set after the first one, and a reactivation the service planned by itself is not a new decision by anybody.

== The iotw Grant: Two Tables, SELECT Only

configs[].sourceType=Condition names a saved view of the portal’s device list, and that data lives in `ui-backend’s own schema. This service reads two tables there and nothing else:

GRANT SELECT ON iotw.iotw_view               TO ftfcc_iotw_ro;
GRANT SELECT ON iotw.iotw_view_column_condition TO ftfcc_iotw_ro;
Table What is read from it, and why only that

iotw_view

id, name, domain_id, view_type. The name is what the details read answers as configs[].conditionName; the domain_id says whose condition it is, and decides visibility. The portal’s own presentation columns — client_type, default_user, default_domain, width, sort_index — are deliberately not read: this service renders no device list and has no business carrying the portal’s presentation state.

iotw_view_column_condition

id, view_id, parent_id, column_key, type, string_value, date_value, logic — the whole criteria tree, fetched flat in one statement. Nothing here is interpreted by the DAO: whether a column_key or a type can be honoured is `FccConditionTranslator’s judgement (the subset it translates), because a DAO that silently dropped a node it did not recognise would be the one way a condition could resolve to a different fleet without anybody being told.

Four properties of this grant are worth stating, because each is a decision rather than a default:

  • No writes, ever. The pool is opened read-only in code (DataSourcesConfig.iotwDataSource) as well as granted read-only. These are the portal’s views, and this is a borrowed read, not a shared model.

  • No transaction manager. Reading a condition is one or two statements with no invariant between them, and a second manager would tempt somebody to join this read into an ftacs transaction — which no two-phase-less pair of datasources can honour.

  • Lazy. An installation with no grant, or with ft.fcc.iotw.enabled=false, starts normally and refuses only that one device source type. The startup check (ft.fcc.iotw.schema-check.enabled) reports a missing grant with the full list of missing columns and warns rather than failing.

  • On Oracle the two tables belong to their own user, so the datasource issues ALTER SESSION SET CURRENT_SCHEMA — without it every unqualified FROM iotw_view would resolve against the connecting account’s own empty schema and answer "table or view does not exist".

== ClickHouse Schema

=== fcc_result_speed

One row per speed diagnostic iteration and direction, in the shared database ftacs_qoe_ui_data (QoE already owns diag_speed_test and diag_ip_ping there — hence the fcc_ prefix and the separate Flyway history, see [flyway-history]).

Column Type Meaning

fcc_test_id, activation_id

Int64

iteration

Int32

domain_id

Int32

Taken from FccResultDto.domainId (added in wire contract 1.3), falling back to the fcc_activation snapshot when the wire value is null or not strictly positive — 0 is not a domain, and is also the sentinel written here when neither source resolves one. hubb_id uses a null-only test, for the reason given in Where domain_id and hubb_id Come From.

serial

LowCardinality(String)

Always present: FccDeviceDto.serial is mandatory on the command, enforced before a command is ever built — see A Device Without a Serial Is Refused — Twice.

hubb_id

LowCardinality(String)

Legacy maps 1:1. Same provenance as domain_id: FccResultDto.hubbId first, the activation snapshot as the fallback — but only a null on the wire falls back, because an empty string is a campaign that legitimately has no hubb id.

direction

LowCardinality(String)

DOWNLOAD | UPLOAD (FccDirectionEnum also has PING, used by fcc_result_latency).

url

String

Not LowCardinality — see [lowcardinality-rule].

tier_kbps

Int32

Not LowCardinality — it is numeric, see [lowcardinality-rule].

started_at

DateTime64(3)

finished_at

Nullable(DateTime64(3))

Nullable because legacy latency rows carry no end date; kept nullable here too for a uniform ETL path across both result tables.

status

LowCardinality(String)

SUCCESS | FAILED | CROSS_TALK | OTHER, computed by this service — see The Pass/Fail Formula. The ACS never publishes a verdict.

description

String

wan_usage_dl_kbps, wan_usage_ul_kbps

Int32

after_stop

UInt8

The measurement arrived once the run had been told to stop. Two sources since M2: the wire flag FccResultDto.afterStop wins when it is set — the ACS knows when it issued the diagnostic — and a local comparison of the arrival instant against fcc_activation.stopped_at is the backstop for an ACS that does not set it at all. Without that backstop the flag would be the only source, and an older ACS would have every one of its after-stop results silently indistinguishable from the rest. See The After-Stop Window for the second, later boundary that parks a measurement instead of storing it.

source

LowCardinality(String)

ACS_STREAM | LEGACY_ETL — see [two-source-dictionaries].

legacy_ug_id, legacy_result_id

Nullable(Int64)

Populated only by the M4 ETL.

created

DateTime64(3)

Partitioning key (toDate(created)).

bom, eom

Nullable(DateTime64(3))

Beginning/end of measurement.

total_bytes, test_bytes

Nullable(Int64)

total_bytes is nullable because the legacy path truncated it through intValue() and could go negative; the ETL maps those to NULL rather than storing a lie. A stored 0 is not a slow transfer, it is no transfer, and the verdict says so — null and any non-positive count alike answer OTHER rather than FAILED, see the speed formula.

connections

Int32

speed_kbps

Float64

Computed by this service; see The Pass/Fail Formula for the exact expression and why it must never use integer division.

Engine: MergeTree, PARTITION BY toDate(created), ORDER BY (fcc_test_id, serial, direction, created), TTL created + INTERVAL 400 DAY.

A TTL clause has not been exercised anywhere in FT before this table. It needs to be verified on ClickHouse 24.8, together with the daily partitioning, that parts are actually dropped at 400 days and that the background TTL merge does not fight the daily partition boundaries.

LowCardinality is applied only to the string columns that have a small, repeating value set: serial, status, source, direction, hubb_id. It is deliberately not applied to tier_kbps: that column is numeric, and wrapping a number in LowCardinality buys nothing while adding a dictionary per part.

The column list is treated as final for M1: everything the M4 legacy ETL will need was added now, on purpose, because an ALTER TABLE …​ ADD COLUMN over tens of millions of rows while the ETL is running is exactly the operation this design avoids discovering it needs later.

The natural de-duplication key is (activation_id, serial, direction, iteration). It is not enforced by MergeTree (no unique constraint exists in ClickHouse), so readers — and FccResultIngestService, implemented and code-read, via one atomic Hazelcast IMap.putIfAbsent per result, see De-Duplication Is One Atomic putIfAbsent — de-duplicate on it themselves.

=== fcc_result_latency

One row per latency (IPPing) diagnostic iteration, created by the Flyway migration V2create-fcc-result-latency-table.sql in the same shared database ftacs_qoe_ui_data, with two columns added by V3add-ping-actuals-to-fcc-result-latency.sql under contract 1.5 — see [contract-1-5-ping-actuals].

A second table rather than more Nullable columns on fcc_result_speed: the two diagnostics share nothing below the common block, a MergeTree part would carry a dictionary and a null map for every column the other diagnostic owns, and the ORDER BY that serves a speed report is not the one that serves a latency report.

The common block is identical to fcc_result_speed in name, order and type, down to the nullability. That is a hard rule, not a coincidence. Every cross-diagnostic read is a UNION ALL of the two tables, and ClickHouse matches a UNION ALL by position — so a column reordered here would not fail, it would silently return a serial in the url column of half the report.

The first twenty columns are therefore exactly those of fcc_result_speed, with the same provenance and the same fallbacks (FccResultIngestService.toLatencyRow fills them from the same helpers toSpeedRow uses). Two of them read oddly for a ping and are kept anyway:

Column Note

direction

Always PING here. Kept — and kept in the ORDER BY — so the two tables stay union-compatible and a reader never has to special-case one of them.

url, tier_kbps

A ping has neither. The columns are non-Nullable and union-compatible with the speed table, so the honest answer is the same "absent" the speed side already writes (empty string, 0), not a special case.

The columns that are the latency half — five from V2, plus the two V3 adds for contract 1.5:

Column Type Meaning

val_avg, val_min, val_max

Nullable(Int32)

Round trip in milliseconds, from FccResultDto.avgMs / minMs / maxMs, passed through unchanged. Nullable on purpose: a ping that lost every packet, and every row the legacy ETL imports from a source that stored only an average, genuinely carry no value. Storing 0 there would be a lie that no reader could tell apart from a sub-millisecond round trip.

packets_received

Int32

FccResultDto.successCount.

packets_sent

Int32

successCount + failureCount, reconstructed from the two counters — never read from the configured repetition count. TR-181 IPPing guarantees SuccessCount + FailureCount = NumberOfRepetitions; the configured value is what was asked for, the sum is what the device reports having done, and a run that ended early must not claim otherwise. An absent counter contributes 0 to that arithmetic, because unlike the val_* columns these two are non-Nullable — a count of zero is a true statement. This does not change with contract 1.5’s two new columns below: repetitions sitting right next to packets_sent is exactly what a future reader will be tempted to "simplify" this arithmetic into — it must not be. See [contract-1-5-ping-actuals].

repetitions

Nullable(Int32)

Contract 1.5. FccResultDto.pingRepetitions — the IPPing.NumberOfRepetitions the device actually ran the diagnostic with, as reported back per iteration, next to packets_sent which is what it reports having done. Falls back to the activation snapshot (fcc_activation.ping_repetitions) when the wire value is null; stays NULL when neither source has one. See [contract-1-5-ping-actuals].

data_block_size

Nullable(Int32)

Contract 1.5. FccResultDto.pingDataBlockSize — the IPPing.DataBlockSize the device actually used. Same fallback as repetitions, to fcc_activation.ping_data_block_size. See [contract-1-5-ping-actuals].

Why repetitions and data_block_size are Nullable(Int32) and not Int32 DEFAULT 0. A 0 for either would be a statement a reader would have to believe — zero repetitions, a zero-byte packet — and every row written before this migration, plus every row an ACS older than contract 1.5 will ever produce, genuinely has no value to report. This is the same argument val_avg / val_min / val_max already carry, above. A defaulted figure is also deliberately not substituted from FccRuntimeDefaults when neither the wire nor the snapshot has one: a defaulted value would be indistinguishable from a reported one in a compliance export, and the column must be able to say "unknown" rather than guess.

Why the ALTER TABLE that adds them is safe, despite fcc_result_latency’s own header (`V2create-fcc-result-latency-table.sql) declaring its column list final precisely to avoid an ADD COLUMN over tens of millions of rows. A ClickHouse ADD COLUMN of a Nullable column with no DEFAULT expression is metadata only: no existing part is rewritten, the column materialises lazily on the next merge and reads back NULL from every existing part until then. The operation V2’s header feared is a column with a DEFAULT or MATERIALIZED expression, which does force a rewrite — this migration deliberately has neither. (V2’s own comment could not be corrected in place to say so, because Flyway checksums the whole migration file including comments; the correction lives here and in `V3add-ping-actuals-to-fcc-result-latency.sql’s header instead.)

Union-compatibility with fcc_result_speed is preserved. The two columns are appended at the very end, past the latency-only block that already differs from the speed table, and are not part of the ORDER BY, the PARTITION BY or the TTL. The common block fcc_test_id …​ created ([fcc-result-speed]) is untouched, so no positional UNION ALL read moves — see the hard rule at the top of this section. fcc_result_speed gets no equivalent columns: they are latency-only, a DOWNLOAD / UPLOAD iteration has no IPPing parameters to report.

Parked dead-letter rows are unaffected: fcc_result_dead_letter.payload is Jackson JSON, so a payload written before contract 1.5 deserialises with both new fields null and falls back to the snapshot on resend exactly as a live 1.5 message would.

Engine, partitioning, ordering and TTL are the same as fcc_result_speed: MergeTree, PARTITION BY toDate(created), ORDER BY (fcc_test_id, serial, direction, created), TTL created + INTERVAL 400 DAY, index_granularity = 8192. The LowCardinality rule of [lowcardinality-rule] applies unchanged, and the natural de-duplication key is again (activation_id, serial, direction, iteration), again unenforced. The same TTL caveat applies — see the warning under [fcc-result-speed].

status is computed by this service, from ping_ok = avg_ms < latencySuccessMs && success_count > 0, and stored in the same SUCCESS \| FAILED \| CROSS_TALK \| OTHER dictionary the speed table uses. The ACS publishes raw measurements only and never a verdict. See The Pass/Fail Formula for the resolution order, and in particular for why success_count == 0 is FAILED and not OTHER.

The column list was treated as final for the M4 legacy ETL for the same reason fcc_result_speed’s is: everything that ETL will need (`legacy_ug_id, legacy_result_id, source) was created up front, because an ALTER TABLE …​ ADD COLUMN over tens of millions of rows while the ETL is running is exactly the operation this design avoids discovering it needs later. Contract 1.5’s repetitions / data_block_size ([contract-1-5-ping-actuals]) are the one exception, and are not a reversal of that rule: they are Nullable with no DEFAULT, so the ADD COLUMN that created them was metadata-only, not the rewrite the rule above was written to avoid. The ETL columns are still not to be extended casually; a Nullable addition with no default is the narrow exception this migration relied on, not a precedent for widening the table freely.

=== The Report Read Path

FccResultQueryDao is the read side of this schema, behind the four M4 reporting endpoints (Endpoints Implemented in M4). It never runs ahead of the write side: every column it selects is one fcc_result_speed / fcc_result_latency already carries, and every predicate it builds comes from the one FccResultFilter all four endpoints resolve through FccReportScopeResolver — which is what stops the run list, the measurement list, the report header and the export from ever disagreeing about which rows a filter means.

==== The UNION ALL Is Generated, Never Hand-Written Twice

ClickHouse matches a UNION ALL by position, not by name — the hard rule [fcc-result-latency] already states for the common block. The read side takes that one step further: the two branches of every cross-diagnostic query are not two pieces of SQL, they are two renderings of one Java list (FccResultQueryDao.COLUMNS), and every column is CAST to the same type on both sides. A column the branch does not own becomes CAST(NULL AS <the same type>) at the same position, which is what makes the two renderings union-compatible by construction rather than by review.

That is also why connections, speed_kbps, packets_sent and packets_received are projected Nullable in a report row although [fcc-result-speed] and [fcc-result-latency] declare them non-Nullable in their own tables: a speed row has nothing to offer packets_sent, and the honest way to say so in a positional union is NULL, not a fabricated 0 a reader could mistake for a real count.

==== The Period Predicate Is Two Predicates

Both tables are PARTITION BY toDate(created) while a report filters on started_at — the instant the device measured, not the instant the row was written. The two differ by the ACS batching, or, for a row the legacy ETL imports, by years. Narrowing on created instead of started_at would therefore silently drop measurements; narrowing on started_at alone forces ClickHouse to read every partition, because the engine has no way to relate the two columns to each other.

The read adds a second, widened predicate on created beside the one on started_at — one day either side of the requested window (FccResultQueryDao.PARTITION_MARGIN_DAYS). It cannot exclude a row the started_at bound keeps, because nothing is written a day before it is measured, and it is what lets ClickHouse prune whole partitions instead of scanning the table. FccReportActivationSpecifications.byFilter applies the plain started_at bound to fcc_activation for the run list, on the relational side, where there is no partition to prune and a second predicate would add nothing — see fcc_activation for the index it now runs against.

==== Model and Manufacturer Are Resolved Through the Frozen Fleet

Neither result table carries a model, a manufacturer or a cpe_id column, and none is being added: V1 and V2 declare their column lists final ([lowcardinality-rule]), and an ALTER TABLE over tens of millions of rows is exactly the operation those comments exist to avoid. FccReportScopeResolver resolves a model or manufacturer filter in three steps instead — manufacturer to model ids through the ACS catalog, model ids to campaigns and serials through fcc_test_device (the frozen snapshot), and those into an fcc_test_id IN (…​) predicate on the result rows, narrowed further to a serial IN (…​) only when the model does not already cover every device of the selected campaigns.

The snapshot, and never the live ftacs.cpe. It describes the fleet a run actually executed against; a CPE re-provisioned into another model since then must not change what a finished run reports — the same argument that put protocol into the snapshot in M3 (Why a Snapshot Carries the Protocol).

The legacy limitation, stated here because it cannot be fixed here. A model or manufacturer filter reaches a LEGACY_ETL row only where the legacy ETL wrote a fcc_test_device fleet snapshot for its anchor campaign. If it does not, step two finds no campaign for that model and the imported half of the report is not there at all, not partially.

There is deliberately no fallback that looks the serial up in the live ftacs.cpe: that would answer with the model the device carries today, which for a measurement taken years ago is not an approximation, it is a different fact. Populating fcc_test_device for every anchor campaign is a requirement on the M4 legacy ETL, not a gap this read path can close on its own.

The ETL now meets that requirement, with one honest limitation it cannot remove. It snapshots a fleet for every anchor it creates, from the serials the measurements themselves carry — but the legacy rows carry only a serial, so the model_id it records is the model the CPE has at import time, and a serial no live cpe row matches is not snapshotted at all. See The Fleet Snapshot Is Mandatory and model_id Is the Model at IMPORT Time.

=== Two Different source Dictionaries — Do Not Merge Them

There are two, unrelated source columns in this system, and they must never be treated as one enum:

Column Values

fcc_test.source / fcc_activation.source (relational, ftfcc)

ACS_STREAM | LEGACY — classifies a campaign: whether it is a live test streamed from the ACS or a synthetic anchor for an imported legacy test.

fcc_result_speed.source / fcc_result_latency.source (ClickHouse)

ACS_STREAM | LEGACY_ETL — classifies a result row: whether it was streamed live or produced by the legacy ETL. This is the column the reports filter source parameter refers to — see Endpoints Implemented in M4.

The Java-side enums mirror the split exactly: FccTestSourceEnum (ACS_STREAM, LEGACY) versus FccResultSourceEnum (ACS_STREAM, LEGACY_ETL) are two distinct types.

=== What a LEGACY_ETL Row Carries, and What It Cannot

An imported row fills the same columns a live one does, but several of them have no source in the legacy schema at all, and how each says so depends on whether the column is Nullable.

Column Value on a LEGACY_ETL row Why

connections, wan_usage_dl_kbps, wan_usage_ul_kbps, after_stop

0

The columns are not Nullable, and zero is the only value available. after_stop = 0 is additionally simply true: an imported run was never stopped, so no measurement of it can have arrived after a STOP.

test_bytes

NULL

Nullable(Int64), so the row can say "unknown" rather than "zero bytes" — the same argument total_bytes already carries in [fcc-result-speed].

repetitions, data_block_size (latency only)

NULL

ug_fcc_latency_result reports neither, and a source that does not report a figure must stay distinguishable from one that reported zero — see [contract-1-5-ping-actuals].

total_bytes

The stored value, or NULL

NULL when the legacy counter overflowed: negative (the Oracle signature) or zero alongside a positive speed (the MySQL signature, where the column is INT UNSIGNED and a negative clamps to 0). Tallied as bytes_overflow_count — see total_bytes: Two Signatures of One Overflow.

url, tier_kbps on a latency row

Filled from the legacy columns

Unlike a live latency row, which writes "" / 0 because an IPPing iteration has neither ([fcc-result-latency]). ug_fcc_latency_result carries both a url and a tier, so they are carried across rather than blanked.

started_at, finished_at

Whole seconds, always

The legacy sub-second is a random draw the ACS stamped on, not a measurement, so it is dropped and nothing is put in its place. That is what makes two imports of one source row byte-comparable, and it is why an imported row’s finished_at can never precede its started_at. bom / eom are the exception and keep their stored milliseconds — see bom and eom Are the Only Genuine Device Timestamps.

domain_id

update_group.location_id

0 is the super domain, not "unknown" and not "no domain" — the platform convention CommonUtils.isSuperDomain states (null, 0, -1 all mean it). It is what update_group.location_id (NOT NULL DEFAULT 0) holds for a super-domain group, and what this service writes for a test a super-domain caller files without naming a domain. 0 is in nobody’s visible-domain list, so such rows are readable only by a caller with X-FT-Domain-Scope: ALL — the operator who owns them. See legacy-import.adoc#domain-zero.

created

When the import wrote the row

Not when the measurement was taken. It is the partitioning key, which is exactly why [report-period-predicate] narrows on started_at and on a widened created — for an imported row the two differ by years.

=== Flyway History Table

spring.flyway.enabled is false: Spring Boot’s own Flyway autoconfiguration never runs, and spring.flyway.table is never set at all. ClickHouseFlywayConfig builds a Flyway instance by hand instead — the community ClickHouse dialect stops at 10.24.0 while Boot manages a newer core — and it is that instance whose history table defaults to flyway_schema_history_fcc, never the Flyway default flyway_schema_history. ACS, ui-backend and ft-device-network all keep their own Flyway histories in this same shared ClickHouse database; a shared default history table name has already caused one incident (see ClickHouseFlywayConfig). The history table name is read from a property (ft.fcc.clickhouse.flyway.table) rather than hardcoded, unlike the raw-SQL self-healing block found in some sibling repos' FlywayConfig classes.

== Entity Diagram

fcc-database-entities