Database
ftfcc Schema
ftfcc is this service’s own schema: read-write, owned and migrated exclusively by this service’s Liquibase (db.changelog/ftfcc/db.changelog-master.yaml).
M1 ships six tables, M2 adds four more — fcc_test_setting_latency, fcc_device_state, fcc_audit, fcc_result_dead_letter — plus two indexes on the existing fcc_activation, and M3 adds two more again: [fcc-model-capability] and [fcc-domain-setting], plus one column on the existing fcc_test_device.
That is twelve tables, and M4 adds the thirteenth: [fcc-legacy-import], the cursor of the legacy ETL.
Every M2 addition is a separate changelog file appended to the master under an explicit # ===== M2 ===== marker, and the two fcc_activation indexes sit in changelog_add_fcc_activation_indexes_m2.xml rather than in the table changelog: that file has already run at every installation and its changesets must keep their checksums.
M3 follows the identical convention under a # ===== M3 ===== marker, and fcc_test_device.protocol is added by changelog_add_fcc_test_device_protocol.xml for exactly that reason — an executed changeset is never edited, because its checksum is what tells a live installation that nothing has moved under it.
Schema-Wide Rules
These rules are visible in every changelog and are not incidental:
| Rule | Why |
|---|---|
Zero foreign keys |
Integrity is app-enforced (FT house convention). A no-FK schema also makes the M4 legacy ETL and any future re-partitioning simpler: nothing to drop and recreate around a bulk load. |
|
An ordinal breaks silently when a value is inserted in the middle of the Java enum (existing rows now mean something else) and is unreadable in a |
|
The legacy schema used |
Audit columns ( |
Filled by Spring Data JPA auditing ( |
Paired |
Every table changelog is written only in terms of the |
Deliberate, named indexes |
Every index in the schema exists to serve one named query (a poller, the watchdog, a specific screen). The comment above each |
fcc_test
The user-authored campaign. It is not a run: every concrete execution is a row in fcc_activation, and a test with reactivation_days > 0 produces many of them over time.
| Column | Type | Meaning |
|---|---|---|
|
BIGINT PK |
|
|
INT NOT NULL |
The |
|
VARCHAR(255) NOT NULL |
|
|
VARCHAR(32) |
IT REPLACED |
|
VARCHAR(32) NOT NULL |
|
|
VARCHAR(32) NOT NULL, default |
|
|
INT |
Legacy |
|
DATETIME(3) |
|
|
INT NOT NULL, default 0 |
A COUNT OF RUNS, and never a duration — |
|
INT, default 1 |
Days between two runs — |
|
DATETIME(3) |
The ANCHOR the slots are counted from — |
|
BIGINT NOT NULL, default 0 |
Monotonic per-test command counter, incremented in the same transaction as the |
|
INT NOT NULL, default 0 |
Number of activations created so far. |
|
BIGINT NOT NULL, default 0 |
Backs the JPA |
Indexes: fcc_test(domain_id, state) serves POST /FCC/items, which is always narrowed to the caller’s domain and in practice almost always also to a state ("show me what is running"). fcc_test(state, scheduled_at) serves the SCHEDULED / SCHEDULED_FOR_REACTIVATION poller, which scans across all domains every poll cycle and must not scan the whole table.
fcc_test_config
ONE CONFIGURATION OF A CAMPAIGN: which diagnostic to run, on which devices, with which timing and retry knobs. A campaign holds SEVERAL of these — one per element of configs[] on PUT /FCC/item/details — and each one has its own settings row, its own frozen fleet and its own ACS activation on every run.
It is Group Update’s update_group_child, and that correspondence is the whole point of the level:
| Group Update | Here | What it holds |
|---|---|---|
|
|
The campaign: name, domain, state, schedule, recurrence rule |
|
|
The CONFIGURATION: what to measure, on which devices, with which knobs |
|
|
The per-configuration parameters |
the child’s device selection |
|
The per-configuration frozen fleet |
MIXED KINDS IN ONE CAMPAIGN ARE ALLOWED — a speed configuration beside a latency one, exactly as one legacy update_group carries a 103 task and a 104 task. That is what removed fcc_test.type: a campaign no longer HAS one type, and a column answering the question anyway would have to pick a configuration and call its type the campaign’s.
The timing columns themselves map to FccScheduleDto + FccRetryPolicyDto in the command, and are shared by both kinds: a latency configuration has the same window, the same iteration length and the same WAN sampling policy as a speed one. The contract states them ONCE per campaign, in activation, so the save writes them identically to every configuration row of a campaign and the details read answers them from the first.
| Column | Type | Meaning |
|---|---|---|
|
BIGINT NOT NULL |
Plain id column, no FK (see Schema-Wide Rules). The campaign this configuration belongs to. |
|
INT NOT NULL, default 1 |
The 1-based POSITION in |
|
VARCHAR(32) NOT NULL |
|
|
TINYINT(1) |
Skip devices the ACS reports as offline instead of queueing the diagnostic. |
|
TINYINT(1) |
Use a connection request instead of waiting for the next Inform. |
|
INT |
|
|
INT |
|
|
INT |
Rest of |
|
VARCHAR(255) |
WHAT THE OPERATOR CHOSE, beside the fleet it resolved to — DEV-1381’s |
|
VARCHAR(32) |
|
|
INT |
The id of a saved condition in the |
|
|
Index: fcc_test_config(test_id, seq) UNIQUE serves SELECT * FROM fcc_test_config WHERE test_id = ? ORDER BY seq (test detail, activation snapshot build) and enforces "one configuration per (campaign, position)" in the database rather than only in the service — a campaign cannot hold two configurations claiming the same slot in configs[]. It replaced fcc_test_config(test_id) UNIQUE, which said "one configuration per campaign" and was the schema half of a limit the service no longer has.
fcc_test_setting_speed
The SPEED parameters of ONE CONFIGURATION (FccSpeedTargetsDto, FccTiersDto, FccWanThresholdsDto) — one row per fcc_test_config whose type is SPEED. LATENCY has its own sibling table, fcc_test_setting_latency — nothing here is shared with it, which is why the two are not squeezed into one wide table.
It hangs off the CONFIGURATION and not off the campaign, which is what lets one campaign carry two speed configurations measuring different URLs or different tiers, and a speed configuration beside a latency one. The row’s owner is config_id and there is deliberately no campaign pointer beside it — see the config_id row of fcc_test_setting_latency for the reasoning, which is the same here.
| Column | Type | Meaning |
|---|---|---|
|
BIGINT NOT NULL |
The fcc_test_config these parameters belong to; the unique index enforces one row per configuration. |
|
INT |
TR-143 |
|
INT |
TR-143 |
|
INT |
WAN cross-talk thresholds in kbps. Above these the WAN is considered busy and the iteration is recorded as |
|
INT |
The subscriber’s provisioned tier in kbps; |
|
VARCHAR(512) |
Two columns, not one: |
|
INT |
Two columns for the same reason as the URLs: |
|
BIGINT |
|
|
VARCHAR(32), default |
|
|
VARCHAR(32), default |
|
|
VARCHAR(16) |
Where the tier came from — the constant name of |
|
INT |
Percentage of the tier that still counts as a pass (default 80, see Configuration Reference). |
|
VARCHAR(64) |
Operator-side correlation id, carried through to the result rows. |
Index: fcc_test_setting_speed(test_id) UNIQUE, same reasoning as fcc_test_config.
fcc_test_setting_latency
The LATENCY-specific half of a test, the exact sibling of fcc_test_setting_speed.
Two narrow tables and not one wide one, for the reason the speed table already gives: nothing here is shared with it.
A LATENCY test has no URL, no connection count and no upload file size; a SPEED test has no ping host, no repetition count and no per-packet timeout.
Merged into one table every row would be half NULL, and the "which columns must be present" rule would live only in Java.
Four columns map one to one onto the TR-181 IPPing object the ACS drives:
Device.IP.Diagnostics.IPPing.Host
Device.IP.Diagnostics.IPPing.NumberOfRepetitions
Device.IP.Diagnostics.IPPing.Timeout
Device.IP.Diagnostics.IPPing.DataBlockSize
| Column | Type | Meaning |
|---|---|---|
|
BIGINT NOT NULL |
Plain id column, no FK. The fcc_test_config these parameters belong to; the unique index enforces one row per configuration. There is no campaign pointer on the row on purpose: a settings row is only ever reached through its configuration, so a second pointer at the campaign would be an unread column that could disagree with the one that is read. |
|
INT |
WAN cross-talk threshold in kbps: above it the WAN is busy and the iteration is recorded as |
|
INT |
The subscriber’s provisioned downstream tier in kbps, carried through to the result rows so a report can state what the latency was measured against. Same single-direction reasoning as |
|
VARCHAR(255) |
|
|
INT |
|
|
INT |
|
|
INT |
|
|
VARCHAR(16) |
Where the tier came from — the constant name of |
|
INT |
The average round trip that still counts as a pass. Not a TR-181 parameter — it is the pass/fail bound this service applies, because the ACS publishes raw measurements and never a verdict, so the criterion has to be stored next to the test that defined it rather than derived at read time from whatever the default happens to be then. Falls back to |
|
VARCHAR(64) |
Operator-side correlation id, carried through to the result rows. |
threshold_dl and tier_dl keep the SPEED spelling on purpose: a latency run still samples the WAN for cross-talk and still reports against the subscriber’s provisioned tier, and using the same column names keeps the fcc_activation snapshot a straight copy.
Index: fcc_test_setting_latency(test_id) UNIQUE, same reasoning as fcc_test_config — it serves the test-detail read, the delete-by-test of an edit and the re-take of the activation snapshot at dispatch time, and enforces the one-row rule in the database rather than only in the service.
|
One type, one settings row.
|
fcc_test_device
The frozen device set of a test — a snapshot, not a live view. serial, model_id, domain_id and — since M3 — protocol are resolved once from ftacs.cpe when the test is saved and are never refreshed: ftacs is read-only and may be a different database instance, so a join at read time is not available, and more importantly a regulatory report has to show the fleet the test actually ran against, not the fleet as it looks today.
How a selection becomes these rows is the last section of Device Resolution; this section is what the rows hold.
| Column | Type | Meaning |
|---|---|---|
|
BIGINT NOT NULL |
The campaign the device was frozen for. It is kept beside |
|
BIGINT NOT NULL |
The fcc_test_config whose fleet this device is in, and the row’s real owner: the fleet belongs to the configuration, and one ACS activation carries one configuration’s devices and no others. |
|
INT NOT NULL |
|
|
VARCHAR(64) |
Required in practice, enforced by the validator rather than by a column constraint: |
|
INT |
|
|
INT |
Copied from the |
|
VARCHAR(32) |
M3, |
Index: fcc_test_device(config_id, cpe_id) UNIQUE does two jobs at once — it rejects a duplicate device in the same CONFIGURATION at insert time (a fleet is assembled from overlapping groups and up to ft.fcc.max-devices-per-test rows can be inserted in one batch, so de-duplicating only in memory is not enough), and it serves the read that builds the device list of every ACTIVATE command.
It replaced fcc_test_device(test_id, cpe_id) UNIQUE, and that is the visible consequence of allowing overlapping fleets. The old index was the statement "a device appears once in a campaign", which is no longer true: a device named by two configurations of one campaign is measured by both, and it then sits in two concurrent ACS activations — which fcc_sa_activation permits, being unique on (activation_id, cpe_id), and which is the operator’s decision to make rather than this schema’s. The replacement says what is still true: a device appears once in a CONFIGURATION. A campaign-level count therefore counts a shared device once per configuration, which is how many measurements of it the run actually performs.
fcc_test_device(test_id, cpe_id) is re-created NON-UNIQUE in the same changeset, because eight reads ask this table a question about the CAMPAIGN — how many devices a campaign would touch (the activation impact dialog), which campaigns hold a device of this model, which serials a report scope covers — and none of them is about configurations. test_id and config_id are written by the one statement that inserts the fleet, from the configuration they belong to, so the pair cannot drift.
Once fcc_test.activation_count > 0, the set is immutable: FccTestValidator.validateDeviceSetEditable throws FccConflictException (HTTP 409) if the submitted fleet differs from the stored one. The comparison is PER CONFIGURATION, and the SET OF CONFIGURATIONS is frozen with it. Each configuration is its own activation measuring its own fleet, so moving a device from the first configuration to the second changes what both of them measured without changing the campaign’s device list at all — which a campaign-wide comparison would have accepted; and adding or removing a configuration is the same thing in a different shape, because a run would then dispatch a number of activations the finished runs did not have. This is a product rule, not a technicality — see Endpoints Implemented in M1 for the exact response, and PUT /FCC/item/export + PUT /FCC/item/import for the escape hatch it points at: the export writes the campaign to a file and the import answers it as a new unsaved one, whose fleet the operator changes before saving.
Since M3 the comparison is made after resolution, on the resulting set of cpeId`s, so a `configs[] selection whose manufacturer, model or conditionId was reworded but which resolves to exactly the stored devices is a legal no-op edit. What a compliance report has to describe is the devices that were measured, and neither the wording of a condition nor the order of a list changes those.
|
The set of models is frozen with the devices, and there is deliberately no A model is |
Why a Snapshot Carries the Protocol
protocol is not a display field. The fleet is frozen precisely so that a regulatory report describes the devices the campaign actually ran against, and the protocol is part of that description.
The ACS of v1 answers SKIPPED(UNSUPPORTED) for a device it cannot drive (USP and Other Non-CWMP Devices Are Included and Flagged), so "how many of these devices were never going to produce a measurement at all" is a question asked of a finished run — weeks later, when a report is assembled and a device count does not match a measurement count.
At that point ftacs.cpe may perfectly well say something different, because the CPE was re-provisioned, or migrated from TR-098 to TR-181, or removed.
Without this column the only way to answer the question is to re-resolve the fleet against a live table, which is exactly what a snapshot exists to avoid.
Three properties of the column are load-bearing:
-
It holds the constant NAME, never an ordinal, like every other enum-valued column in
ftfcc(seeFccDbConstants). The ACS’s ownCpeProtocolEnumis persisted by ordinal on its side, which is a recorded known issue there — inserting a constant ahead ofLWM2Msilently re-maps every stored device. A name cannot do that. -
It is a plain
Stringon the entity and not@Enumerated, unlike the enum columns offcc_test. Those hold this service’s own vocabulary; this one holds a foreign one. A constant that a later build of this service no longer declares must still be readable in a report years from now, and a mapped enum would instead fail the whole read of that campaign’s fleet on an unknown name. The protocol is already a name on the wire (FccResolvedDeviceItem.protocol), so nothing downstream needs it typed. -
The mapping from
cpe.protocol_idhappens in exactly one place,FccTestDeviceDao.bind, throughFccCpeProtocolEnum.fromProtocolId. The raw id travels out of the ACS schema untouched — a null there means TR-069 rather than "unknown", and only that method knows it — so the coercion is applied on the way into the snapshot and never repeated by a caller.
Nullable, and never back-filled. A fleet frozen before this column existed has no protocol, and filling it in from a live table would be a snapshot rewritten with today’s facts, which is the one thing this table must never contain.
Two Indexes for the M4 Report Filters
changelog_add_fcc_test_device_model_index.xml (M4) adds two more, in a changelog of its own so changelog_create_fcc_test_device.xml keeps its checksums.
| Index | Query it serves |
|---|---|
|
The model chain of |
|
The device filter of the same resolver: |
Neither predicate is served by the existing (test_id, cpe_id) UNIQUE index above: a composite index can only be used from its leading column, and both new reads start from model_id or from cpe_id, never from test_id.
fcc_activation
ONE ACTIVATION IS ONE CONFIGURATION OF ONE RUN. A campaign with two configurations and reactivation_days = 7 produces fourteen of these over its lifetime: seven runs, two activations each.
A RUN IS (test_id, seq) AND NOT A ROW. The ACS command carries one configuration and one device list, so a campaign of N configurations is N commands — each with its own idempotency_key and its own command_seq, which is what keeps the ACS from taking the second for a retry of the first and what stops it discarding all but one as stale. They share the seq, because the RUN is one slot of the recurrence rule: two configurations must not consume two of the campaign’s daily runs. fcc_activation(test_id, seq) was already non-unique — it was written so because a failed dispatch may be retried — and it serves this for a different reason.
That N-rows-per-run shape is why the campaign’s own state machine is fed a run-ended event only by the last activation of the run to reach a terminal state: letting the first configuration to finish move the campaign would take it out of RUNNING and, worse, PLAN_REACTIVATION would create the next run while the rest of this one was still measuring. The predicate is a query over the siblings' state and deliberately not a counter — see FccActivationTransitionService.isLastOfRunToEnd.
| Column | Type | Meaning |
|---|---|---|
|
BIGINT NOT NULL |
|
|
BIGINT |
The fcc_test_config this run measures. NULLABLE, AND NEVER BACK-FILLED: a run dispatched before configurations were addressable had exactly one, but attaching it to the configuration row that happens to exist NOW would claim the run measured a configuration that may have been edited, re-ordered or replaced since — and this table’s whole purpose is to record what the command actually carried, independently of the test rows it was built from. NULL says "a single-configuration run from before configurations were addressable", which is the truth. |
|
INT NOT NULL |
1 for the first run, +1 for every reactivation. Shared by every activation of one run — see [a-run-is-not-a-row]. |
|
VARCHAR(64) UNIQUE NOT NULL |
UUID, generated once when the row is created and reused verbatim on every republish of the command. The ACS answers |
|
BIGINT NOT NULL |
Value of |
|
VARCHAR(32) NOT NULL |
|
|
VARCHAR(32) |
|
|
VARCHAR(255) |
Free text from the ACS ack or the local failure. Deliberately not an enum — it is diagnostic text, not a value other code branches on. |
|
VARCHAR(32), default |
Same dictionary as |
|
DATETIME(3) |
The window the ACS is allowed to run this activation in. |
|
DATETIME(3) |
Set when the ACS reports |
|
DATETIME(3) |
|
|
INT, default 0 |
Devices the ACS still had in flight when it acknowledged a |
|
INT, default 0 |
Size of the device set this run was dispatched with. |
settings snapshot (30 columns: |
mixed |
See [fcc-activation-snapshot] below. |
Indexes: fcc_activation(idempotency_key) UNIQUE (see above); fcc_activation(state, window_end_at) serves the window watchdog (WHERE state = 'RUNNING' AND window_end_at IS NOT NULL AND window_end_at ⇐ ?); fcc_activation(test_id, seq), deliberately not unique, serves the per-test run history behind the test detail screen and the regulatory export — forbidding a repeated seq here would turn a recoverable dispatch retry into an insert error.
M2 adds two more, in changelog_add_fcc_activation_indexes_m2.xml.
Neither is served by the existing (state, window_end_at) index: a composite index can only answer a range on its first non-equality column, so a predicate on window_start_at or on stopped_at behind an equality on state falls back to scanning every row in that state.
| Index | Query it exists for |
|---|---|
|
The |
|
The stop-grace watchdog: |
The IS NOT NULL in both predicates is part of the statement and not an assumption about the data: window_end_at and stopped_at are nullable — a run whose test carries no total_duration_h has no computed end — and NULL ⇐ ? is unknown rather than false, so leaving it out would make the statement’s meaning depend on the optimizer rather than on the schema.
M4 adds one more index, in the same changelog_add_fcc_test_device_model_index.xml (Two Indexes for the M4 Report Filters): fcc_activation(started_at), serving the period filter of POST /FCC/reports/list (WHERE started_at >= ? AND started_at ⇐ ? AND test_id IN (SELECT id FROM fcc_test …)). All three existing indexes on this table lead with state, which a report never constrains — it reports on runs in every state — so a report over a date range alone had no index to use at all before this one. started_at and not created, because the filter is about when the measurements were taken, not when the row was inserted — see [report-period-predicate].
fcc_acs_outbox
The transactional outbox toward fccCommandQueue. A Hazelcast offer() is not part of a database transaction, so the command is written here in the same transaction that creates the activation and bumps fcc_test.command_seq; a separate dispatcher performs the offer(). Nothing is ever sent that is not committed, and nothing committed is silently lost.
| Column | Type | Meaning |
|---|---|---|
|
BIGINT NOT NULL |
|
|
VARCHAR(64) NOT NULL |
Copied from |
|
BIGINT NOT NULL |
|
|
VARCHAR(32) NOT NULL |
|
|
CLOB / LONGTEXT NOT NULL |
The |
|
VARCHAR(32), default |
|
|
INT, default 0 |
|
|
DATETIME(3) |
|
|
DATETIME(3) NOT NULL |
NOT NULL so the dispatcher predicate never special-cases a null; a row due immediately is written with "now". |
|
VARCHAR(512) |
Since M2 the outbox carries STOP rows written by three different paths, all through the single FccStopCommandPublisher: the operator stopping or pausing a campaign, the operator deleting one whose run is still live, and FccStalledActivationService giving up on a run the ACS never acknowledged.
A STOP is always a new command — its own idempotency key, command_seq = fcc_test.command_seq + 1 — and never a republish, because the dispatcher re-sends a stored payload verbatim and reusing the ACTIVATE’s key would make the two commands indistinguishable to the ACS.
See An Effect Is a Declaration, Not an Action.
Indexes: fcc_acs_outbox(state, next_attempt_at) serves the 5-second dispatcher (WHERE state = 'PENDING' AND next_attempt_at ⇐ ? ORDER BY next_attempt_at, id); fcc_acs_outbox(state, sent_at) serves the 30-second ack-timeout claim (WHERE state = 'SENT' AND sent_at ⇐ ? ORDER BY sent_at, id), which the first index cannot answer; a unique index on idempotency_key serves the lookup FccAckService does for every ack and, being unique, also rules out the duplicate that would make that Optional lookup throw. fcc_acs_outbox(activation_id) serves ack correlation. Of these four, (state, next_attempt_at) and (activation_id) ship with the table itself in changelog_create_fcc_acs_outbox.xml; only (state, sent_at) and the unique idempotency_key index were missing and were added afterwards, in their own changelog_add_fcc_acs_outbox_indexes.xml, a separate changelog so the table changelog keeps its checksums.
A fifth index, fcc_acs_outbox_state_created_idx (state, created), was added in M5 (changelog_add_fcc_acs_outbox_metrics_index.xml, again a separate changelog so the earlier two keep their checksums), to serve FccMetricsSampler’s outbox sample — see Outbox Metrics and the sampler’s latency isolation. The M5 code review found the sampler’s own javadoc had been claiming an access path that did not exist: the query it described as index-served was `SELECT state, COUNT(*), MIN(created), MIN(sent_at) … GROUP BY state over the three live states, and no index on this table carries created — (state, next_attempt_at) and (state, sent_at) both answer the state predicate but neither can serve MIN(created) without a row lookup per matching entry, which is exactly the access path both database optimisers abandon in favour of a full scan once that cost is priced in. ABANDONED was also one of the three states counted, and it should never have been on a 15-second tick at all: markAbandoned is terminal and nothing in this service purges the row afterward, so ABANDONED accumulates for the life of the installation exactly like ACKED does — it is not bounded by work in flight the way PENDING and SENT are. The fix is now one statement per state rather than one grouped query: PENDING and SENT are counted from (state, created) and the pre-existing (state, sent_at) respectively, both index-only reads with MIN(…) answered as the head of the range in a single descent on both vendors, and ABANDONED is counted separately, on its own five-minute cadence rather than every sample tick, since a range over a set that only grows is not something to put on a 15-second poll.
fcc_device_state
Where one device of one run currently is, as last reported by the ACS on fccDeviceStateQueue.
This is current state, not history: one row per (activation_id, cpe_id), overwritten in place.
The history of a run is its measurements in ClickHouse; this table answers the single question the per-device status screen asks — "what is this CPE doing right now, and how many times has it retried".
| Column | Type | Meaning |
|---|---|---|
|
BIGINT NOT NULL |
The run. Keyed on the run and not on the test: a device that took part in three daily runs of one campaign has three rows. |
|
INT NOT NULL |
|
|
VARCHAR(64) |
|
|
VARCHAR(32) NOT NULL |
|
|
VARCHAR(32) |
|
|
INT NOT NULL, default 0 |
Monotonic counters, merged with |
|
DATETIME(3) NOT NULL |
The instant the ACS observed the state, not the instant this row was written — |
|
Six of these columns are written and never read back, and they stay that way on purpose.
They are not dropped, because the data is the only record of why a device ended where it did and
it cannot be reconstructed afterwards: the ACS sends each state once, and a run that finished with
4 000 devices |
Why the counters are NOT NULL with default 0: GREATEST returns NULL for a NULL argument.
The row is written by an atomic upsert — INSERT … ON DUPLICATE KEY UPDATE on MySQL, MERGE on Oracle (FccDeviceStateDao) — whose whole point is that iteration, wan_retries, failed_retries and skip_count only ever move forward, via GREATEST(stored, incoming).
GREATEST returns NULL as soon as one argument is NULL, on both MySQL and Oracle, so a single nullable counter would silently erase the stored value the first time a message arrived without it — not lose to it, erase it.
Three halves of the same rule, and all three are required:
-
NOT NULL DEFAULT 0in the schema; -
primitive
intcomponents inFccDeviceStateRow, bound withsetIntand neversetObject, so the type system makes it impossible to forget; -
FccDeviceStateService.orZerosubstituting0for an absent wire value, where "the ACS did not say" still means "nothing new to raise it by".
Why GREATEST at all: Hazelcast guarantees delivery, not order, so a message reporting iteration 4 can arrive after one reporting iteration 5, and both are true statements about the past.
Taking the maximum makes the four columns immune to that reordering — the stored value is the furthest the device is known to have got, and no arrival order can walk it backwards.
serial, state and state_reason are not monotonic — a device legitimately goes SENT, then ERROR, then WAN_CHECK, and there is no "greater" state — so the only correct merge for them is newest-wins, decided by observed_at with >= rather than >: two messages can carry the same observation instant, and the later arrival is then at least as informed as the stored one.
observed_at itself takes the maximum, so a reordered older message updates the counters it can only raise and leaves both the state and the watermark alone.
| Index | Why |
|---|---|
|
The index the atomic upsert is built on. Without a unique key on exactly these two columns, MySQL’s |
|
The per-device status screen and the grouped progress counters behind it ( |
=== fcc_audit
Who did what to a campaign — an append-only trail, never updated and never deleted.
The four audit columns every table already carries answer "when was this row last touched and by whom"; they do not answer "who stopped this campaign on the 14th", because an update overwrites its predecessor. A compliance campaign disturbs subscriber lines and its results end up in a regulatory filing, so the lifecycle actions have to be a trail of their own. And because the product has no role model, this row is the only accountability there is — see No Authentication by Design.
| Column | Type | Meaning |
|---|---|---|
|
VARCHAR(32) NOT NULL |
|
|
BIGINT |
The business key of the row the action was taken on, never a surrogate id: |
|
VARCHAR(32) NOT NULL |
|
|
INT |
|
|
INT |
The domain the test belonged to at the time, so the trail stays readable after a test is moved or removed. |
|
VARCHAR(1024) |
Free-text summary — |
Index: fcc_audit(entity_type, entity_id, created) is built for the read behind the history tab of a test, WHERE entity_type = ? AND entity_id = ? ORDER BY created DESC — but FccAuditRepository declares no query method at all today, and nothing else reads fcc_audit either. The index anticipates a read that does not exist yet; that is a defensible thing to have done, since the append-only rule and the column shapes already have to be right before a reader can be built on top of them, but it means the index is currently unexercised rather than serving anything.
created is the third column so that, once a reader is added, the sort is answered from the index instead of a filesort over a trail that only grows.
The leading entity_type is what lets the same index serve the M3 entity types without a second one.
==== Three Entity Types, One Action Vocabulary
FccAuditService gained an overload in M3 (record(entityType, action, entityId, domainId, userId, details)); the original five-argument method is now a thin delegate that passes AUDIT_ENTITY_TYPE_TEST.
action stays the closed FccAuditActionEnum while entity_type is a set of String constants, and the split is deliberate: CREATE / UPDATE / DELETE mean the same thing on all three entities, so per-entity constants such as MODEL_CAPABILITY_UPDATE would make the trail impossible to read across them — "what changed on the 14th" would become three queries instead of one — for exactly no gain.
entity_type |
entity_id |
domain_id |
Written by |
|---|---|---|---|
|
|
The test’s domain |
|
|
The model id |
|
|
|
The domain id |
The same domain id |
|
The null domain_id on MODEL_CAPABILITY is a statement, not a gap. The change belongs to no domain — the table is installation-global ([fcc-model-capability]) — and recording the actor’s own domain would suggest the change was scoped to it, which is precisely the misreading the X-FT-Domain-Scope: ALL requirement exists to prevent.
Both new paths write in the same transaction as the change they describe, enforced by Propagation.MANDATORY exactly as the campaign paths are.
=== fcc_result_dead_letter
The last resort for a measurement that could not be stored.
A measurement is the product of a compliance campaign: the ACS disturbed a subscriber line to produce it and it may end up in a regulatory filing, so it must never be silently lost.
Through M1 the only place a result could live between arriving on the Hazelcast queue and reaching ClickHouse was an in-memory buffer, and exhausting that buffer — or restarting the service while it was full — destroyed the rows.
This table is where they go instead; it replaces both TODO(M2) drop paths and the shutdown drop.
| Column | Type | Meaning |
|---|---|---|
|
BIGINT / BIGINT / VARCHAR(64) / VARCHAR(32) / INT |
All five are denormalised out of the payload purely so an operator answering "what did we lose for run X" can use an index and a |
|
VARCHAR(32) NOT NULL |
|
|
VARCHAR(32), default |
|
|
INT NOT NULL, default 0 |
Bounds nothing in v1. There is no resend poller and no attempt ladder yet, so this column is |
|
VARCHAR(512) |
|
|
DATETIME(3) NOT NULL |
When the result reached this service — not when it was parked. A row written after a long outage still has to record the arrival it describes, and it is this instant, never |
|
CLOB / LONGTEXT NOT NULL |
The |
reason |
What it means and what happens next |
|---|---|
|
The result is valid, the store is not reachable — or the in-memory retry buffer was exhausted, or the JVM was shutting down with rows still buffered. Written as |
|
The result arrived more than |
A LATE row is parked straight into DISCARDED, never into PENDING.
PENDING means "waiting to be resent", and a LATE row is never resent by design.
It is also what keeps the recovery path working at all: LATE rows are written in bursts, when a stopped run’s grace expires, so they are both numerous and the oldest rows in the table.
Leaving them PENDING would fill every oldest-first read with rows the resend can only skip, and no CLICKHOUSE_UNAVAILABLE measurement would ever be recovered again.
The read guards against this from the other side too: FccResultDeadLetterDao.findPending carries AND reason <> 'LATE' in both vendor statements, so rows written by an earlier build — when LATE did land in PENDING — cannot block the head of the read either. A resend that still meets one counts it (fcc.result.resend.skipped.late) — there is no fcc.result.deadletter.* meter — and there is deliberately no per-row log line for it: FccResultDeadLetterService.resend sits in a loop over up to ft.fcc.dead-letter.resend-batch-size rows, so a per-row line would print as many times as the skip does, and what an operator acts on is the aggregate fcc dead letter resend read=… skippedLate=… line logged once per call plus the counter itself, which is the one signal that the table holds rows the recovery path cannot use.
The reason column is kept either way: DISCARDED says what happened to the row, LATE says why, and an operator asking "what did this run measure after it was stopped" needs the second one.
Index: fcc_result_dead_letter(state, created) serves the only reader, WHERE state = 'PENDING' AND reason <> 'LATE' ORDER BY created, id.
The state prefix is what keeps the read proportional to the backlog rather than to the whole table, since RESENT and DISCARDED rows are kept forever and never removed by the service.
FccResultDeadLetterService.store runs in its own transaction (REQUIRES_NEW) and never throws.
The ingest path that calls it is already in the middle of a failure; a record written in the caller’s transaction could be rolled back with it, and the one thing this table must never do is vanish together with the problem it documents.
If even that insert fails, the measurement is genuinely gone — not stored, not parked, nothing will retry it — and only then is fcc.result.lost incremented, separately from fcc.result.parked.
=== fcc_model_capability
What one CPE model can be asked to do, as far as this installation has been told. At most one row per model.
| Column | Type | Meaning |
|---|---|---|
|
BIGINT PK |
Surrogate. |
|
INT NOT NULL |
|
|
INT |
Ceiling for TR-143 |
|
INT |
The same for |
|
TINYINT(1) NOT NULL, default 0 |
Whether the model can run a TR-143 time-based test ( |
A model with no row here is UNCONSTRAINED, which is why a null ceiling is not zero.
The table is an allow-list of known limits, not a registry of every model. The vast majority of an installation’s models will never have a row — nobody has read their datasheet — and reading an absent row as "zero connections" would refuse every test on every one of them, with a 400 naming a ceiling nobody entered.
Null therefore means "nothing is known about this model, so neither this service nor the UI may narrow the operator’s choice". Three places implement that one reading, and all three must keep it:
-
FccModelCapabilityProvider.capabilitiesOfleaves an absent model out of the returned map entirely, and callers fall back toFccModelCapability.unknown()—(null, null, false). -
FccCatalogServicerenders those nulls straight ontoFccCatalogModelItem.connectionsDlMax/connectionsUlMax, so the UI leaves the input unconstrained. -
FccTestValidator.validateConnectionCeilingstakes the minimum non-null ceiling across the fleet’s models, so a null neither lowers nor raises it — see The Capability Ceiling Is Refused, Not Clamped.
The one place a missing row is legitimately a 404 is POST /FCC/admin/model-capability/details, and that is not a contradiction — see 404 on Read, 403 on Write.
Installation-global, not per-domain. There is no domain_id column, because a connection ceiling is a property of the hardware: the same model behaves the same way in every domain. That is also why the admin write path requires X-FT-Domain-Scope: ALL — a row here constrains every operator of the installation, and this product has no role model that could express "installation administrator" any other way (The Two Administration Tables Are Not Authorized the Same Way).
Index: fcc_model_capability(model_id) UNIQUE. It enforces one row per model in the database rather than only in the admin service — the PUT is an upsert keyed on model_id, and two rows for one model would make the ceiling that is applied depend on which one the read happened to return first. It also serves the only two reads there are: WHERE model_id = ? (the admin details screen) and WHERE model_id IN (…) (FccModelCapabilityProvider, which resolves the whole model set of a catalog page or of a resolved fleet in one statement, chunked at 1000 for Oracle).
Oracle additionally gets a FCC_MODEL_CAPABILITY_0 sequence, the same convention every other table of this schema uses.
=== fcc_domain_setting
The middle level of the three-level settings chain — see [settings-resolution-chain] for what the chain does and when it runs. At most one row per domain.
The per-domain level exists for multi-ISP installations: one deployment serves several ISPs, each is a domain, and each has its own test servers. A single global URL is wrong for all but one of them, while making every operator retype the same two URLs into every campaign is how they end up mistyped.
| Column | Type | Meaning |
|---|---|---|
|
BIGINT PK |
Surrogate; the domain id is the business key. |
|
INT NOT NULL |
|
|
VARCHAR(512) |
This ISP’s own test servers: |
|
INT |
WAN cross-talk thresholds in kbps, per direction: above these the link is considered busy with somebody else’s traffic and the iteration is recorded as |
|
INT |
Percentage of the provisioned tier that still counts as a pass. Unlike the URLs this one does have a global level below it (FT Configs, then |
|
INT |
The pass/fail bound this service applies to a measured average round trip, in milliseconds; global fallback |
|
VARCHAR(255) |
The ISP’s default ping target, written into TR-181 |
Every field is nullable, and a null is indistinguishable from the absence of the row: both mean "this domain publishes no opinion on this value". FccDomainSettingResolver.forDomain returns an Optional and FccTestService.chain treats an empty one and a null field identically, on purpose — a caller that had to tell them apart would be encoding a distinction that has no meaning.
Index: fcc_domain_setting(domain_id) UNIQUE, enforced in the database and not only in the admin service. The PUT is an upsert keyed on domain_id, and two rows for one domain would make the URL a campaign is frozen with depend on which row the read happened to return first — the exact silent divergence the save-time freeze exists to prevent. It serves the read that runs on every save of every test (WHERE domain_id = ?, FccDomainSettingResolver) and the scoped admin list (WHERE domain_id IN (…), the caller’s visible domains).
Oracle additionally gets a FCC_DOMAIN_SETTING_0 sequence.
=== fcc_legacy_import
The cursor and the progress of the legacy ETL, which reads the FCC results the ACS wrote before this service existed and replays them into ClickHouse — see Legacy FCC Import for what the import reconstructs and why.
Exactly two rows, one per result_kind, and that is the whole cardinality of the table.
The two legacy sources are imported by independent cursors because they are not the same shape: ug_fcc_speed_result carries end_date, type and the bom/eom counters, and ug_fcc_latency_result has none of those and carries the round-trip aggregates instead.
A single shared cursor would have to be the union of both column sets — and, worse, a stall on one kind (an unreadable row, a ClickHouse outage on one table) would hold the other kind’s import hostage.
Two rows let one kind sit FAILED while the other keeps making progress, and let an operator restart one of them alone.
| Column | Type | Meaning |
|---|---|---|
|
VARCHAR(32) NOT NULL |
|
|
VARCHAR(32) NOT NULL, default |
|
|
VARCHAR(64) |
Identifies one import attempt, and fences every mutating statement of the table: each one carries |
|
VARCHAR(64) |
The zone this run interpreted the naive legacy timestamps in. It is a run parameter and not a property of the data: |
|
INT / VARCHAR(64) / VARCHAR(32) / DATETIME(3) / BIGINT / INT |
One position, six columns, all five position values taken from one source row. |
|
BIGINT / BIGINT / INT |
The legacy id range of the batch handed to ClickHouse and the campaign it was read for, committed before the insert and nulled after it. Non-null at startup means a previous run died inside exactly that window. See [legacy-import-pending-range]. |
|
DATETIME(3) |
The completeness mark: every row of this kind whose |
|
BIGINT |
The highest legacy id a completed pass has covered. A diagnostic high-water mark that bounds no read; its one use is recognising a late arrival. See [legacy-import-watermark]. |
|
BIGINT NOT NULL, default 0 |
|
|
BIGINT NOT NULL, default 0 |
Data-quality tallies, kept apart rather than summed because they need different answers: an overflowed |
|
INT NOT NULL, default 0 |
Rows created in this schema, bounded by the number of legacy groups and devices rather than by the number of results — hence |
|
BIGINT |
Counted once at the start, only so progress can be reported as a fraction. It is an estimate and nothing ever decides that the import is finished from it — the legacy flow keeps writing, so the number is stale the moment it is taken. |
|
TINYINT(1) NOT NULL, default 0 |
The cooperative stop: the operator sets it, the running import notices it between batches and leaves a clean resumable cursor instead of being killed mid-batch. |
|
DATETIME(3) |
What distinguishes a slow import from a dead one, and what the claim’s staleness arm compares against. Refreshed between batches and over the slow parts of a pass that are not batches: the per-campaign |
|
DATETIME(3) |
|
|
VARCHAR(1024) |
Truncated, not a CLOB: this is the operator-facing reason the import stopped, and the full stack trace belongs in the log rather than in a status row. |
Index: fcc_legacy_import(result_kind) UNIQUE — the two-rows-only rule enforced in the database and not only in the service.
Every access is "the row for this kind": the ETL loads it by kind at start, updates it by kind after each batch, and /status reads both.
A second row for one kind would hand a resuming import a stale cursor and re-import a range that was already written, which on ClickHouse cannot be undone.
The unique index is also what makes the get-or-create at startup safe under two processes racing to begin the same import.
Oracle additionally gets a FCC_LEGACY_IMPORT_0 sequence, the same convention every other table of this schema uses.
M4 also adds one index to the existing fcc_test, in changelog_add_fcc_test_legacy_ug_index.xml so the table changelog keeps its checksums: fcc_test(legacy_ug_id, type) UNIQUE, the key the import finds its own anchor campaign by — see One Update Group Can Become Two Anchors for why the type column is in it.
Four of the columns above — watermark_started_before, pending_ug_id, rows_late_skipped and rows_invalid — arrived with the M4 review, in changelog_add_fcc_legacy_import_watermark.xml and not in changelog_create_fcc_legacy_import.xml, for the same reason: that file has already run at every installation and its changesets have to keep their checksums.
Each changeset is guarded by a not columnExists precondition with onFail="MARK_RAN", so an installation that already carries the column records the changeset as run rather than failing on it.
==== Why the Cursor Is Composite and Not a Last-Id
Rows are read in the order
(ug_id, serial, direction, start_date truncated to the second, id)
because that is exactly the order the per-group iteration rank is defined over: iteration numbers the measurements of one (window, serial, direction) group in the order they were taken, and the truncation to the second is what makes rows written within the same second rank deterministically instead of by insertion accident.
A plain "resume after id N" cursor cannot express a position in that order — ids are not monotonic along it — so a batch that ended in the middle of a group could not resume the group’s ranking.
Hence the five position columns, plus cursor_iteration, which carries the running rank of the group the cursor currently sits in: a group split across a batch boundary continues at iteration N+1 on the next batch instead of restarting at 1 and producing two measurements that both claim to be the first.
cursor_result_id on its own therefore says nothing about how far the import has got. Only the tuple does.
Two rules the tuple has to obey, both of which the M4 review found broken:
-
cursor_directionstores the RAW legacytypeas text, never a direction name.type0, 1, 3 and null all map to the nameDOWNLOAD, so the mapping is many-to-one and a name cannot be inverted into the number the legacy table is compared on. A cursor restored from a name would either re-admit rows that are already imported or drop the direction term from the traversal order half-way through a campaign. -
All five position values come from ONE source row, and a null in any of them is refused rather than stored. In SQL both
x > NULLandx = NULLareUNKNOWN, so such a cursor matches no row at all — the next batch comes back empty and the runner reads that as the end of the campaign. The pass halts instead; see A Null Cursor Component Halts the Pass.
==== Why pending_from_id / pending_result_id / pending_ug_id Exist
ClickHouse MergeTree has no unique constraint and no upsert: a row inserted twice stays in the table twice, forever, and there is no statement that can remove it.
In a regulatory report that is not a cosmetic defect — it silently inflates the sample.
So the id range of a batch is written here and committed before the ClickHouse insert is attempted, and cleared to null only after the insert is acknowledged.
A non-null pending_result_id found at startup therefore means the previous run died inside exactly that window and the batch’s fate is unknown; the recovery re-checks that bounded id range against ClickHouse — and only that range — before re-importing it, instead of having to reconcile the whole import.
The range is MIN..MAX of the batch and not first..last, because the traversal order above is deliberately not id order and the ids of one batch are scattered.
pending_ug_id ties the range to the campaign it was read for, and the recovery set is offered to that campaign and to no other.
A run that died in the first batch of a campaign committed no cursor for it, so the stored cursor still names the previous campaign — which has nothing left to skip. Without the column the recovery set is handed to that previous campaign and cleared there, and the batch that genuinely is in doubt is re-imported into a table with no unique constraint.
For the same crash, a non-null pending_result_id makes the row resumable on its own, with no cursor behind it: a fresh claim would otherwise erase the only record of which ids are in doubt before the probe could read it.
FccLegacyImportDao is hand-written JDBC and no method on it is transactional, which is what this contract rests on: each statement runs on autocommit, so the pending marker is durable the instant the call returns. Wrapping the import in a transaction would defer every cursor commit to the end of a run that may take hours and lose all of them to the crash they exist to survive. The counters are written as column = column + ? for the same reason — a commit adds what the batch actually did rather than overwriting the row with a stale absolute.
==== Why the Watermark Is a Date, and Why watermark_result_id Is Only a Diagnostic
A pass needs a completeness mark separately from the cursor at all because the cursor columns cannot answer the question: they name a position in a traversal order that is deliberately not id order, so a position in it says nothing about which rows are complete.
The cursor resumes an interrupted pass; the watermark is what makes a new pass cheap and non-duplicating — without one a second run would re-read every row, and on a MergeTree with no unique constraint the duplicates could never be removed.
watermark_started_before is that mark. A pass covers start_date >= watermark_started_before AND start_date < cutoff, and on completion sets watermark_started_before = cutoff.
The mark and the inclusion predicate are the same column, so the intervals of successive passes tile the timeline with no gap and no overlap; completeness is true by construction rather than by argument.
It is a local wall clock because the legacy column it is compared against is one, and nullable because a kind that has never completed a pass has no lower bound — null reads as "start from the beginning".
An id watermark was unsound, and the argument that it was sound is the one this section used to make.
That argument was: only closed day-windows are imported, closed windows lie wholly in the past, and the legacy id is auto-increment in insertion order — so "all ids at or below the watermark are done" must hold.
It does not hold, because the id is global across every campaign and device of the installation while start_date is whenever the measurement was taken, and the two order independently. A device that buffers a result and flushes it late gets a high id with an old start_date; a device reporting in real time gets a lower id with a start date inside a window that is still open. Import the first, write its id as the watermark, and the second is below the watermark forever — above its own cutoff, never importable again, in a store where nothing can be back-filled into a ranked window.
The per-campaign cutoff compounded it: two campaigns whose ug_reactivate fires at different hours had cutoffs up to 24 hours apart, so one campaign’s watermark excluded another campaign’s importable rows. The cutoff is now pass-wide — see the 25-hour lag it costs.
watermark_result_id is kept, and is now only a diagnostic high-water mark. It records the highest legacy id any completed pass has covered, it bounds no read, and its one remaining use is the late-arrival predicate: a row matching start_date < watermark_started_before AND id > watermark_result_id reached ftacs after its day-window had already been imported, and is counted as rows_late_skipped rather than back-filled into a closed and already ranked window (Late Arrivals Are Counted).
It only ever moves up: a later pass covers a later date interval whose highest id is not necessarily higher, and letting the mark fall would over-report rows_late_skipped.
The comment inside changelog_create_fcc_legacy_import.xml that calls it the completeness mark describes the superseded design, and says so in place.
== The Settings Resolution Chain
the request's own value -> fcc_domain_setting (the test's domain) -> global FT Configs
first non-null wins, reading left to right
FccTestService.chain is the one implementation, and resolveConfigDefaults / resolveSpeedDefaults / resolveLatencyDefaults are its three callers.
Why this is a section of its own and not a subsection of [fcc-activation-snapshot].
There are two freezes in this system, at two different moments, for two different reasons, and they are routinely confused:
| Freeze | When | What it protects against |
|---|---|---|
The chain, into |
Save time, once |
An administrator changing a global or per-domain default between two runs of one campaign |
The snapshot, into `fcc_activation’s 30 columns |
Every dispatch |
An operator editing the campaign after a run happened |
Nesting the chain under the snapshot section would put it under a heading about activation, and the single most important fact about it is that it does not run at activation time. The two sections cross-reference each other instead.
=== Which Fields Have Which Levels
Not every field has all three, and the gaps are deliberate.
| Field | Domain level | Global level | Note |
|---|---|---|---|
|
Yes |
No |
There is no sensible installation-wide default for a URL. A domain with no row and a test with no URL simply stays unactivatable until somebody supplies one, which is what |
|
Yes |
No |
Same reasoning. The requirement survives the domain default; see [fcc-domain-setting]. |
|
Yes |
No ( |
A cross-talk threshold with no value anywhere is legitimately absent; the WAN sampling then simply produces no |
|
Yes |
Yes — FT Configs, then |
A policy figure most ISPs share. |
|
Yes |
Yes — FT Configs, then |
Same. |
|
No |
Yes — |
Timing knobs of the WAN sampling protocol, not an operator’s policy. |
|
No |
Yes — |
Protocol defaults rather than policy: there is no published FT Configs setting for them and nothing ISP-specific about how many times one pings. |
|
No |
No |
The campaign’s own subject matter. A tier is the subscriber’s provisioned plan; defaulting it would be defaulting the thing being measured against. |
testMode is not a chain at all: it is FILE_BASED when absent, full stop.
=== Blank Text Is Absent, Not a Value
chain and chainText differ in exactly one respect, and it is not cosmetic.
chainText treats a blank string as absent.
A form that posts "urlDl": "" for a field the operator left empty is the normal shape of an FT UI request, and a strict first-non-null chain would let that empty string beat the domain default — so the per-domain URL would never apply for exactly the caller it exists for, and the campaign would fail validateRunnable later with no indication that a default was available.
Blank is also what validateRunnable already reads as missing, so this keeps the two halves agreeing on what "absent" means.
And "absent" is stored as NULL, never as "".
Having decided that a blank is not a value, returning one would write an empty string into a nullable column whose entire vocabulary for "no value" is IS NULL — and every report, every migration and every hand-written query over fcc_test_setting_speed.url_dl would then have to know about a second spelling of nothing.
So when neither the request nor the domain has anything, the result is null.
=== It Runs Once, at Save Time, and Never at Activation
This is the whole point of the design and the easiest thing to get wrong.
A campaign with reactivation_days = 7 runs for a week and is activated daily.
If the chain were walked on each activation, an administrator who changed a global or per-domain URL on day three would silently make days 1-3 and days 4-7 measure different servers — and nothing in the report would show it: the test row, the activation snapshots and the measurements would all look internally consistent, each describing the configuration that was in force when it was written.
A regulatory filing has to state one set of parameters for the whole campaign.
So FccTestService freezes the resolved values into fcc_test_config / fcc_test_setting_speed / fcc_test_setting_latency when the user saves, and each activation additionally snapshots them into fcc_activation ([fcc-activation-snapshot]).
The consequences, stated plainly because they are what an operator will ask about:
-
Editing
fcc_domain_settingaffects tests saved afterwards and nothing already stored. A campaign that already froze the old URL keeps measuring against it for the rest of its reactivations. -
Deleting a
fcc_domain_settingrow makes future tests of that domain fall through to the global defaults, and changes nothing about the campaigns that already froze their values. -
POST /FCC/item/detailsreads back the resolved values, not what was sent. A body that omittedspeed.downloadUrlcomes back carrying the domain’s URL, because that is the stored campaign. -
PUT /FCC/item/exportwrites the settings as stored, and re-importing them does not re-resolve the chain. The file carries resolved values, so on import they are explicit request values and win over the target installation’s own defaults — an operator who wants the target domain’s test servers deletesdownloadUrl/uploadUrl/testHostfrom the file before importing, because the chain only fills fields that are absent. See The Settings Chain, and How to Opt Out of It. -
The audit trail matters more here than for a campaign, not less. A campaign saved after a domain edit looks exactly as though the operator had typed those numbers themselves; the
entity_type = 'DOMAIN_SETTING'row in [fcc-audit] is the only record that they did not.
=== There Is No Walk Up the Domain Tree, and There Cannot Honestly Be One
FccDomainSettingResolver.forDomain is an exact domain_id match.
A domain with no row of its own does not inherit its parent’s; it falls straight through to the global default.
That is a deliberate limitation rather than an omission, and it is not a limitation this service can lift by trying harder. Two independent facts stand in the way:
-
The ACS encodes its domain hierarchy inside the dotted
isp.namestring and has no parent column. There is noparent_id, no closure table, no adjacency list — the shape of the tree lives in a naming convention. The only inheritance available from here would be a guess by string prefix, which breaks the first time a domain name contains a dot for some other reason. -
This service is never handed the tree at all.
ui-backendsends a flat list of visible domain ids inX-FT-Domain-Ids(Identity Contract) and nothing else — no names, no parents, no ordering that means anything. Even if the ACS had a parent column, this service does not readisp, and adding that read would put the hierarchy of an entire product into a table this service holds a read-only account on and does not own.
And a guess is the wrong kind of wrong here: a wrongly inherited URL is a campaign that measures the wrong server while looking perfectly configured. It produces numbers, they go into a report, and nothing anywhere says which host they came from.
An installation that wants one URL for a subtree writes a row per domain of that subtree. That is more rows, and they are rows somebody can read.
== The fcc_activation Settings Snapshot
The 32-column block on fcc_activation (test_type, test_mode, wan_sample_mode, threshold_dl/ul, tier_dl/ul, url_dl/ul, connections_dl/ul, file_size_ul, time_based_duration_sec, time_based_interval_sec, ping_host, ping_repetitions, ping_timeout_ms, ping_data_block_size, total_duration_h, single_duration_min, batch_interval_min, retry_failed, retry_wan, wan_sample_gap_sec, wan_retry_interval_sec, wan_step_timeout_sec, diag_timeout_sec, speed_success_percent, latency_success_ms, only_online, push, hubb_id — 28 names above, four of which are the dl/ul pairs counted as one each, for 32 columns in the table itself) duplicates fcc_test_config / fcc_test_setting_speed on purpose.
test_mode AND THE TIME_BASED PAIR ARE THE SNAPSHOT COLUMNS THAT SELECT A FORMULA, which makes them load-bearing in a way the rest of the block is not. Everything else here is reported; these three are computed with. FccSpeedStatusCalculator reads test_mode to choose between the file-based rate (total_bytes over bom..eom) and the time-based one (test_bytes_under_full_loading over period_of_full_loading_micros) — see The Pass/Fail Formula. Taking the mode from the CAMPAIGN instead would re-judge a finished run the moment somebody switched the campaign to the other mode, which is the exact failure this whole snapshot exists to prevent, except that it would change a verdict rather than a displayed figure.
An absent test_mode is read as FILE_BASED, because that is what every run dispatched before contract 1.6 actually was.
The test is editable; the run is not. A regulatory report has to show the parameters the ACS actually applied at the time a run happened, not the ones somebody typed into the test afterwards. Reading the settings back off fcc_test at report time would silently rewrite history every time an operator edits a test.
The consequence: editing a test’s settings takes effect only from the next reactivation. FccTestService.resolveConfigDefaults / resolveSpeedDefaults / resolveLatencyDefaults resolve absent values once, at save time — never at activation time — and since M3 they walk the full three-level chain (request, then fcc_domain_setting, then FT Configs) rather than only the global one: see [settings-resolution-chain]. A test with reactivationDays=7 runs for a week and is activated daily; if the defaults were re-read on every activation, an administrator changing a global or per-domain default mid-run would silently make days 1-3 and days 4-7 measure different things. The defaults are frozen into the test when the user saves it, and each activation additionally snapshots them into fcc_activation — two freezes, for two different reasons (the test freeze protects against a default drifting between two runs of one campaign; the activation freeze protects the report against a later edit of the test).
The ping_* and latency_success_ms columns were created in M1, before any latency campaign could exist, precisely so the M2 latency milestone became a code change and not an ALTER TABLE on a table that by then held live activations.
They are in use since M2.
=== The Snapshot Is Type-Driven, and the Other Half Is Written NULL
FccActivationMapper.applySnapshot switches on fcc_test_config.type — the CONFIGURATION’s type and not the campaign’s, which no longer exists: a SPEED configuration’s activation gets the fcc_test_setting_speed columns, a LATENCY one gets the fcc_test_setting_latency ones.
The half that does not belong to the type is written null, explicitly, rather than left as it was.
This row is what a regulatory report renders, and a leftover URL next to a ping result is a report stating parameters the run never used.
Leaving the other half alone would do exactly that on the two paths that re-snapshot an existing row: a test whose type was changed between two runs, and FccActivationHandler.dispatch, which re-takes the snapshot of a row that was planned under the previous type.
For a LATENCY run the upload columns (threshold_ul, tier_ul) stay null as well — a ping is symmetric and the contract states FccWanThresholdsDto.ulKbps / FccTiersDto.ulKbps are null for a latency test — and so do test_mode and wan_sample_mode, which describe how a throughput measurement is taken and have no meaning for a ping.
=== A Planned Reactivation Re-Takes Its Snapshot at Dispatch
FccActivationHandler.planReactivation writes a provisional snapshot when it creates tomorrow’s run as PLANNED, so that a planned row is a complete, readable run rather than a stub the UI has to special-case.
FccActivationHandler.dispatch then re-takes the snapshot from the current fcc_test_config / fcc_test_setting_* rows at the moment the run is actually sent.
That re-take is what implements the product rule stated above: the run already on the ACS keeps the configuration its command carried — a republish re-sends the stored fcc_acs_outbox.payload and never re-runs the mapper — while the next one picks up whatever the operator changed in the meantime.
Snapshotting once at planning time instead would freeze a configuration hours or days before it is used, and an operator who fixed a wrong URL would watch the wrong one run anyway.
command_seq is the one column planReactivation deliberately leaves unset, storing the placeholder 0 until dispatch assigns the real sequence in the same transaction as the outbox row.
The sequence is consumed by the command, not by the activation row: the ACS discards a command whose sequence is not greater than the last one it accepted, so a reactivation planned today and sent tomorrow would otherwise burn a sequence now and make every command written in between look stale.
activation_count is not bumped by planReactivation either — it counts the activations an operator asked for, which is what freezes the device set after the first one, and a reactivation the service planned by itself is not a new decision by anybody.
== The iotw Grant: Two Tables, SELECT Only
configs[].sourceType=Condition names a saved view of the portal’s device list, and that data lives in `ui-backend’s own schema.
This service reads two tables there and nothing else:
GRANT SELECT ON iotw.iotw_view TO ftfcc_iotw_ro;
GRANT SELECT ON iotw.iotw_view_column_condition TO ftfcc_iotw_ro;
| Table | What is read from it, and why only that |
|---|---|
|
|
|
|
Four properties of this grant are worth stating, because each is a decision rather than a default:
-
No writes, ever. The pool is opened read-only in code (
DataSourcesConfig.iotwDataSource) as well as granted read-only. These are the portal’s views, and this is a borrowed read, not a shared model. -
No transaction manager. Reading a condition is one or two statements with no invariant between them, and a second manager would tempt somebody to join this read into an
ftacstransaction — which no two-phase-less pair of datasources can honour. -
Lazy. An installation with no grant, or with
ft.fcc.iotw.enabled=false, starts normally and refuses only that one device source type. The startup check (ft.fcc.iotw.schema-check.enabled) reports a missing grant with the full list of missing columns and warns rather than failing. -
On Oracle the two tables belong to their own user, so the datasource issues
ALTER SESSION SET CURRENT_SCHEMA— without it every unqualifiedFROM iotw_viewwould resolve against the connecting account’s own empty schema and answer "table or view does not exist".
== ClickHouse Schema
=== fcc_result_speed
One row per speed diagnostic iteration and direction, in the shared database ftacs_qoe_ui_data (QoE already owns diag_speed_test and diag_ip_ping there — hence the fcc_ prefix and the separate Flyway history, see [flyway-history]).
| Column | Type | Meaning |
|---|---|---|
|
Int64 |
|
|
Int32 |
|
|
Int32 |
Taken from |
|
LowCardinality(String) |
Always present: |
|
LowCardinality(String) |
Legacy maps 1:1. Same provenance as |
|
LowCardinality(String) |
|
|
String |
Not |
|
Int32 |
Not |
|
DateTime64(3) |
|
|
Nullable(DateTime64(3)) |
Nullable because legacy latency rows carry no end date; kept nullable here too for a uniform ETL path across both result tables. |
|
LowCardinality(String) |
|
|
String |
|
|
Int32 |
|
|
UInt8 |
The measurement arrived once the run had been told to stop. Two sources since M2: the wire flag |
|
LowCardinality(String) |
|
|
Nullable(Int64) |
Populated only by the M4 ETL. |
|
DateTime64(3) |
Partitioning key ( |
|
Nullable(DateTime64(3)) |
Beginning/end of measurement. |
|
Nullable(Int64) |
|
|
Int32 |
|
|
Float64 |
Computed by this service; see The Pass/Fail Formula for the exact expression and why it must never use integer division. |
Engine: MergeTree, PARTITION BY toDate(created), ORDER BY (fcc_test_id, serial, direction, created), TTL created + INTERVAL 400 DAY.
A TTL clause has not been exercised anywhere in FT before this table. It needs to be verified on ClickHouse 24.8, together with the daily partitioning, that parts are actually dropped at 400 days and that the background TTL merge does not fight the daily partition boundaries.
LowCardinality is applied only to the string columns that have a small, repeating value set: serial, status, source, direction, hubb_id. It is deliberately not applied to tier_kbps: that column is numeric, and wrapping a number in LowCardinality buys nothing while adding a dictionary per part.
The column list is treated as final for M1: everything the M4 legacy ETL will need was added now, on purpose, because an ALTER TABLE … ADD COLUMN over tens of millions of rows while the ETL is running is exactly the operation this design avoids discovering it needs later.
The natural de-duplication key is (activation_id, serial, direction, iteration). It is not enforced by MergeTree (no unique constraint exists in ClickHouse), so readers — and FccResultIngestService, implemented and code-read, via one atomic Hazelcast IMap.putIfAbsent per result, see De-Duplication Is One Atomic putIfAbsent — de-duplicate on it themselves.
=== fcc_result_latency
One row per latency (IPPing) diagnostic iteration, created by the Flyway migration V2create-fcc-result-latency-table.sql in the same shared database ftacs_qoe_ui_data, with two columns added by V3add-ping-actuals-to-fcc-result-latency.sql under contract 1.5 — see [contract-1-5-ping-actuals].
A second table rather than more Nullable columns on fcc_result_speed: the two diagnostics share nothing below the common block, a MergeTree part would carry a dictionary and a null map for every column the other diagnostic owns, and the ORDER BY that serves a speed report is not the one that serves a latency report.
The common block is identical to fcc_result_speed in name, order and type, down to the nullability.
That is a hard rule, not a coincidence.
Every cross-diagnostic read is a UNION ALL of the two tables, and ClickHouse matches a UNION ALL by position — so a column reordered here would not fail, it would silently return a serial in the url column of half the report.
The first twenty columns are therefore exactly those of fcc_result_speed, with the same provenance and the same fallbacks (FccResultIngestService.toLatencyRow fills them from the same helpers toSpeedRow uses).
Two of them read oddly for a ping and are kept anyway:
| Column | Note |
|---|---|
|
Always |
|
A ping has neither. The columns are non- |
The columns that are the latency half — five from V2, plus the two V3 adds for contract 1.5:
| Column | Type | Meaning |
|---|---|---|
|
Nullable(Int32) |
Round trip in milliseconds, from |
|
Int32 |
|
|
Int32 |
|
|
Nullable(Int32) |
Contract 1.5. |
|
Nullable(Int32) |
Contract 1.5. |
Why repetitions and data_block_size are Nullable(Int32) and not Int32 DEFAULT 0.
A 0 for either would be a statement a reader would have to believe — zero repetitions, a zero-byte packet — and every row written before this migration, plus every row an ACS older than contract 1.5 will ever produce, genuinely has no value to report. This is the same argument val_avg / val_min / val_max already carry, above. A defaulted figure is also deliberately not substituted from FccRuntimeDefaults when neither the wire nor the snapshot has one: a defaulted value would be indistinguishable from a reported one in a compliance export, and the column must be able to say "unknown" rather than guess.
Why the ALTER TABLE that adds them is safe, despite fcc_result_latency’s own header (`V2create-fcc-result-latency-table.sql) declaring its column list final precisely to avoid an ADD COLUMN over tens of millions of rows. A ClickHouse ADD COLUMN of a Nullable column with no DEFAULT expression is metadata only: no existing part is rewritten, the column materialises lazily on the next merge and reads back NULL from every existing part until then. The operation V2’s header feared is a column with a DEFAULT or MATERIALIZED expression, which does force a rewrite — this migration deliberately has neither. (V2’s own comment could not be corrected in place to say so, because Flyway checksums the whole migration file including comments; the correction lives here and in `V3add-ping-actuals-to-fcc-result-latency.sql’s header instead.)
Union-compatibility with fcc_result_speed is preserved. The two columns are appended at the very end, past the latency-only block that already differs from the speed table, and are not part of the ORDER BY, the PARTITION BY or the TTL. The common block fcc_test_id … created ([fcc-result-speed]) is untouched, so no positional UNION ALL read moves — see the hard rule at the top of this section. fcc_result_speed gets no equivalent columns: they are latency-only, a DOWNLOAD / UPLOAD iteration has no IPPing parameters to report.
Parked dead-letter rows are unaffected: fcc_result_dead_letter.payload is Jackson JSON, so a payload written before contract 1.5 deserialises with both new fields null and falls back to the snapshot on resend exactly as a live 1.5 message would.
Engine, partitioning, ordering and TTL are the same as fcc_result_speed: MergeTree, PARTITION BY toDate(created), ORDER BY (fcc_test_id, serial, direction, created), TTL created + INTERVAL 400 DAY, index_granularity = 8192.
The LowCardinality rule of [lowcardinality-rule] applies unchanged, and the natural de-duplication key is again (activation_id, serial, direction, iteration), again unenforced.
The same TTL caveat applies — see the warning under [fcc-result-speed].
status is computed by this service, from ping_ok = avg_ms < latencySuccessMs && success_count > 0, and stored in the same SUCCESS \| FAILED \| CROSS_TALK \| OTHER dictionary the speed table uses.
The ACS publishes raw measurements only and never a verdict.
See The Pass/Fail Formula for the resolution order, and in particular for why success_count == 0 is FAILED and not OTHER.
The column list was treated as final for the M4 legacy ETL for the same reason fcc_result_speed’s is: everything that ETL will need (`legacy_ug_id, legacy_result_id, source) was created up front, because an ALTER TABLE … ADD COLUMN over tens of millions of rows while the ETL is running is exactly the operation this design avoids discovering it needs later. Contract 1.5’s repetitions / data_block_size ([contract-1-5-ping-actuals]) are the one exception, and are not a reversal of that rule: they are Nullable with no DEFAULT, so the ADD COLUMN that created them was metadata-only, not the rewrite the rule above was written to avoid. The ETL columns are still not to be extended casually; a Nullable addition with no default is the narrow exception this migration relied on, not a precedent for widening the table freely.
=== The Report Read Path
FccResultQueryDao is the read side of this schema, behind the four M4 reporting endpoints (Endpoints Implemented in M4). It never runs ahead of the write side: every column it selects is one fcc_result_speed / fcc_result_latency already carries, and every predicate it builds comes from the one FccResultFilter all four endpoints resolve through FccReportScopeResolver — which is what stops the run list, the measurement list, the report header and the export from ever disagreeing about which rows a filter means.
==== The UNION ALL Is Generated, Never Hand-Written Twice
ClickHouse matches a UNION ALL by position, not by name — the hard rule [fcc-result-latency] already states for the common block. The read side takes that one step further: the two branches of every cross-diagnostic query are not two pieces of SQL, they are two renderings of one Java list (FccResultQueryDao.COLUMNS), and every column is CAST to the same type on both sides. A column the branch does not own becomes CAST(NULL AS <the same type>) at the same position, which is what makes the two renderings union-compatible by construction rather than by review.
That is also why connections, speed_kbps, packets_sent and packets_received are projected Nullable in a report row although [fcc-result-speed] and [fcc-result-latency] declare them non-Nullable in their own tables: a speed row has nothing to offer packets_sent, and the honest way to say so in a positional union is NULL, not a fabricated 0 a reader could mistake for a real count.
==== The Period Predicate Is Two Predicates
Both tables are PARTITION BY toDate(created) while a report filters on started_at — the instant the device measured, not the instant the row was written. The two differ by the ACS batching, or, for a row the legacy ETL imports, by years. Narrowing on created instead of started_at would therefore silently drop measurements; narrowing on started_at alone forces ClickHouse to read every partition, because the engine has no way to relate the two columns to each other.
The read adds a second, widened predicate on created beside the one on started_at — one day either side of the requested window (FccResultQueryDao.PARTITION_MARGIN_DAYS). It cannot exclude a row the started_at bound keeps, because nothing is written a day before it is measured, and it is what lets ClickHouse prune whole partitions instead of scanning the table. FccReportActivationSpecifications.byFilter applies the plain started_at bound to fcc_activation for the run list, on the relational side, where there is no partition to prune and a second predicate would add nothing — see fcc_activation for the index it now runs against.
==== Model and Manufacturer Are Resolved Through the Frozen Fleet
Neither result table carries a model, a manufacturer or a cpe_id column, and none is being added: V1 and V2 declare their column lists final ([lowcardinality-rule]), and an ALTER TABLE over tens of millions of rows is exactly the operation those comments exist to avoid. FccReportScopeResolver resolves a model or manufacturer filter in three steps instead — manufacturer to model ids through the ACS catalog, model ids to campaigns and serials through fcc_test_device (the frozen snapshot), and those into an fcc_test_id IN (…) predicate on the result rows, narrowed further to a serial IN (…) only when the model does not already cover every device of the selected campaigns.
The snapshot, and never the live ftacs.cpe. It describes the fleet a run actually executed against; a CPE re-provisioned into another model since then must not change what a finished run reports — the same argument that put protocol into the snapshot in M3 (Why a Snapshot Carries the Protocol).
The legacy limitation, stated here because it cannot be fixed here. A model or manufacturer filter reaches a LEGACY_ETL row only where the legacy ETL wrote a fcc_test_device fleet snapshot for its anchor campaign. If it does not, step two finds no campaign for that model and the imported half of the report is not there at all, not partially.
There is deliberately no fallback that looks the serial up in the live ftacs.cpe: that would answer with the model the device carries today, which for a measurement taken years ago is not an approximation, it is a different fact. Populating fcc_test_device for every anchor campaign is a requirement on the M4 legacy ETL, not a gap this read path can close on its own.
The ETL now meets that requirement, with one honest limitation it cannot remove. It snapshots a fleet for every anchor it creates, from the serials the measurements themselves carry — but the legacy rows carry only a serial, so the model_id it records is the model the CPE has at import time, and a serial no live cpe row matches is not snapshotted at all. See The Fleet Snapshot Is Mandatory and model_id Is the Model at IMPORT Time.
=== Two Different source Dictionaries — Do Not Merge Them
There are two, unrelated source columns in this system, and they must never be treated as one enum:
| Column | Values |
|---|---|
|
|
|
|
The Java-side enums mirror the split exactly: FccTestSourceEnum (ACS_STREAM, LEGACY) versus FccResultSourceEnum (ACS_STREAM, LEGACY_ETL) are two distinct types.
=== What a LEGACY_ETL Row Carries, and What It Cannot
An imported row fills the same columns a live one does, but several of them have no source in the legacy schema at all, and how each says so depends on whether the column is Nullable.
| Column | Value on a LEGACY_ETL row |
Why |
|---|---|---|
|
|
The columns are not |
|
|
|
|
|
|
|
The stored value, or |
|
|
Filled from the legacy columns |
Unlike a live latency row, which writes |
|
Whole seconds, always |
The legacy sub-second is a random draw the ACS stamped on, not a measurement, so it is dropped and nothing is put in its place. That is what makes two imports of one source row byte-comparable, and it is why an imported row’s |
|
|
|
|
When the import wrote the row |
Not when the measurement was taken. It is the partitioning key, which is exactly why [report-period-predicate] narrows on |
=== Flyway History Table
spring.flyway.enabled is false: Spring Boot’s own Flyway autoconfiguration never runs, and spring.flyway.table is never set at all. ClickHouseFlywayConfig builds a Flyway instance by hand instead — the community ClickHouse dialect stops at 10.24.0 while Boot manages a newer core — and it is that instance whose history table defaults to flyway_schema_history_fcc, never the Flyway default flyway_schema_history. ACS, ui-backend and ft-device-network all keep their own Flyway histories in this same shared ClickHouse database; a shared default history table name has already caused one incident (see ClickHouseFlywayConfig). The history table name is read from a property (ft.fcc.clickhouse.flyway.table) rather than hardcoded, unlike the raw-SQL self-healing block found in some sibling repos' FlywayConfig classes.
== Entity Diagram