Operations

Configuration Reference

ft.fcc.* Properties

Bound by FccProperties (@ConfigurationProperties("ft.fcc")), all overridable via environment variables in application.yaml.

Property Default Meaning

ft.fcc.command-ack-timeout (FCC_COMMAND_ACK_TIMEOUT)

60s

How long the dispatcher waits for a FccCommandAckDto before republishing — see Ack Timeout and the Republication Window.

ft.fcc.activation-max-delay (FCC_ACTIVATION_MAX_DELAY)

30m

Beyond this, a late-arriving ack no longer starts the run; the activation fails with ACS_UNAVAILABLE and a STOP is published defensively.

ft.fcc.stop-grace (FCC_STOP_GRACE)

15m

Three uses, one knob. (1) Maximum time an activation stays STOPPING waiting for inFlightCount to reach 0 before the watchdog forces it to STOPPED. (2) The after-stop window on the result path: a measurement reaching this service later than stopped_at + stop-grace is parked in fcc_result_dead_letter as LATE instead of being written to ClickHouse. The two are the same window seen from the two ends of it — see The After-Stop Window. (3) Since contract 1.5, the value is also copied, in whole seconds, into config.retry.stopDrainGraceSec of every ACTIVATE (FccCommandMapper), telling the ACS how long to hold in-flight results after a STOP; the default 15m therefore reaches the wire as 900. A change reaches only the next activation, never a run already dispatched — a republish re-sends the stored fcc_acs_outbox.payload JSON and never re-runs the mapper — see What Contract 1.5 Changed. Not the same knob as ft.fcc.watchdog.finish-grace, which bounds a RUNNING run overrunning its window; see [two-graces].

ft.fcc.max-devices-per-test (FCC_MAX_DEVICES_PER_TEST)

20000

Hard cap enforced by FccTestValidator.resolveDevices; see PUT /FCC/item/details. PER CONFIGURATION, not per campaign — one configuration is one ACS activation with one device list, so a campaign of several may name more devices in total, bounded by the knob below. The property keeps its name: it is set by FCC_MAX_DEVICES_PER_TEST in every deployed environment, and renaming a knob to re-describe it would silently revert installations that override it.

ft.fcc.max-configs-per-test (FCC_MAX_CONFIGS_PER_TEST)

10

How many configurations one campaign may carry — configs[] on PUT /FCC/item/details. IT BOUNDS THE SAVE AND NOT THE ACS: each configuration becomes its own activation with its own idempotency key, which the ACS serves without a change, but N configurations resolve N fleets of up to max-devices-per-test devices each in ONE transaction, and an unbounded N is how that save takes the service down. See Several Configurations, and Whose Limit Bounds Them.

ft.fcc.outbox-batch-size (FCC_OUTBOX_BATCH_SIZE)

100

Rows the dispatcher pulls per tick. Flat spelling kept alongside ft.fcc.outbox.batch-size below — see ft.fcc.outbox.* / ft.fcc.result.* / ft.fcc.dedup.* — Nested Scheduling and Buffering Properties.

ft.fcc.outbox-max-attempts (FCC_OUTBOX_MAX_ATTEMPTS)

10

Attempts (initial publish plus every ack-timeout republish) before FccOutboxPublisher marks the outbox row ABANDONED. Reaching ABANDONED also fails the paired activation in the same transaction (FccStalledActivationService.failAbandoned) — see The Transactional Outbox.

ft.fcc.hazelcast.cluster-name (HZ_CLUSTER_NAME)

dev

ft.fcc.hazelcast.members (HZ_MEMBERS_FALLBACK)

(empty)

Fallback list, only consulted when the HZ_MEMBERS environment variable is absent — see [hz-members].

ft.fcc.hazelcast.connect-timeout-ms (HZ_CONNECT_TIMEOUT_MS)

60000

ft.fcc.ftacs.schema-check.enabled (FTACS_SCHEMA_CHECK_ENABLED)

true

Disabled in the localdev profile group — a developer machine has no ftacs data to check against.

ft.fcc.iotw.enabled (IOTW_ENABLED)

true

Gates a feature, not a check. With it off, a campaign configuration naming a conditionId is refused with a 400 that says so, and the other three device source types are unaffected — see rest-api.adoc#saved-conditions. Disabled in the localdev profile group.

ft.fcc.iotw.schema-check.enabled (IOTW_SCHEMA_CHECK_ENABLED)

true

Checks the two iotw tables at startup and warns rather than failing — see The iotw Grant Check: The One That Warns Instead of Failing. Off leaves the lazy pool closed until the first condition read, with the feature still working. Disabled in the localdev profile group.

ft.fcc.clickhouse.enabled (CLICKHOUSE_ENABLED)

true

Gates both ClickHouseFlywayConfig and ClickHouseMigrationRunner.

ft.fcc.clickhouse.health.cache-ttl (CLICKHOUSE_HEALTH_CACHE_TTL)

10s

/actuator/health is polled far more often than a live ClickHouse probe is worth; the result is cached for this long. It is also the interval of the scheduled refresh in ClickHouseHealthIndicator, which keeps the DEGRADED gate bounded in staleness even where nothing polls the endpoint; while ClickHouse is already known to be down that scheduled probe backs off to one attempt a minute.

ft.fcc.clickhouse.health.query-timeout-sec (CLICKHOUSE_HEALTH_QUERY_TIMEOUT_SEC)

3

ft.fcc.clickhouse.health.schema-check-interval (CLICKHOUSE_HEALTH_SCHEMA_CHECK_INTERVAL)

5m

M4. How often a UP ClickHouse is asked whether migrations are still pending. Much longer than cache-ttl on purpose: the answer cannot change without a JVM restart or a successful migration retry, and asking every ten seconds would add a flyway_schema_history_fcc read per probe for a question whose answer is almost always the one from last time. Once something is pending the check runs on every probe instead, so a retry that fixes the schema is noticed within cache-ttl rather than up to five minutes later — see the three ClickHouse states.

ft.fcc.clickhouse.migration.retry-interval (CLICKHOUSE_MIGRATION_RETRY_INTERVAL)

60s

M4. How often ClickHouseMigrationRunner retries the migration while the status is not UP. A no-op tick while it is — see [clickhouse-migration-retry].

ft.fcc.clickhouse.flyway.locations (CLICKHOUSE_FLYWAY_LOCATIONS)

classpath:db/clickhouse

ft.fcc.clickhouse.flyway.table (CLICKHOUSE_FLYWAY_TABLE)

flyway_schema_history_fcc

Never the Flyway default — see Flyway History Table.

spring.mvc.async.request-timeout (FCC_EXPORT_REQUEST_TIMEOUT)

10m

M4. Not an ft.fcc.* property, but the one Boot setting POST /FCC/reports/export cannot leave at its default. That endpoint answers with a StreamingResponseBody, which makes the request asynchronous, and Tomcat aborts an async request after its own default of 30 seconds — far less than a large regulatory extract legitimately takes. The abort arrives after the status line has already been sent, so without this property the operator gets a truncated file with no error anywhere, and no client-side timeout can detect or fix it. The frontend brief already requires a client timeout of at least five minutes; this is the server side of the same number, with headroom. See [export-truncated].

ft.fcc.outbox.* / ft.fcc.result.* / ft.fcc.dedup.* — Nested Scheduling and Buffering Properties

Bound by the nested records of FccProperties (Outbox, Result, Dedup). outboxBatchSize and outboxMaxAttempts above keep their original flat spelling because an installation may already set them; these nested blocks were added later for the scheduling and buffering knobs that did not exist yet.

Property Default Meaning

ft.fcc.outbox.dispatch-interval (FCC_OUTBOX_DISPATCH_INTERVAL)

5s

How often FccOutboxDispatcher claims and offers PENDING rows.

ft.fcc.outbox.ack-timeout-interval (FCC_OUTBOX_ACK_TIMEOUT_INTERVAL)

30s

How often FccAckTimeoutPoller re-checks SENT rows without an ack, and (same tick) checks for stalled activations — see Ack Timeout and the Republication Window.

ft.fcc.outbox.batch-size (FCC_OUTBOX_BATCH_SIZE)

100

Same environment variable as the flat ft.fcc.outbox-batch-size above — one knob, two binding paths, so overriding either overrides both.

ft.fcc.result.batch-size (FCC_RESULT_BATCH_SIZE)

100

Rows per ClickHouse INSERT; the FT house figure for ClickHouse batch size.

ft.fcc.result.flush-interval (FCC_RESULT_FLUSH_INTERVAL)

5s

The result buffer flushes at batch-size rows OR this interval, whichever comes first — see Result Ingest.

ft.fcc.result.queue-capacity (FCC_RESULT_QUEUE_CAPACITY)

10000

Bound of the in-memory result buffer, which also serves as the ClickHouse retry queue. Exhausting it flips the service to DEGRADED and parks the rows it cannot hold in fcc_result_dead_letter (fcc.result.parked) rather than dropping them — see Troubleshooting.

ft.fcc.dedup.ttl (FCC_DEDUP_TTL)

1h

Lifetime of one entry in the fccResultDedup Hazelcast map — see De-Duplication Is One Atomic putIfAbsent.

ft.fcc.scheduler.* / ft.fcc.watchdog.* / ft.fcc.dead-letter.* — M2

Bound by the nested Scheduler, Watchdog and DeadLetter records of FccProperties.

Property Default Meaning

ft.fcc.scheduler.poll-interval (FCC_SCHEDULER_POLL_INTERVAL)

30s

How often FccActivationScheduler ticks. One tick, two passes: due SCHEDULED campaigns and due PLANNED reactivations. They share a cadence because both only answer "is anything due yet", and a due run picked up half a minute late is invisible next to a measurement window of hours.

ft.fcc.scheduler.batch-size (FCC_SCHEDULER_BATCH_SIZE)

100

Rows claimed per tick, per pass. It bounds the work of one tick, not the backlog: whatever is left over is claimed next tick, and a claim that comes back full is the signal that the interval is too long for the installation rather than a reason to raise this. Capped internally at FccDbConstants.IN_CLAUSE_MAX_EXPRESSIONS (1000), since the claimed ids become an Oracle IN list.

ft.fcc.watchdog.poll-interval (FCC_WATCHDOG_POLL_INTERVAL)

60s

How often FccActivationWatchdog ticks. Slower than the scheduler on purpose: it closes work rather than starting it, so a minute of latency costs nothing.

ft.fcc.watchdog.batch-size (FCC_WATCHDOG_BATCH_SIZE)

100

Runs claimed per tick, per pass (expired RUNNING, expired STOPPING). Same capping as the scheduler.

ft.fcc.watchdog.finish-grace (FCC_WATCHDOG_FINISH_GRACE)

15m

How long past window_end_at a RUNNING run is still given to produce its own ACTIVATION_FINISHED before the watchdog closes it as FINISHED(WATCHDOG). A separate knob from ft.fcc.stop-grace — see [two-graces].

ft.fcc.dead-letter.resend-batch-size (FCC_DEAD_LETTER_RESEND_BATCH_SIZE)

500

Parked measurements read per POST /FCC/admin/result-dead-letter/resend call. It is both the default and the ceiling: a larger requested limit is clamped to it, because one call must not pull an unbounded backlog into the in-memory result buffer and re-trigger the very overflow the dead letter exists to absorb. Far larger than the ClickHouse insert batch (100) on purpose: a dead letter is only written after an outage, so the backlog is a burst of thousands and draining it in hundred-row reads would take longer than the outage did.

There is no spring.task.scheduling.pool.size / FCC_SCHEDULING_POOL_SIZE any more, and it never did anything in this service — see [scheduling-pool]. As M5 first documented it, this table listed it here as "set explicitly to 5" — M5 having raised it from the M2-era default of 4 alongside the new metrics sampler; the property only ever feeds ThreadPoolTaskSchedulerBuilder, which this service never builds, because spring.threads.virtual.enabled: true makes Boot build a SimpleAsyncTaskScheduler instead, so that raise changed nothing. The M5 code review found this and removed the key; it is called out here, at the point a reader would otherwise go looking for it, rather than only in [scheduling-pool].

ft.fcc.resolve.* and the Multipart Limits — M3, extended M5

The device-resolution endpoints of the campaign wizard. See Device Resolution for what they bound.

Property Default Meaning

ft.fcc.resolve.default-page-size (FCC_RESOLVE_DEFAULT_PAGE_SIZE)

100

Page size of POST /FCC/devices/resolve when the caller names none. It sizes the grid the operator looks at, not the fleet. Bound by the nested Resolve record of FccProperties.

spring.servlet.multipart.max-file-size (FCC_SERIAL_FILE_MAX_SIZE)

10MB

Largest file part the multipart variant of /FCC/devices/resolve, /FCC/devices/count and, since M5, PUT /FCC/item/import and PUT /FCC/devices/import accepts. The upload is read whole into memory (MultipartFile.getBytes()), and this service has no authentication of its own, so an unbounded upload is a denial of service anybody on the docker network can trigger. 10 MB is far more than a 20 000-serial list needs — a 20-character serial per line is roughly 400 KB. + The M5 arithmetic on the same 10 MB is a different one, and it is worth stating on its own: a full 20 000-device campaign exports at roughly 720 KB, so about fourteen of them exhaust this limit, and the service would refuse a document it produced itself. Raising the value is still the wrong fix, because it would raise the same limit’s denial-of-service ceiling on POST /FCC/devices/resolve by the same amount — which is the reason the limit exists in the first place. Split a large export into several imports instead. This limit also bounds only the bytes of the document. There is no second, element-count bound any more and none is needed: PUT /FCC/item/import answers one campaign, so a second <test> element is a 400 before anything is parsed, and ft.fcc.config-import.max-tests — which bounded the many-campaign import this endpoint used to be — is gone with it.

spring.servlet.multipart.max-request-size (the same FCC_SERIAL_FILE_MAX_SIZE)

10MB

Largest whole multipart request: the file part plus the optional request part. One environment variable feeds both, so raising the limit raises it coherently — setting only the file size and leaving the request size at Boot’s default would make the pair silently inconsistent.

There is deliberately no maximum-page-size knob for device resolution.

The ceiling on a page is ft.fcc.max-devices-per-test, not a knob of its own: the wizard has to be able to ask for the whole resolved fleet in one call in order to save the test, and the read is bounded by that same limit anyway. A second knob could only ever be set to a value that makes saving a full fleet impossible — and it would be discovered at that moment, by an operator whose campaign will not save.

The same reasoning is why FccProperties.Resolve carries one field.

Exceeding the limit answers HTTP 413 with FT code 9413, and the message names the limit when one is knowable.

FccExceptionHandler carries a dedicated MaxUploadSizeExceededException handler:

413 and not 400, deliberately: the upload was well formed and there is nothing in it for the caller to correct except its size, so the limit is what the answer has to name.

{"code": 9413, "message": "the uploaded file is too large",
 "description": "the upload exceeds the configured maximum of 10485760 bytes;
                 send a shorter serial list, or split the fleet into several tests"}

The figure comes from the configuration, not from the exception — and that is not a detail. The handler used to render MaxUploadSizeExceededException.getMaxUploadSize(), which reads naturally and is wrong here: that accessor is populated by whichever multipart parser raised the exception, and Tomcat does not populate it at all. The limit is enforced by the connector, which raises the exception without carrying the figure back, so the accessor hands back -1 — and a real caller was told "the upload exceeds the configured maximum of -1 bytes". The status, the code and the remedy sentence were right the whole time; only the number was nonsense, which is exactly the kind of defect that survives a review.

FccExceptionHandler now injects MultipartProperties and reads spring.servlet.multipart.max-file-size. That is also simply the better number to print: it is the one an operator can go and change, whereas the parser’s figure — when a parser bothers to supply one — is whatever that parser happened to enforce.

A limit neither source knows is left out of the message rather than printed.

The exception’s own figure is kept as a fallback for the case where the property is unset, and if that is missing too the sentence is emitted without any number at all: "the upload exceeds the configured maximum; send a shorter serial list…​".

A remedy with no number beats a remedy with a wrong one — the whole -1 incident was a wrong number being more convincing than no number. In this service the fallback is unreachable in practice, because application.yaml binds the property to ${FCC_SERIAL_FILE_MAX_SIZE:10MB} and a default is therefore always present; it exists for the deployment that overrides the property with an empty value.

A client must not parse the byte figure out of the description. It is prose, it is not guaranteed to be there, and the machine-readable part of the answer is code: 9413.

Three Spring-MVC handlers on the multipart path, and none of them is redundant. Spring resolves each of these into an exception whose default status it knows perfectly well, but DefaultHandlerExceptionResolver is the last resolver in the chain while an @ExceptionHandler(Exception.class) in an advice runs before it — so the catch-all masks Spring’s own answer and turns every one of them into a 500 with no hint of what the caller sent wrong.

Mistake Answer Note

Upload larger than the limit

413 / 9413

Names the configured maximum in bytes, read from spring.servlet.multipart.max-file-size and not from the exception — and omits the figure entirely if no source can state one.

No file part in the multipart request

400 / 9400

MissingServletRequestPartException. The message states that file is required and the JSON request part is not.

A part offered with a content type nothing accepts

415 / 9415

HttpMediaTypeNotSupportedException. Most often the optional request part sent with no content type of its own — which is curl’s default for -F, so it is offered as text/plain and no converter takes it. The message says to give each part its own Content-Type, application/json for the request part.

9413 and 9415 follow the catalog’s own convention: the last three digits are the HTTP status the failure is answered with. See the one exception to that rule.

ft.fcc.stop-grace and ft.fcc.watchdog.finish-grace are two knobs, deliberately, and raising one must not raise the other.

Knob Bounds It is a property of

ft.fcc.stop-grace

a STOPPING run draining its in-flight devices

how fast the ACS reacts to a command

ft.fcc.watchdog.finish-grace

a RUNNING run overrunning its measurement window

how long a single diagnostic takes

finish-grace exists because the end of the window and the end of the last iteration are not the same instant — the ACS finishes the diagnostic it has already started, then reports — and closing on the window boundary alone would relabel a perfectly normal completion as a watchdog kill.

Tying the two together would mean that making stops more patient also delays every watchdog close. They both default to 15m, which is a coincidence of the defaults and not a relationship between them.

ft.fcc.legacy-import.* — M4

Bound by the nested LegacyImport record of FccProperties. They tune the legacy ETL — see Legacy FCC Import for what it does and The Legacy Import Runbook for how to drive it.

None of these makes the import run. There is no poller and no schedule: the import starts only from POST /FCC/admin/legacy-import/start, which is why it does not appear in The Five Pollers.

Property Default Meaning

ft.fcc.legacy-import.enabled (FCC_LEGACY_IMPORT_ENABLED)

true

Whether /start is allowed at all. false answers 400 with a message saying so, and /status keeps answering either way — an installation that has finished its migration turns the start path off and can still read what was imported. It does not disable or hide anything already in ClickHouse.

ft.fcc.legacy-import.batch-size (FCC_LEGACY_IMPORT_BATCH_SIZE)

10000

Rows per ftacs read, per ClickHouse insert and per cursor commit — the three bracket the same rows, because together they are the two-phase commit that makes a crash recoverable (The Two-Phase Cursor Commit). A smaller value means more frequent commits and a smaller range to re-check after a crash, at the cost of throughput. This is a default, not a ceiling: /start may raise or lower it for one run, bounded internally at 100000 because the batch is held in memory as mapped rows before it is inserted. (The opposite convention to ft.fcc.dead-letter.resend-batch-size, where the configured value is the ceiling — there a larger batch can re-trigger the overflow the dead letter exists to absorb; here it only costs memory.)

ft.fcc.legacy-import.zone (FCC_LEGACY_IMPORT_ZONE)

(empty)

The time zone the naive legacy timestamps are read in. Empty means the JVM’s own zone, never UTC by default. start_date, end_date, bom and eom carry no offset anywhere, so this decides which instant every imported measurement happened at; set it to the zone the ACS wrote them in whenever this service does not run in it. The value actually used is recorded in fcc_legacy_import.zone_id and returned by /status, and a resumed pass keeps the zone it began with regardless of this property — see Time Zone Is a Run Parameter. An unparseable value is a 400 on /start rather than a silent fallback.

ft.fcc.legacy-import.window-settle (FCC_LEGACY_IMPORT_WINDOW_SETTLE)

1h

Grace after a day-window closes before its measurements are believed complete. Only closed windows are imported: the iteration rank of a window still receiving rows would be renumbered by a later pass, and nothing can be deleted from a ClickHouse MergeTree — Only Closed Day-Windows Are Imported. It is not the whole lag. A pass reads up to started_at - window-settle - 24h, so with the default a measurement is first importable about 25 hours after it is taken; raising this property raises that by the same amount. The extra day is what makes one pass-wide cutoff correct for campaigns whose day-windows start at different hours — see Expect Roughly 25 Hours of Lag on the Newest Measurements. Raising it never affects what has already been imported.

ft.fcc.legacy-import.stale-after (FCC_LEGACY_IMPORT_STALE_AFTER)

5m

How long a RUNNING import may go without a heartbeat before another /start may take its row over. This is the takeover window after a process was killed mid-import — a live import heartbeats between batches and over the slow non-batch work too (the per-campaign MAX(id) prologue, the anchor resolve, the fleet top-up), so it never approaches it. Lowering it below the duration of one batch would let a second start claim a kind that is still being imported. The takeover itself is safe rather than merely unlikely: every mutating statement is fenced on run_id, so the evicted worker’s next write matches no row and it aborts — The Claim Is Enforced on Every Write.

The Five Pollers

Poller Every What it claims and does

FccOutboxDispatcher → FccOutboxPublisher

5s

fcc_acs_outbox WHERE state = 'PENDING' AND next_attempt_at ⇐ now. Offers each payload to fccCommandQueue and moves the row to SENT only after a successful offer().

FccAckTimeoutPoller → FccOutboxPublisher + FccStalledActivationService

30s

fcc_acs_outbox WHERE state = 'SENT' AND sent_at ⇐ now - command-ack-timeout, republished with the same key and seq; and, on the same tick, fcc_activation WHERE state = 'DISPATCHING' AND window_start_at ⇐ now - activation-max-delay, failed as ACS_UNAVAILABLE with a defensive STOP. PLANNED is deliberately excluded — see Ack Timeout and the Republication Window.

FccActivationScheduler → FccActivationSchedulingService, pass 1

30s

fcc_test WHERE state = 'SCHEDULED' AND scheduled_at ⇐ now. Starts the campaign through the ordinary FccActivationHandler.activate, with no identity — userId = null, so the audit row says the engine did it and the audit columns fall back to the system auditor.

FccActivationScheduler → FccActivationSchedulingService, pass 2

30s

fcc_activation WHERE state = 'PLANNED' AND window_start_at ⇐ now. Dispatches the daily reactivation planned 24 hours earlier, re-taking the settings snapshot on the way out — which is what makes an edit apply from the next reactivation. A row dispatch refuses is closed as STOPPED(WATCHDOG) in the same transaction rather than left PLANNED, or it would be re-claimed every tick forever — see A Skip Is Not a "Try Again Next Tick".

FccActivationWatchdog → FccActivationClosingService, two passes

60s

fcc_activation WHERE state = 'RUNNING' AND window_end_at ⇐ now - watchdog.finish-grace → FINISHED(WATCHDOG); and WHERE state = 'STOPPING' AND stopped_at ⇐ now - stop-grace → STOPPED.

Four more scheduled methods exist that are not pollers over a claim: FccResultIngestService.flushTick (ft.fcc.result.flush-interval, 5s), the ClickHouseHealthIndicator refresh (ft.fcc.clickhouse.health.cache-ttl, 10s), ClickHouseMigrationRunner.retry (ft.fcc.clickhouse.migration.retry-interval, 60s — see [clickhouse-migration-retry]; a no-op the moment the status is UP, so it costs a pool slot only during a ClickHouse outage), and, since M5, the metrics sampler FccMetricsSampler.sample (ft.fcc.metrics.sample-interval, FCC_METRICS_SAMPLE_INTERVAL, default 15s), which refreshes every fcc_queue_* / fcc_outbox_* gauge /actuator/prometheus reads from — see why the gauges are sampled rather than computed on scrape.

There is no scheduler thread pool in this service, and spring.task.scheduling.pool.size (FCC_SCHEDULING_POOL_SIZE) is gone — an M5 code review found it had never done anything here.

spring.threads.virtual.enabled: true is set in application.yaml, and that single switch also decides which scheduler Boot builds: with it on, TaskSchedulingConfigurations$TaskSchedulerConfiguration is @ConditionalOnThreading(VIRTUAL) and picks taskSchedulerVirtualThreads, which constructs a SimpleAsyncTaskScheduler — not a ThreadPoolTaskScheduler. spring.task.scheduling.pool.size only ever feeds ThreadPoolTaskSchedulerBuilder, on the platform-thread branch this service never takes, so the key was dead configuration from the day virtual threads were switched on, and M5’s raise of its value from 4 to 5 (alongside the metrics sampler) changed nothing. This was verified by decompiling the exact autoconfiguration class this project resolves, not by reading the reference documentation. The property the scheduler actually reads, spring.task.scheduling.simple.concurrency-limit, is deliberately left unset — unset means unbounded, so every tick below gets a virtual thread of its own rather than queueing behind a fixed-size pool.

The eight-tick inventory this note used to justify a pool size for is still accurate and still worth keeping, because it is what makes "unbounded" the right answer rather than a risk: the outbox dispatcher (5s), the result flush (5s), the ClickHouse health probe (10s), the metrics sampler (FccMetricsSampler, 15s, since M5), the ack-timeout poller (30s), the activation scheduler (30s), the run watchdog (60s) and the ClickHouse migration retry (60s, a no-op while the store is UP). The three campaign ticks are not short — each claims up to a hundred rows and then, in one transaction, applies transitions, plans reactivations and writes STOP commands — and it is exactly because they are not short that a shared, bounded pool would have been the wrong design: a watchdog tick stuck in a connection timeout would park its own virtual thread and delay nothing else, whereas a concurrency limit of N puts every tick beyond the N-th back in a queue behind it — the starvation a pool size was always meant to prevent, reproduced by the very knob meant to prevent it. Do not set spring.task.scheduling.simple.concurrency-limit without a measured reason, and if an incident response ever reaches for "raise the scheduling pool", it will change nothing — see the note at ft.fcc.* Properties above.

Every one of these claims with the two-step FOR UPDATE SKIP LOCKED pattern of Vendor-Specific Claim SQL, holds the claim and the writes in one transaction, and lives in a bean separate from its @Scheduled tick — see The Scheduler / Transaction Split for why both of those are load-bearing rather than stylistic.

A failing tick is caught, counted and throttled (FccFailureThrottle): the first occurrence of a failure signature is logged at WARN in full, repetitions of the same signature at DEBUG, and the signature is re-logged at WARN once REPEAT_AFTER (5 minutes) has elapsed since the last WARN — a recovery resets the throttle outright, so the next outage is loud again from its own first tick. An unreachable database during a scheduler tick therefore produces one WARN every five minutes for the length of the outage, not one per 30-second tick — an hour-long outage writes about twelve WARN lines, not a hundred and twenty. There is no poller for fcc_result_dead_letter: draining it is operator-triggered, through POST /FCC/admin/result-dead-letter/resend.

HZ_MEMBERS (the raw environment variable, read directly, not through Spring binding) wins over ft.fcc.hazelcast.members — this is the docker-compose convention shared with the ACS and ui-backend. A bare host with no port gets :5701 appended (HazelcastClientConfig.resolveMembers). If neither is set, the client falls back to 127.0.0.1:5701.

Hazelcast Structures This Service Uses

Beyond the general client configuration above, this service touches a fixed, small set of Hazelcast structures on the shared hazelcast:ft cluster — see External Dependencies — Exactly Five and Hazelcast Queues for the full queue contract.

Structure Kind Role

fccCommandQueue

Queue (produced to)

FccOutboxPublisher offers FccCommandDto here; the ACS’s fcc-core module consumes it.

fccCommandAck

Queue (consumed)

Dedicated take() thread, FccAckListener / FccAckService.

fccResultQueue

Queue (consumed)

Dedicated take() thread, FccResultListener / FccResultIngestService.

fccDeviceStateQueue

Queue (consumed)

Dedicated take() thread fcc-device-state-listener, FccDeviceStateListener / FccDeviceStateService. One atomic upsert batch per message into fcc_device_state, plus the STOPPING drain recount.

fccActivationEventQueue

Queue (consumed)

Dedicated take() thread fcc-activation-event-listener, FccActivationEventListener / FccActivationEventService. The control channel: ACTIVATION_STARTED / _FINISHED / _FAILED / _STOPPED.

fccResultDedup

IMap<String, Boolean>

Result de-duplication key, see De-Duplication Is One Atomic putIfAbsent. Unlike the queues, this map is internal to ft-fcc-service and is not shared or prefixed by the ACS’s cache factory (FccMapNames).

fccResultDedup needs a max-size with an eviction policy configured in the cluster’s hazelcast.xml, in addition to the per-entry ft.fcc.dedup.ttl this service sets. hazelcast.xml lives in the ft-products-compose repository (per FccMapNames’s own operations note, maintained by the `ui-backend author), not this one — a per-entry TTL alone bounds how long an entry lives, not how many entries a sudden burst of results can create before the map itself needs to be capped. This is a cross-repo action item, not something FccProperties can express.

app.ds.* Datasources

Four Hikari-backed DataSources, each configured under app.ds.<name> with its Hikari block at app.ds.<name>.hikari (do not cross-wire these prefixes — a documented live bug in a sibling service’s DataSourcesConfig is exactly this kind of mismatch, and this service’s DataSourcesConfig javadoc calls it out explicitly so it is not repeated here).

DataSource Access Pool notes

app.ds.ftfcc

Read-write, @Primary

maximum-pool-size from vars.pool.ftfcc.max (default 20), minimum-idle default 5.

app.ds.ftacs

Read-only

read-only: true in the Hikari config and dataSource.setReadOnly(true) in code (DataSourcesConfig.ftacsDataSource) — belt and suspenders. Default pool 10/2.

app.ds.clickhouse

Read-write, but the bean is @Lazy

initialization-fail-timeout: -1 so a down ClickHouse never blocks the pool from being created; minimum-idle is deliberately 0 so an outage-tolerant pool does not hammer an unreachable ClickHouse in the background. Default max pool 10.

app.ds.iotw

Read-only, and the bean is @Lazy

The portal’s schema, two tables, SELECT only (the grant). read-only: true in the Hikari config and in code (DataSourcesConfig.iotwDataSource), with a small pool — 4/1 by default, IOTW_MAX_POOL/IOTW_MIN_POOL — because it is consulted only when a campaign configuration names a conditionId, one or two SELECTs per save. Lazy all the way through: an installation with no grant, or one with ft.fcc.iotw.enabled=false, starts normally and refuses only that source type. There is deliberately no transaction manager for it — reading a condition is one or two statements with no invariant between them, and a second manager would tempt somebody to join this read into an ftacs transaction, which no two-phase-less pair of datasources can honour.

An Oracle URL on any of these DataSources gets ALTER SESSION SET TIME_ZONE='UTC' and oracle.jdbc.timezoneAsRegion=false applied automatically (DataSourcesConfig.initDataSource). app.ds.iotw additionally gets ALTER SESSION SET CURRENT_SCHEMA=<schema> on Oracle, because the two iotw tables belong to their own user there and every query in IotwConditionDao is unqualified.

mysql / oracle / local Profile Groups

application.yaml defines three Spring profile groups:

Group Composition

mysql (default, SPRING_PROFILES_ACTIVE default)

common + mysql — MySQL dialect and JDBC URLs for both ftfcc and ftacs.

oracle

common + oracle — Oracle dialect and JDBC URLs; both schemas are expected to live in the same Oracle service (FTACS19 by default) under different Oracle schema/users.

local

common + mysql + localdev — additionally disables the ftacs schema check and points the Hazelcast client at 127.0.0.1:5701, and raises com.friendly.fcc logging to DEBUG.

ClickHouse Flyway Properties

Flyway for ClickHouse is built programmatically in ClickHouseFlywayConfig, with Boot’s own Flyway autoconfiguration switched off (spring.flyway.enabled: false). This is because the community ClickHouse Flyway plugin (flyway-database-clickhouse:10.24.0) is built against flyway-parent 11.9.0, while Spring Boot 3.5.6 manages a different flyway-core line — pom.xml pins flyway-core to 11.7.2 and the ClickHouse plugin to 10.24.0 explicitly rather than letting Boot’s dependency management pick a mismatched pair. The Flyway instance’s baselineOnMigrate(true), validateOnMigrate(false) and outOfOrder(true) are all set explicitly in code, not through properties.

Health

GET /actuator/health exposes health, info and prometheus. The health status order is explicitly restated in application.yaml as DOWN, OUT_OF_SERVICE, DEGRADED, UP, UNKNOWN with an HTTP mapping that replaces Boot’s default:

Status HTTP

DOWN, OUT_OF_SERVICE

503

DEGRADED

200

UP, UNKNOWN

200

DEGRADED maps to HTTP 200 on purpose. If it were left unmapped (falling back to Boot’s default alongside DOWN), the service would answer 503 for a condition it is explicitly designed to survive, and container orchestration would restart a still-useful instance.

Since M5, the aggregate /actuator/health carries two more components (livenessState, readinessState, once management.endpoint.health.probes.enabled: true) and can answer 503 during a normal graceful shutdown — a status this endpoint never produced before M5, on a different status (OUT_OF_SERVICE, not DEGRADED) than the one this section describes. This is deliberate, not a regression, and it is documented once, at the deployment page’s probe section, rather than restated here.

DEGRADED means FccDegradedState.clickhouseStatus() is not UP. Through M3 that state was a boolean; M4 replaced it with FccClickHouseStatusEnum, three values, because "the result store cannot be used" has always had two different causes with two different remedies, and a boolean could only ever say one of them:

FccClickHouseStatusEnum What it means Operator’s next action

UP

Reachable, and every ClickHouse migration this build carries has been applied.

None.

SCHEMA_BEHIND

Reachable, but migrations are pending — the connecting user has no CREATE/ALTER on the shared ftacs_qoe_ui_data, or a startup migration failed and its retry has not yet succeeded.

Grant the missing privilege, or investigate why the retry keeps failing. Restarting the instance does not help by itself: the migration is already retried on a timer regardless — see ClickHouse Migration at Startup.

UNAVAILABLE

Not reachable at all: the connectivity probe, or a ClickHouse batch insert, failed.

Check that ClickHouse itself is up and reachable from this service’s network. The ordinary "the database is down" case.

Both non-UP states report the same /actuator/health status (DEGRADED, HTTP 200, per the mapping above) and close the same gate — see [gate-refuses]. They are told apart only by details.reason, and it is worth reading before reaching for a fix: restarting a database that is merely SCHEMA_BEHIND does nothing, and granting a privilege to a ClickHouse that is UNAVAILABLE cannot even connect to apply it.

details.reason never contains the enum names SCHEMA_BEHIND or UNAVAILABLE, and never contains the sentences "the result store is unreachable" or "the result store schema is behind: pending ClickHouse migrations". Those two sentences are constants of FccResultStorageGate (UNREACHABLE, SCHEMA_BEHIND) used to build the 503 body a caller of POST /FCC/activate or a reporting endpoint receives — they are not what /actuator/health prints. details.reason is built at ClickHouseHealthIndicator.pendingMigrations() / .describePending() from what Flyway actually reports pending (e.g. "1 clickhouse migration(s) pending: 3", naming the version), or, for an unreachable database, from the raw driver exception message (ClickHouseHealthIndicator.describe(Exception)). An operator grepping health output for either quoted sentence above, or for the literal word SCHEMA_BEHIND, finds nothing there — only in the 503 body of an activation or a report request. The status field (DEGRADED) is the same either way; the fact of which of the two states applies has to be read from the shape of details.reason itself (a migration count means SCHEMA_BEHIND, a driver error means UNAVAILABLE), not from a literal name.

 
{"status": "DEGRADED",
 "components": {"clickhouse": {"status": "DEGRADED",
   "details": {"reason": "1 clickhouse migration(s) pending: 3"}}}}

FccDegradedState is written from three independent places: ClickHouseHealthIndicator (a cached connectivity probe — SELECT 1, TTL ft.fcc.clickhouse.health.cache-ttl — followed, once connectivity is confirmed, by a look at Flyway’s pending migrations, cached separately and far longer at ft.fcc.clickhouse.health.schema-check-interval), ClickHouseMigrationRunner at startup and on its own retry timer, and FccResultIngestService itself when a ClickHouse batch insert fails or the in-memory result buffer is exhausted — see Result Ingest. The ingest path and the migration runner only ever set UNAVAILABLE, never SCHEMA_BEHIND: neither knows why an insert or a migration failed, and guessing a classification from driver wording would be wrong the first time the wording changes. Only the health indicator’s own probe — the one place that can actually ask Flyway info().pending() — is entitled to report SCHEMA_BEHIND, and it refines a plain UNAVAILABLE into that reading on its next probe once connectivity turns out to be fine.

The operational consequence: CRUD keeps working (creating, editing and listing tests touches only ftfcc), but every operation that depends on the result store is refused with 503 / FT code 9503 — FccResultStorageGate.require(…​) checks FccDegradedState.clickhouseStatus() before it starts anything and throws FccDegradedException whenever it is not UP. Since M4 that is not only POST /FCC/activate and the dead-letter resend, but all four /FCC/reports/** endpoints too: a result that cannot ultimately be written to ClickHouse should not be allowed to start, and a report should not be produced as though the store held everything it should. See POST /FCC/activate and Endpoints Implemented in M4.

The db health contributor is replaced by DataSourceHealthConfig so that only ftfcc and ftacs are probed under it; ClickHouse is deliberately excluded from the generic db contributor and reported only through the dedicated DEGRADED indicator, so a ClickHouse outage does not drag the aggregate status to DOWN. Hazelcast’s own health indicator is disabled (management.health.hazelcast.enabled: false): the client starts asynchronously by design (ClientConnectionStrategyConfig.setAsyncStart(true)), so "not connected yet" is an expected transient state during startup, not a reason to report DOWN.

Startup Behaviour

ftacs Read-Only Schema Check

FtacsSchemaCheckRunner runs as an ApplicationRunner (gated by ft.fcc.ftacs.schema-check.enabled, on by default, off in localdev). It detects the vendor from DatabaseMetaData.getDatabaseProductName() and queries information_schema.columns (MySQL) or ALL_TAB_COLUMNS (Oracle, resolved against app.ds.ftacs.schema or the connected user) for every column this service reads. A missing table or column throws IllegalStateException and fails startup with the full list of missing table.column entries, rather than surfacing as a broken query mid-campaign.

Only columns some statement actually names are listed, and that is a rule rather than an accident of maintenance. This check aborts startup, so every entry is a reason the service refuses to come up at an installation; requiring a column no query reads turns a harmless schema difference into an outage. The M4 review found several such entries and removed them — see [schema-check-trimmed].

The exact columns checked today (FtacsSchemaCheckRunner.REQUIRED_COLUMNS), with the M3 additions marked in bold and the M4 ones — everything the legacy import reads — marked as such:

Table Read by Columns

cpe

Device resolution, legacy fleet snapshot

id, serial, location_id, product_class_id, is_online, ` `protocol_id, firmware

product_class

Device resolution, catalog

id, model, manuf_id, group_id

product_class_group

Device resolution, catalog

id, manufacturer_name, product_class, ` `protocol_id

manufacturer (whole table, new in M3)

Catalog, device grid

id, name

ug_fcc_speed_result

Legacy import

id, action_id, serial, tier, ` `type, url, ` `speed, start_date, end_date, total_bytes, status, bom, eom, hubb_id, description

ug_fcc_latency_result

Legacy import

id, action_id, serial, tier, ` `url, start_date, val_avg, val_min, val_max, packets_sent, packets_received, status, hubb_id, description. No end_date, type, bom or eom is required — unlike the speed table, this one has none of them.

ug_fcc (whole table, M4)

Legacy import

id, ug_task_id, threshold_dl, threshold_ul, tier_dl, tier_ul, hubb_id

ug_task (whole table, M4)

Legacy import

id, ug_id, task_type

update_group_child

Legacy import

id, parent_id

update_group

Legacy import

id, location_id, ` `name, online_cpe_only, push

fcc_properties (whole table, M4)

Legacy import

id, ug_id, total_duration, single_duration, batch_interval, retry_failed, retry_wan

ug_reactivate (whole table, M4)

Legacy import

id, ug_id, expression

The four whole tables marked M4 are all in the contract’s §8 grant list already, so unlike the M3 manufacturer addition they need no grant change on a correctly provisioned account — see [legacy-import-grant].

The M4 review removed ten columns from this set, none of which any statement reads.

update_group.state, .scheduled, .created, .creator; ug_reactivate.repeat_count, .from_date, .to_date, .fail_only; ug_fcc.type; and update_group_child.group_id.

They were listed on the reasonable-looking principle that the import "reads those tables", but the gate is not about tables — it is about the exact table.column pairs FtacsLegacyGroupDao, FtacsLegacyResultDao, FtacsDeviceResolveDao and FtacsCatalogDao select, join or filter on. Each surplus entry was a way for an installation that had everything this service needs to be refused startup over a column difference that could not affect it.

Two are worth naming individually. ug_fcc.type records the speed/latency distinction a second time on the settings row, and nothing reads it: the traversal already reaches that row through ug_task.task_type. ug_reactivate is consulted for exactly one thing, the expression that gives the hour a day-window starts at — the repeat count and the date range of a legacy schedule are not part of the reconstruction.

The complement of this rule is ug_fcc_speed_result.bom / eom, which are read and therefore do belong here even though they make ACS 6.4.12 a hard prerequisite — see [acs-minimum-version].

The rule above has a live exception, and it is worth stating rather than restating a rule that has a hole in it. FtacsSchemaCheckRunner (FtacsSchemaCheckRunner.java:141) also requires product_class_group.product_class, and no statement in this service — not FtacsCatalogDao, not FtacsDeviceResolveDao, not either legacy DAO — selects, joins or filters on it; FtacsCatalogDao reads product_class_group only for id, manufacturer_name and protocol_id (g.id, g.manufacturer_name, joined on g.id = pc.group_id). The column stayed in the required set through the same M4 review that removed the ten columns above, most likely because a table already required for other columns is an easy one to under-trim rather than a case anybody argued for on the merits. It is left as a documented exception here rather than silently pretending the "exact table.column pairs a statement reads" rule has no counterexample — an installation whose grant is missing only this one column fails startup over a column this service does not, in fact, need.

With one exception this is a subset of the full read-only table list the read-only account is granted SELECT on (00-common-contract.md §8): cpe, product_class, product_class_group, view, isp, cpe_parameter_name, ug_fcc, ug_fcc_speed_result, ug_fcc_latency_result, fcc_properties, update_group, update_group_child, ug_task, ug_diagnostic, ug_get_param, ug_set_param_value, ug_reactivate, ug_cpe_completed. The columns checked at startup reflect what the service actually reads today — the four-table device-resolution read added in M3 (The ACS Schema This Service Reads), plus, since M4, every table the legacy import reads (Legacy FCC Import). ug_diagnostic, ug_get_param, ug_set_param_value, ug_cpe_completed, view, isp and cpe_parameter_name are in the grant list and are not read by anything in this service today — ug_set_param_value in particular is deliberately out of scope for the import, which is why an imported campaign carries no URLs and no ping host.

The exception is manufacturer, and it is the whole point of the warning below: it is required by the startup check and is not in the contract’s §8 grant list. An account provisioned strictly from that list fails the check and the service does not start. The grant list is the document that has to catch up — the read is real, deliberate and load-bearing ([m3-grant-change]) — so treat §8 as one item short rather than treating the check as over-strict.

An existing installation’s read-only grant set now needs SELECT on manufacturer, or the service refuses to boot.

manufacturer is a whole table that M3 added to the gate: the catalog and the device grid both resolve a manufacturer display name through it (LEFT JOIN manufacturer m ON m.id = pc.manuf_id), with product_class_group.manufacturer_name only as a fallback. An account provisioned for M1/M2 will not have it, because nothing read it before.

The failure is at startup, loud, and complete:

java.lang.IllegalStateException: ftacs schema check failed: missing columns
[manufacturer.id, manufacturer.name]. The read-only ftacs account must have
SELECT on these tables.

The runner cannot tell a missing grant from a missing table — both simply fail to appear in information_schema.columns / ALL_TAB_COLUMNS for that account — so the message names both possibilities. Check the grant before concluding the table is absent; manufacturer has been part of the ACS schema for as long as product_class has.

The three new columns (cpe.protocol_id, cpe.firmware, product_class_group.protocol_id) do not need a grant change, because a table-level SELECT already covers them — they are listed here so that an installation whose ACS predates those columns learns it at startup rather than mid-campaign, which is the whole purpose of this runner.

Upgrade order therefore matters: extend the grant, then deploy. Deploying first produces an instance that will not start, and the M3 additions are the first time this check has ever demanded a grant an earlier release did not.

The check is skipped entirely in the localdev profile group and can be disabled with FTACS_SCHEMA_CHECK_ENABLED=false — which is the right lever for an emergency rollout and the wrong one for a permanent installation, since it converts a startup failure into a broken query in the middle of a campaign.

This service requires ACS 6.4.12 or later, and the startup check is where an older ACS is discovered. Know this before installing, not from a failed startup.

FtacsSchemaCheckRunner requires bom and eom on ug_fcc_speed_result, and those two columns were introduced by ACS changelog 6.4.12.a. An installation whose ACS predates that changelog does not have them, the gate lists them as missing, and the service does not start:

java.lang.IllegalStateException: ftacs schema check failed: missing columns
[ug_fcc_speed_result.bom, ug_fcc_speed_result.eom]. The read-only ftacs account
must have SELECT on these tables.

It is a real requirement and not an over-strict gate. bom and eom are the beginning and end of measurement as the device reported them — the only genuine device-side timestamps an imported row has (bom and eom Are the Only Genuine Device Timestamps) — and they are carried into fcc_result_speed with their stored milliseconds. Without them an imported measurement has no device-side timing at all.

As with every other finding of this runner, the message cannot tell a missing column from a missing grant. Here it is almost always a missing column, because ug_fcc_speed_result is in the §8 grant list already: check the ACS version first.

ug_fcc_latency_result is deliberately not held to the same standard — it has no bom, no eom, no end_date and no type in any ACS version, so none of them is required of it. Requiring them symmetrically would fail every installation on earth.

The lever of last resort is the same one: FTACS_SCHEMA_CHECK_ENABLED=false starts the service against an older ACS, and the legacy import will then fail on the first batch of a speed pass instead of at startup. That is a worse failure, not a cheaper one.

The iotw Grant Check: The One That Warns Instead of Failing

IotwSchemaCheckRunner is the second ApplicationRunner over somebody else’s schema, and it is deliberately the opposite of the first in exactly one respect: it does not abort startup.

ftacs is where every campaign’s devices and results live, so a service that cannot read it can do nothing and ftacs Read-Only Schema Check refuses to come up. iotw is borrowed for one of four device source types (sourceType=Condition); an installation that never uses saved conditions is fully functional without the grant, and taking the service down over it would turn an unused feature into an outage.

So why check at all? Because the iotw datasource is lazy, and the alternative is silence: without this, the first person to learn the grant is missing is an operator half-way through a campaign wizard, from a 400. The warning is in the startup output an installer already reads, it names every missing table.column at once rather than one per failed save, and it tells apart "no grant" from "wrong schema" from "unreachable" — three different tickets.

WARN  c.f.fcc.startup.IotwSchemaCheckRunner : iotw schema check FAILED: missing columns
[iotw_view_column_condition.logic]. Campaign configurations that name a conditionId
(configs[].sourceType=Condition) will be refused with a 400; every other device source
type is unaffected. Grant this service SELECT on the two iotw tables, or set
ft.fcc.iotw.enabled=false to refuse the source type by configuration and silence this check.
Situation What the runner does

Grant complete

INFO iotw schema check passed vendor=MySQL tables=2.

A column or table missing

The warning above, with the full list. The service starts; sourceType=Condition is refused with a 400 that names the condition.

Schema unreachable (database down, wrong credentials, no route)

The same warning, carrying the connection failure’s message. This is the one place in the service that opens the lazy iotw pool at startup, so it is also where an unreachable portal database is noticed early instead of on the first campaign save.

ft.fcc.iotw.enabled=false

INFO iotw schema check skipped and the pool is not touched at all — the one thing a disabled feature must not do. The source type is refused by configuration, with a message that says so.

ft.fcc.iotw.schema-check.enabled=false

The runner is not registered. The feature still works: an installation that wants the pool left closed until the first condition read — or whose portal database starts after this service — turns the check off and keeps the source type.

The required set is the two tables of the grant and nothing more: iotw_view(id, name, domain_id, view_type) and iotw_view_column_condition(id, view_id, parent_id, column_key, type, string_value, date_value, logic). The same rule as for ftacs applies — only columns some statement actually names — for a different reason here: a grant narrowed to exactly what this service needs should pass, not warn.

ClickHouse Migration at Startup

ClickHouseMigrationRunner runs the Flyway migration in a @PostConstruct method, not as part of Boot’s own startup sequence, specifically so a failed migration sets FccDegradedState to UNAVAILABLE and lets the service start DEGRADED instead of refusing to start at all.

Three migrations ship today: V1 creates fcc_result_speed, V2create-fcc-result-latency-table.sql (M2) creates fcc_result_latency, and V3add-ping-actuals-to-fcc-result-latency.sql (contract 1.5) adds its two nullable ping-actual columns. All three are recorded in flyway_schema_history_fcc, never the Flyway default — see Flyway History Table.

The M4 baseline fix, and how to confirm an installation has it. ftacs_qoe_ui_data is shared with QoE and is never empty — diag_speed_test and diag_ip_ping are already there on a clean FCC installation. Before M4, Flyway met that populated schema with no history table of its own, baselined it at the default version 1, and then considered V1__create-fcc-result-speed-table.sql already applied — the table this service writes every speed measurement into was never created, and nothing said so; every insert failed from the first campaign onward. ClickHouseFlywayConfig now baselines at version 0, below every migration this service owns, so V1 onward all run on a clean install.

Confirm a given installation has the fix by reading the history table directly:

SELECT version, description FROM flyway_schema_history_fcc ORDER BY installed_rank;

A healthy, up-to-date node shows a baseline row at version 0, then 1, 2, 3 in order. A node still carrying the old defect shows a baseline at version 1 with no V1 row beneath it — fcc_result_speed does not exist, and every insert into it has been failing since the service first started.

Since M4 the migration is retried on a timer (ft.fcc.clickhouse.migration.retry-interval, default 60s) for as long as the status is not UP, in addition to the @PostConstruct attempt. Without it, an instance that started during a ClickHouse outage stayed schema-less for the life of the process — nothing else ever ran the migration, so the outage would outlive itself and only a manual restart would end it. The tick is a no-op the moment the status is UP, so it costs nothing outside an outage.

Two instances retrying at the same moment is safe and deliberately unguarded: the ClickHouse Flyway plugin cannot take the history-table lock the other databases use (ClickHouse has none), so the safety comes from the migrations themselves, all three of which are idempotent (CREATE TABLE IF NOT EXISTS, ADD COLUMN IF NOT EXISTS). The worst outcome of a race is a duplicated history row and a second, redundant run of statements that change nothing.

A failed attempt — startup or retry — reports UNAVAILABLE without inspecting the exception: driver wording is not a classification, and ClickHouseHealthIndicator refines it to SCHEMA_BEHIND on its own next probe if connectivity turns out to be fine and migrations are merely pending — see [clickhouse-three-states].

Because a migration failure degrades rather than kills startup, a node that missed V2 or V3 runs the campaigns those migrations enable perfectly well and only fails on the INSERT, which surfaces as fcc.clickhouse.result.failed and rows parked in the dead letter — see Troubleshooting. Since M4 that node also keeps retrying the migration on its own, so the failure is expected to self-heal once its cause (usually a missing grant) is fixed, without a restart.

The Legacy Import Runbook

What the legacy ETL is, and every rule about what it reconstructs, is Legacy FCC Import; its endpoints are The Legacy Import Endpoints; its properties are ft.fcc.legacy-import.* — M4. This section is the operator’s sequence.

Before the First Start

Check Why, and what goes wrong without it

ACS 6.4.12 or later

bom / eom on ug_fcc_speed_result arrived in changelog 6.4.12.a and the startup gate requires them — see [acs-minimum-version]. An older ACS means the service does not start at all, so this is a check made before installing, not before importing.

SELECT on the import’s ftacs tables

ug_fcc, ug_task, ug_fcc_speed_result, ug_fcc_latency_result, update_group, update_group_child, fcc_properties, ug_reactivate. All eight are in the contract’s §8 grant list, so an account provisioned from that list needs no change for M4 — unlike the M3 manufacturer addition ([m3-grant-change]). What the account must not have is any write grant: the import never writes to ftacs, and the pool is opened read-only in both the Hikari config and in code.

/actuator/health reports the ClickHouse indicator UP

/start runs through the same FccResultStorageGate as an activation and answers 503 / 9503 while the store is not UP — see What a 9503 Means.

ft.fcc.legacy-import.zone is set to the zone the ACS wrote its timestamps in

Unless this service runs in that zone already, in which case the empty default is correct. Getting it wrong displaces every imported measurement by a fixed offset, silently. Changing it afterwards does not move rows that are already imported, and a resumed pass will not even adopt the new value — so this is the one property to get right first.

Expect Roughly 25 Hours of Lag on the Newest Measurements

A pass does not read up to the present, and an operator planning a cut-over has to know by how much it falls short.

One pass covers the half-open start_date interval [watermark_started_before, cutoff), where

cutoff = the pass's started_at - ft.fcc.legacy-import.window-settle - 24h

so with the default window-settle of 1h a measurement taken now is first eligible for a pass started about 25 hours from now. Raising window-settle raises the lag by exactly that much.

The extra day is not padding. A day-window is at most 24 hours long, and the hour a window starts at differs from campaign to campaign — it comes from each group’s ug_reactivate cron. Subtracting a full day makes the cutoff correct whatever hour a campaign’s day begins at, which is what lets one pass-wide cutoff serve every campaign; a completed pass writes a single date as its watermark, so it has to have a single coherent covered interval.

The trade is a day of lag in exchange for a watermark that cannot skip a row, and on a store with no delete and no back-fill a skipped row is skipped permanently. See Only Closed Day-Windows Are Imported.

Two operational consequences follow:

  • "The import has completed but yesterday’s measurements are missing" is normal, not a fault. They become importable on a later pass; COMPLETED has never meant "caught up to now".

  • The final catch-up pass of a migration has to be run at least 25 hours after the legacy flow is switched off, or it will not cover the last day the legacy flow wrote.

Start

Measure first. dryRun claims nothing, writes nothing and is safe to call at any time, including while an import is running:

POST /FCC/admin/legacy-import/start
X-FT-Domain-Scope: ALL
{"kinds": ["SPEED"], "dryRun": true}

The answer is the ordinary status body, with totalRowEstimate filled in for the named kinds. Then start for real:

POST /FCC/admin/legacy-import/start
X-FT-Domain-Scope: ALL
{"kinds": ["SPEED", "LATENCY"]}

Omitting kinds means both. They are imported one after the other by a single worker and each carries its own cursor, so starting one alone is a legitimate way to retry a kind that failed without disturbing the one that is fine. batchSize overrides ft.fcc.legacy-import.batch-size for this run only; lower it on a slow ClickHouse.

The call returns as soon as the kinds are claimed — the import itself runs on the fcc-legacy-import worker thread and is watched through /status.

Watch Progress

POST /FCC/admin/legacy-import/status (scope ALL, like the other two) returns one entry per kind.

Field How to read it

state

RUNNING while a pass is in flight; STOPPING once a stop has been recorded and before the current batch ends; COMPLETED when a pass finished — which is not terminal, since the legacy flow keeps writing and the next start catches up from the watermark; IDLE after a clean interruption; FAILED after an exception.

rowsRead / totalRowEstimate

The progress fraction, and only that. The denominator was taken when the pass started and the source keeps growing, so it is stale by design; nothing decides that the import is finished from it.

heartbeatAt

Refreshed between batches, and over the slow work that is not a batch: the per-campaign MAX(id) prologue, the anchor resolve and the fleet top-up. A RUNNING row whose heartbeat has stopped moving is a dead run — see Restart After an Interruption.

cursorUgId, cursorSerial, cursorDirection, cursorStartedAt, cursorResultId, cursorIteration

One position, read together. cursorResultId on its own says nothing about progress: the traversal order is not id order. cursorUgId is the useful one for "which campaign is it on". cursorDirection is the raw legacy type as text, not a direction name.

watermarkStartedBefore

What is complete: every measurement whose start_date is below it has been imported. Moves only when a pass completes, and then to that pass’s cutoff. Null means no pass has ever completed.

watermarkResultId

The highest legacy id a completed pass has covered — a diagnostic high-water mark that bounds no read. It is half of the late-arrival predicate and nothing else.

pendingResultId, pendingFromId, pendingUgId

Non-null means a batch is with ClickHouse right now, or a previous run died inside one. Both are normal; the next start re-checks exactly that id range, for the campaign pendingUgId names.

rowsLateSkipped

Measurements that reached ftacs after their day-window had already been imported. They are deliberately not back-filled — their window is closed and already ranked, and inserting into it would duplicate iteration positions in a store nothing can be deleted from. A non-zero value is a known limitation made visible; re-running the import produces the same number. See Late Arrivals Are Counted.

rowsInvalid

Source rows that cannot be expressed as a cursor position — a null serial, start_date or speed type. One of them halts the pass on an unmoved cursor and an unadvanced watermark, and lastError names the campaign and the row id. This one does need action: fix the source row and start again. See A Null Cursor Component Halts the Pass.

bytesOverflowCount, tierParseFailed, statusUnknownCount, propertiesMissingCount, devicesUnresolved

Data-quality tallies. None of them is an error to act on during the run — every one of those rows was imported anyway — but a large devicesUnresolved predicts that model-filtered reports over this data will be thin, and a large tierParseFailed means many rows carry tierKbps = 0.

zoneId

The zone this run is using, which for a resumed pass may differ from the configured one. The response’s top-level configuredZoneId is what a new pass would use.

lastError

Why the run stopped, when it did not stop because it was asked to.

The counters describe the current or last run: a fresh pass zeroes them, a resumed one continues them.

Micrometer counters under /actuator/prometheus carry the same figures across restarts of the pass: fcc.legacy.import.rows.read, .rows.written, .rows.skipped, .batches, .anchors.created, .activations.created, .devices.snapshotted, .devices.unresolved, .passes.completed, .passes.failed.

Log lines worth grepping: fcc legacy import pass started / pass completed / pass halted / pass failed / pass evicted, fcc legacy import group done (one per campaign), fcc legacy anchor created, fcc legacy fleet snapshot, fcc legacy import late arrivals skipped (a WARN naming the campaign, one per campaign that has any), fcc legacy import row cannot be a cursor position (the ERROR that halts a pass), and fcc legacy import batch at DEBUG.

Note that rowsLateSkipped and rowsInvalid have no Micrometer counter; they are visible on /status and in the log, and nowhere else.

Stop

POST /FCC/admin/legacy-import/stop
X-FT-Domain-Scope: ALL
{"kinds": ["SPEED"]}

The stop is cooperative and returns immediately: the flag is recorded, the running import notices it between two batches, and the row ends at IDLE on a committed cursor. Expect state = STOPPING in between; that state exists so an operator can see the request was recorded rather than pressing stop again.

There is deliberately no forced variant. Being killed mid-batch is exactly what the pending id range exists to survive, not something to offer a button for. Stopping an import that is not running is not an error — the flag is simply cleared by the next start.

A JVM shutdown does the same thing by itself: @PreDestroy requests a stop on every claimed kind and waits up to 10 seconds for the worker to finish its batch.

Restart After an Interruption

Call /start again with the same kinds. That is the whole procedure; nothing has to be reset, cleaned or reconciled by hand.

What happens under it:

  1. The kind is resumed rather than begun afresh when its cursor is intact or a pending range is outstanding — the counters continue, started_at is kept (so the window cutoff is still the one that pass set out with), and the traversal picks up strictly after the committed position. A pending range counts on its own because a crash in the very first batch of a pass leaves no cursor at all, and treating that as a fresh start is how the record of which ids are in doubt gets erased.

  2. If pendingResultId is non-null, the run first re-reads exactly that id range from ClickHouse and skips whatever is already there, counting it as rowsSkipped. This read happens at most once per resumed run, and the ids it finds are skipped for the campaign pendingUgId names and for no other.

  3. A kind whose row says RUNNING but whose heartbeat is older than ft.fcc.legacy-import.stale-after is taken over — that is what a killed process leaves behind. One whose heartbeat is fresh is refused; see What a 409 Means.

  4. If the old worker turns out to be alive after all, the takeover is still safe: every mutating statement is fenced on run_id, so the evicted run’s next write matches no row and it aborts without touching the row again (fcc legacy import pass evicted in the log). It does not get marked FAILED — the row belongs to the new worker, and writing a failure on it would stop a healthy import.

Never edit fcc_legacy_import by hand to "unstick" an import. The cursor, the watermark and the pending range are the only record of what has already been written into a store from which nothing can be deleted; clearing watermark_started_before re-imports every measurement below it, and every duplicated measurement is permanent.

What a 409 Means

FT code 9409. The kind is already being imported by a run whose heartbeat is still fresh — possibly on another service instance, since the claim is a database statement and not a process-local lock.

It is a 409 and not a 423 because the request is perfectly valid and will succeed as soon as the other run ends. The message names the holding runId and its last heartbeat.

Two remedies, in order: stop the other run and retry, or wait for ft.fcc.legacy-import.stale-after to elapse without a heartbeat if the holder is genuinely dead.

A start that names two kinds and can claim only one answers 409 and releases the one it took — a refused start must not leave half an import running.

What a 9503 Means

The same FccResultStorageGate every activation and every report goes through: the ClickHouse result store is not UP. Read details.reason on /actuator/health to tell UNAVAILABLE from SCHEMA_BEHIND — see [clickhouse-three-states].

It appears in two places for this feature, and they are different events:

  • on /start, refusing to begin at all. Beginning would mean reading millions of rows, failing every insert, and leaving a cursor that has moved past rows nothing wrote;

  • mid-run, where a store that goes down ends the pass cleanly at IDLE on its last committed cursor, with lastError naming the gate’s reason. Nothing is lost and nothing is duplicated; start it again once the store recovers.

No restart of the service is needed in either case. dryRun is not gated and keeps working during an outage, because it writes nothing.

Deployment

The image, the Helm chart, and — the most important part of it — why liveness and readiness are two different probes hitting two different endpoints, are on their own page: Deployment. It also covers the two ACS-side Hazelcast startup failures this service’s own init container cannot fix, and the minimum ACS version ([acs-minimum-version]).

The container’s port 8080 must never be published outside the docker network. This service has no authentication of its own (see No Authentication by Design); its only access control is that it is unreachable except from inside the compose/Kubernetes network, where ui-backend is the sole caller. Publishing this port to the host or an ingress would expose a completely unauthenticated API that can create, read and activate FCC campaigns across every domain. See Deployment for the full reasoning.

Troubleshooting

Symptom Likely Cause / Action

Activation stuck in DISPATCHING

The outbox row was marked SENT but no ack has arrived yet, or arrived and was not correlated. Check fcc_acs_outbox for the row’s state, attempts and next_attempt_at; check whether the ACS-side fcc.command.consumer.enabled flag is true (see The Closed REJECTED Reason Vocabulary — with the consumer disabled, no ack of any kind arrives).

Outbox rows stuck in PENDING

Hazelcast is unreachable from this service, or offer() is timing out (queue at max-size). This is the "waiting for messaging" state the UI is expected to surface. Check the Hazelcast client’s connection state and HZ_MEMBERS.

Outbox rows reaching ABANDONED

ft.fcc.outbox-max-attempts (default 10) publish/republish attempts elapsed with no successful offer() or no ack. The same transaction fails the paired activation (FccStalledActivationService.failAbandoned: state = FAILED, a failure_reason naming the exhausted outbox row, the owning test to ERROR, plus a fresh STOP), and does nothing if the run has meanwhile left PLANNED/DISPATCHING. The independent, age-based check in the same service still applies and, with the shipped defaults, normally fires first: once window_start_at is older than ft.fcc.activation-max-delay (default 30m) the run is failed regardless of the outbox row’s state. An ABANDONED row is therefore a signal that the ACS side (or Hazelcast) has been unreachable for a long time — check ACS-side liveness and whether fccCommandAck is being consumed at all.

An activation is refused with 503 (FccDegradedException)

FccResultStorageGate.require(…​) saw FccDegradedState.clickhouseStatus() != UP for the whole POST /FCC/activate request — no test in the request was processed. Check /actuator/health’s `clickhouse indicator and its details.reason — it does not print the literal SCHEMA_BEHIND or UNAVAILABLE; a migration count ("N clickhouse migration(s) pending: …​") means SCHEMA_BEHIND, a raw driver error message means UNAVAILABLE — see [clickhouse-three-states]. Also check the FCC-specific metrics fcc.result.parked, fcc.result.lost and fcc.clickhouse.result.failed, since FccResultIngestService can set UNAVAILABLE itself on a batch-insert failure or an exhausted result buffer even while the health probe’s own SELECT 1 still succeeds intermittently — see Result Ingest.

A /FCC/reports/** request is refused with 503 / 9503

Same gate as activation, since M4 — all four reporting endpoints call FccResultStorageGate.require(…​) before doing anything else. Read details.reason on /actuator/health to pick the fix — it names what Flyway or the driver actually reported, not either fixed sentence: a migration count ("N clickhouse migration(s) pending: <versions>") means ClickHouse answers but flyway_schema_history_fcc is not current — grant CREATE/ALTER on ftacs_qoe_ui_data if that is missing, or check why ClickHouseMigrationRunner’s retry ([clickhouse-migration-retry]) keeps failing; a raw driver error message means ClickHouse itself is down — check connectivity. (The 503 body of the request itself does carry the fixed sentences `"the result store is unreachable" / "the result store schema is behind: pending ClickHouse migrations" — those come from FccResultStorageGate, not from /actuator/health.) Neither case needs a restart: activations and reports resume automatically once the underlying cause clears and the next probe or retry succeeds.

POST /FCC/reports/export produces a file that ends abruptly, with no error visible to the caller

Almost always spring.mvc.async.request-timeout (FCC_EXPORT_REQUEST_TIMEOUT, default 10m) being reached, or a client-side timeout shorter than the frontend brief’s minimum of five minutes. Once the status line and the first bytes are on the wire, a failure on this endpoint cannot become an HTTP error any more — see Endpoints Implemented in M4 — so a timeout, a dropped connection or a ClickHouse hiccup mid-stream all look identical to the caller: a file that stops. Check the service log for fcc export aborted (a caught IOException, usually a client that went away or a timeout) versus fcc export failed (an unexpected RuntimeException, worth investigating); both lines carry the row count already written, which tells you how far the export got. If large exports routinely truncate around the same size or duration, raise FCC_EXPORT_REQUEST_TIMEOUT and confirm the client (browser, ui-backend, or whatever issued the request) uses a timeout at least as large.

POST /FCC/reports/export produces an XLSX file some readers refuse to open

Confirm first that the download is actually complete — see the row above; a truncated ZIP is the most common cause of "will not open" and is a timeout, not a format defect. If the file is complete and still will not open, check the row count against POST /FCC/reports/results for the same filter (a shape mismatch would mean the export and the screen predicates have diverged, which the report read path documents they must not) and check the description values of the offending rows for control characters from the device — FccXlsxStreamWriter drops the XML 1.0-illegal code points but keeps everything else, including characters outside the basic multilingual plane, so a reader that mishandles those is a reader defect rather than a file defect.

Results are arriving but no rows show up in fcc_result_speed

Two distinct, non-error causes, both normal: (1) the row is genuinely still buffered — it flushes at ft.fcc.result.batch-size (100) rows or ft.fcc.result.flush-interval (5s), so a quiet stream can sit in memory for up to that interval; (2) the natural key (activationId, serial, direction, iteration) is still marked in the fccResultDedup Hazelcast map from an earlier delivery of the same result (TTL ft.fcc.dedup.ttl, default 1h) — check the fcc.result.dedup.skipped counter. If neither explains it, check fcc.result.parked (the row was moved to fcc_result_dead_letter — buffer exhausted or ClickHouse unreachable, DEGRADED), fcc.result.late (the run had been stopped too long ago and the row was excluded on purpose) and fcc.clickhouse.result.failed (insert rejected — see the batch error log line and ClickHouseHealthIndicator). For a LATENCY campaign, also confirm you are looking at fcc_result_latency rather than fcc_result_speed.

/actuator/health reports DEGRADED

ClickHouse (ftacs_qoe_ui_data) is unreachable, its migration failed at startup, or FccResultIngestService could not write a batch / exhausted its buffer — see Health. CRUD keeps working; activation requests are refused with 503 until availability is restored.

After upgrading, a TIME_BASED run reports a speed but no interval breakdown, or V4 is pending

The ClickHouse V4 migration adds the time-based result columns and is purely additive: three Nullable(Int64) scalars (period_of_full_loading_micros, test_bytes_under_full_loading, total_bytes_under_full_loading) and five parallel Array columns holding the TR-143 IncrementalResult rows (increment_index, increment_start, increment_end, increment_test_bytes, increment_total_bytes). Every statement is ADD COLUMN IF NOT EXISTS, metadata-only and idempotent, exactly like V3 — and outOfOrder(true) + validateOnMigrate(false) mean it applies cleanly to an installation still sitting at V2. Confirm it in flyway_schema_history_fcc, and if it is pending read details.reason on /actuator/health as [clickhouse-three-states] describes — a migration count there means the schema is behind rather than ClickHouse being down.

THE INTERVALS ARE ARRAYS IN THE SAME ROW AND NOT EXTRA ROWS, which is the one thing to know before querying them. The de-duplication key is (activationId, serial, direction, iteration) and the table’s ORDER BY is (fcc_test_id, serial, direction, created); a row per interval would have needed a fifth component in that key and would have changed the meaning of every report that counts rows per device. The five arrays are index-aligned — position n of each describes the same interval — so a breakdown is arrayJoin over them, and an empty array means the device returned no table, which is normal when timeBasedIntervalSec was left unset.

A time-based run with a speed but no breakdown is not a fault. The device reported PeriodOfFullLoading and the full-loading byte counters (so the rate is real) and no IncrementalResult table — either because the interval was never set, or because its model does not implement the table. A run with no full-loading block at all is the other case and reports OTHER with no speed by design: see Two Formulas, One Verdict.

After upgrading, a campaign’s settings or fleet are gone, or changelog_add_fcc_multi_config failed

The configurations migration moves every row that describes "what is measured" from the campaign down to the configuration — see fcc_test_config. It is the one changelog in this schema that is not purely additive, so it is worth knowing what it does before running it on a database you care about:

* fcc_test_setting_speed, fcc_test_setting_latency and fcc_test_device gain config_id, back-filled from fcc_test_config by test_id. The settings tables then DROP test_id; fcc_test_device keeps it. * fcc_test DROPS type, after legacy_result_kind has been back-filled from it. Nothing reads type afterwards, and there is no way back to it short of a restore. * ORPHANS ARE DELETED, not carried. A configuration whose campaign is gone, or a settings row or frozen device whose configuration is gone, describes nothing: no endpoint can reach it, FccTestDeleteHandler deletes all four tables together so it should not exist, and the NOT NULL constraints cannot be asserted while it does. Such rows are reachable rather than impossible, because this schema has no foreign keys (Schema-Wide Rules) — so if a campaign comes back from the upgrade without its settings, the rows were orphaned BEFORE it and the upgrade is what made that visible. * The three unique indexes it replaces — fcc_test_config(test_id), fcc_test_setting_*(test_id), fcc_test_device(test_id, cpe_id) — are dropped and re-created on the new keys. A database that somehow holds two configurations for one campaign at the old index therefore fails the (test_id, seq) create rather than silently keeping one.

A FRESH DATABASE NEEDS NOTHING AND IS NOT AFFECTED: it creates the old shape, migrates it in the same startup and ends in the new one, with every table empty. Every changeSet carries preConditions with MARK_RAN, so a re-run is a no-op.

A save or the legacy import fails with Column 'reactivation_repeat_every' cannot be null / ORA-01400

A development database that ran add-fcc-test-reactivation-rule before that changeSet was corrected. The column was created NOT NULL with a DEFAULT 1, and a DEFAULT does not apply to an explicit NULL — which is what Hibernate sends, because it puts every mapped column in the INSERT. Any creator of fcc_test that did not know about the column therefore failed outright, FccLegacyCampaignHandler.createAnchor included. The changeSet now creates it nullable and carries validCheckSum ANY, but its preConditions are columnExists + MARK_RAN, so a database that already has the column is left exactly as it was — deliberately, because a second changeSet dropping a constraint would carry the mistake into the schema history of every installation forever (fcc_test). Fix such a database by hand, once:

[source,sql] ---- — MySQL ALTER TABLE fcc_test MODIFY reactivation_repeat_every INT NULL; — Oracle ALTER TABLE fcc_test MODIFY (reactivation_repeat_every NUMBER NULL); ----

A FRESH DATABASE NEEDS NOTHING: it creates the column nullable from the start. Only the development environments that migrated between the two versions of the changeSet are affected, and no release ever carried the NOT NULL one.

The service will not start: ftacs schema check failed: missing columns [manufacturer.id, manufacturer.name]

M3 upgrade. The read-only ftacs account lacks SELECT on manufacturer, which M3 added to the startup gate — see [m3-grant-change]. Grant it and restart; do not reach for FTACS_SCHEMA_CHECK_ENABLED=false except as an emergency rollback, since that converts a startup failure into a broken query in the middle of a campaign. The same message naming cpe.protocol_id, cpe.firmware or product_class_group.protocol_id instead means the ACS schema itself predates those columns, which is a genuine version mismatch rather than a grant problem.

Every request returns 403

X-FT-Domain-Scope is missing or not one of ALL/LIST — almost always a ui-backend proxy misconfiguration rather than a client bug, since this service trusts the header unconditionally and never falls back to a default. Also check for scope LIST with an empty X-FT-Domain-Ids, which is refused the same way. Both answer FT code 9401, not 9403; a 403 carrying 9403 is an ordinary authorization refusal from a controller and means the headers were fine — see Two Different FT Codes Answer HTTP 403.

The wizard’s device count is smaller than the operator expects, with no error

Work through the three silent narrowings in order. (1) Domain scope: a scope-LIST caller who named condition.domainIds outside their visible set gets an empty result by design and never an unfiltered read (The Empty Intersection Selects Nothing). (2) protocol_id IS NULL: if the caller filtered by protocol without TR069, every legacy CWMP device — which carries a null in that column — is correctly excluded; with TR069 present the null arm is added and they are included (the protocol_id IS NULL arm). (3) A model-based condition and a device with no product class: cpe.product_class_id is nullable, every join is a LEFT one, and a comparison against NULL is never true — so a device with no product class is found by id or by serial but is excluded by modelIds, models, manufacturerIds, manufacturers, productClassGroupIds and firmwares alike (A Device With No Product Class). It shows up in countsByModel as the bucket with a null modelId, which is the quickest way to see how many there are. Note that unsupportedProtocolCount is not a narrowing — those devices are included and only flagged.

A campaign refuses to save with speed.downloadConnections=N exceeds M

Not a bug and not clampable: fcc_model_capability records a lower ceiling for at least one model of the fleet, and the message names those model ids. See The Capability Ceiling Is Refused, Not Clamped. The operator’s choices are to lower the value or to split the fleet; an administrator’s choice is to correct or remove the capability row through PUT / DELETE /FCC/admin/model-capability, which requires X-FT-Domain-Scope: ALL. Removing a row does not change any campaign that has already been saved.

fcc model capability upsert raced / fcc domain setting upsert raced in the log

Benign, and it means the retry did its job. Two writers saved the same model or the same domain at the same moment, both found no row, both inserted, and the unique index refused the loser. The admin service catches the violation and calls the handler a second time, where the row now exists and the UPDATE branch cannot violate anything — see Both Admin Upserts Retry Once. The caller saw a normal 200. The first attempt rolled back, so it left no fcc_audit row: one accepted PUT is still one audit entry, and an auditor will not see the collision here, only in this log line. What is not benign is a DataIntegrityViolationException reaching the 500 handler on these endpoints: that is a violation which survived an UPDATE, so it is a constraint nobody expected rather than contention, and it is deliberately not retried again.

fcc devices resolved no longer appears in the log at INFO

Deliberate, and do not raise it back. That line is emitted by FccDeviceResolveService.resolveFleet, which is shared by /FCC/devices/resolve, by /FCC/devices/count, by PUT /FCC/devices/import and by the save path — and the count is what the wizard re-issues while the operator is still typing. At INFO, one operator narrowing a condition writes a line per keystroke, each carrying a different device total, and the events that matter drown in them. It is therefore logged at DEBUG; raise the level for com.friendly.fcc.service.FccDeviceResolveService when you actually need it. What stayed at INFO is fcc fleet resolved for save, from resolveForSave — one line per campaign saved, carrying totalCount, unsupportedProtocolCount and the number of duplicated serials, which is the low-cardinality business event a compliance report describes and the only place the duplicate figure is recorded at all. Refusals are unaffected: every one is logged at WARN with its message and FT code by FccExceptionHandler, whichever endpoint raised it.

A per-domain default was changed but existing campaigns still use the old one

Working as designed. The settings chain is walked once, when a test is saved, never at activation — see It Runs Once, at Save Time. A campaign with reactivation_days = 7 keeps the values it froze for the rest of its reactivations. The fcc_audit row with entity_type = 'DOMAIN_SETTING' is the record of when the default changed; fcc_test_setting_speed / _latency hold what each campaign actually froze. To apply a new default, make a new campaign: export the old one (PUT /FCC/item/export), delete the element whose default you want to inherit from the file, and import it back (PUT /FCC/item/import) — the chain only fills fields that are absent, so a value left in the file wins over the new default exactly as it did when it was first saved. See The Settings Chain, and How to Opt Out of It.

A campaign is stuck in STOPPING

fcc_activation.in_flight_count has not reached 0 and ft.fcc.stop-grace (15m) has not elapsed since stopped_at. First: this is the normal state for up to that grace. The figure is recounted from fcc_device_state, never decremented (The STOPPING Drain), so check SELECT state, COUNT() FROM fcc_device_state WHERE activation_id = ? GROUP BY state — devices still SENT are the ones being waited for. If the count is not falling, the fccDeviceStateQueue consumer is the suspect: check the fcc-device-state-listener thread, and the counters fcc.device.state.applied, fcc.device.state.unroutable (the run is unknown to this service — the mirror of ACTIVATION_GONE) and fcc.device.state.invalid (a malformed message, an ACS bug). Past the grace the watchdog closes it as STOPPED regardless; if it has *not, the watchdog tick itself is failing — look for fcc watchdog tick failed and note that the message is throttled, so absence of repeats does not mean recovery. A run whose stopped_at is NULL is never claimed by that pass at all, which should not happen: SET_STOPPED_AT is applied on every path into STOPPING.

Measurements are landing in fcc_result_dead_letter

Read reason first, because the two are different incidents, and note that state follows from it: CLICKHOUSE_UNAVAILABLE is written PENDING, LATE straight as DISCARDED. CLICKHOUSE_UNAVAILABLE: the store was unreachable or the in-memory buffer overflowed — correlate with fcc.result.parked, fcc.clickhouse.result.failed and /actuator/health. Once ClickHouse is back, drain the backlog with POST /FCC/admin/result-dead-letter/resend; nothing drains it on a schedule. LATE: the measurement arrived more than ft.fcc.stop-grace after its run’s stopped_at and was deliberately excluded from the run (fcc.result.late). These are never resent, by design — see The After-Stop Window — and a steady stream of them means either a stop-grace too short for the fleet’s diagnostic duration, or an ACS that keeps measuring long after a STOP. fcc.result.lost above zero is the genuinely bad signal: the row could not even be parked and is gone. A non-zero fcc.result.resend.skipped.late means the table still holds LATE rows left PENDING by an earlier build, which the read now filters out; this is counted, not logged per row — only an aggregate log.info names the count once a resend call finishes.

A reactivation did not fire

Work from the campaign outward. (1) Does the campaign owe one — fcc_test.reactivation_days versus the seq of the last run? The campaign completes when its reactivation_days-th run finishes. (2) Is the campaign in SCHEDULED_FOR_REACTIVATION and is there a PLANNED row in fcc_activation? If the previous run never reached FINISHED, nothing planned one — check whether it is still RUNNING (the ACS never reported, and the window watchdog has not fired yet) or STOPPED/FAILED (which plan nothing at all, by design). (3) Is window_start_at in the future? Missed slots are skipped, not fired: after an outage the next window is the next real daily slot, not the one that was missed, so a campaign down for two days resumes tomorrow and simply loses those runs — look for fcc reactivation planned with the computed windowStartAt. (4) If a PLANNED row is due and still there, the scheduler tick is failing (fcc scheduler tick failed, throttled) or dispatch is skipping it — fcc dispatch skipped names the state machine’s reason, typically a campaign the operator stopped while the window waited, or missing settings rows.

A LATENCY campaign runs but fcc_result_latency is empty

Most likely the ClickHouse V2__create-fcc-result-latency-table.sql migration has not run on this node. ClickHouseMigrationRunner executes Flyway in a @PostConstruct and a failure there degrades rather than kills startup, so the service comes up DEGRADED and the failure is a startup log line, not a crash. Verify against the dedicated history table, never the Flyway default: SELECT version, success FROM flyway_schema_history_fcc ORDER BY installed_rank. A missing V2 row with V1 present means the migration never ran; a success = 0 row means it ran and failed. Note that the insert failure surfaces on the result path rather than at startup, as fcc.clickhouse.result.failed and rows parked as CLICKHOUSE_UNAVAILABLE — so check the dead letter too. Also confirm the campaign really is LATENCY and its results carry direction = PING: a null direction is routed to fcc_result_speed on purpose.

Latency results stop landing in ClickHouse while speed results keep landing, with ClickHouse errors naming No such column repetitions (or data_block_size) on insert into fcc_result_latency

V3__add-ping-actuals-to-fcc-result-latency.sql (contract 1.5, see What Contract 1.5 Changed) has not been applied on this node, while the service is nevertheless writing the 1.5 column list. Since the first half of M4 this state is self-healing and visible, and the paragraph that used to stand here described the behaviour before it: @PostConstruct is only the first attempt, ClickHouseMigrationRunner then retries on a timer until the schema is current, and the health probe asks flyway.info().pending() in addition to connectivity, so an instance sitting at V2 reports SCHEMA_BEHIND rather than flipping back to available the moment ClickHouse answers SELECT 1. What is left for the operator is therefore the cause, because a retry cannot grant itself a privilege or repair a failed history row. Plausible triggers: ClickHouse unreachable at startup, the connecting user lacking ALTER on the shared database, or Flyway having recorded V3 as success = 0 — ClickHouse has no DDL transactions, so a failure partway through the file leaves a failed history row that makes every later migrate() throw until a repair. + Why this matters more than a plain missing-migration symptom. V1 and V2 are CREATE TABLE, so their failure breaks both diagnostics visibly and nothing else silently compensates. V3 is the first migration that ALTER`s a table the service already writes to, so the failure is asymmetric: the speed insert succeeds while the latency insert throws. Until the first half of M4 both halves shared one `try in FccResultIngestService.flush(), so the whole batch went back and every retry re-wrote the same speed rows — roughly 48 duplicate copies of each per day at the 30-minute backoff ceiling, in a MergeTree that has no uniqueness to remove them with. The two halves now sit in separate try blocks (see Result Ingest): the speed half commits, only the latency half is returned, and the duplicates are gone. What remains is that both halves share one backoff ladder, so while fcc_result_latency is broken the healthy speed rows land on the ladder’s cadence rather than immediately — a delay of up to 30 minutes, not a loss. Watch fcc.clickhouse.latency.failed climbing while fcc.clickhouse.result.inserted still advances. + What to do. Confirm the applied version in the Flyway history table (flyway_schema_history_fcc in the shared ftacs_qoe_ui_data, never the default flyway_schema_history — see Flyway History Table): SELECT version, success FROM flyway_schema_history_fcc ORDER BY installed_rank. Check the startup log for clickhouse migration failed, the only signal the runner emits. Verify the connecting user’s ALTER grant on ftacs_qoe_ui_data. Run flyway repair if a failed V3 row is parked, then restart the instance so @PostConstruct runs the migration again: the two ALTER TABLE …​ ADD COLUMN IF NOT EXISTS statements are idempotent and metadata-only (see the migration note under fcc_result_latency), so re-running them is safe, and outOfOrder(true) plus validateOnMigrate(false) mean V3 applies cleanly to an installation still sitting at V2. + Known limitation. The durable fix is for the DEGRADED gate not to re-open on connectivity alone when the migration itself never succeeded — today `ClickHouseHealthIndicator’s probe and `ClickHouseMigrationRunner’s startup check write to the same flag with no memory of which one last failed. That is tracked separately, not something this service does today.

The legacy import says RUNNING but rowsRead and heartbeatAt have stopped moving

The worker is gone — the process was killed, or the pass threw somewhere the handler did not catch. Do not edit fcc_legacy_import. Wait for ft.fcc.legacy-import.stale-after (default 5m) to elapse without a heartbeat and call /start again with the same kinds; the claim’s staleness arm takes the row over and the run resumes from its committed cursor, re-checking pendingResultId’s range first. A `/start before that elapses is a 409 naming the stale runId. See Restart After an Interruption.

POST /FCC/admin/legacy-import/start answers 409 / 9409

The kind is held by a run whose heartbeat is still fresh — possibly on another service instance, since the claim is a conditional UPDATE and not a process-local lock. Stop that run, or wait out stale-after. See What a 409 Means.

POST /FCC/admin/legacy-import/* answers 403

Every one of the three requires X-FT-Domain-Scope: ALL, including /status, which is stricter than any other endpoint of this service. The import creates campaigns in whatever domains the legacy groups belonged to, its cursor knows nothing about domains, and its counters disclose every domain’s data volume — see Why All Three Require Scope ALL.

A report filtered by model or manufacturer returns nothing for an imported campaign

Two causes, in this order. (1) No fleet snapshot, or a thin one: that filter resolves only through fcc_test_device, so check devicesSnapshotted and devicesUnresolved on /status — a serial whose ftacs.cpe row no longer exists is not snapshotted, and its measurements are reachable only by an unfiltered report (A Serial With No Live cpe Row). (2) The model is the model at import time, not at measurement time, so a CPE re-provisioned since the campaign ran answers to its current model (model_id Is the Model at IMPORT Time). Neither is fixable by re-running the import.

An imported campaign is invisible to an operator, and visible with X-FT-Domain-Scope: ALL

Working as designed. Its anchor carries domain_id = 0, which is the super domain (CommonUtils.isSuperDomain: null, 0, -1) — and 0 is in nobody’s X-FT-Domain-Ids, so only X-FT-Domain-Scope: ALL reads it. See legacy-import.adoc#domain-zero. Changing the anchor’s domain is an operator decision about a campaign this service did not author; the import will not do it, and a re-run will not revisit it.

The legacy import reports a large tierParseFailed or bytesOverflowCount

Neither drops a measurement and neither needs action during the run. tierParseFailed means the contract tier could not be reconstructed from the legacy string and the row carries tierKbps = 0 (tier_kbps); bytesOverflowCount means the legacy total_bytes had wrapped and was imported as NULL rather than as a number four gigabytes wrong (total_bytes). Both are properties of the source data, so re-running the import produces the identical tallies. The lossless tier string is still in ftacs.ug_fcc_speed_result.tier, which the import never touches.

The legacy import stopped at IDLE and lastError names a campaign and a result id

A source row whose serial, start_date or speed type is null, counted as rowsInvalid. It cannot be made into a cursor position — in SQL every comparison against NULL is UNKNOWN, so such a cursor would match nothing and the rest of that campaign would be dropped in silence — so the pass halts there with its cursor unmoved and its watermark unadvanced. Nothing has been declared imported that was not. Fix the named row in ftacs (all three columns are NOT NULL in the ACS DDL, so it should not exist) and call /start again. See A Null Cursor Component Halts the Pass.

A pass completed but rowsLateSkipped is non-zero

Measurements reached ftacs after their day-window had already been imported and ranked. They are deliberately not back-filled, because inserting into a closed window would give two measurements the same iteration position in a store from which nothing can be deleted. The WARN line names the campaign. There is no remedy and no re-run that changes it — it is a known limitation made visible, and the surviving copy of the data is in ftacs, untouched. Running the import before the legacy flow has finished writing a day is what produces it; see Expect Roughly 25 Hours of Lag on the Newest Measurements. Late Arrivals Are Counted.

The legacy import completed, but the last day or two of measurements is not in a report

Expected. A pass reads only up to started_at - window-settle - 24h, so the newest roughly 25 hours are never in the pass that has just finished; they are picked up by a later one. COMPLETED does not mean "caught up to now" — see Expect Roughly 25 Hours of Lag on the Newest Measurements.

A Prometheus dashboard shows NaN on fcc_queue_* or fcc_outbox_*, or a series it expected is missing entirely

Read Where NaN Appears, and What It Means before assuming a bug: several of these are structural (the command-queue and command-ack ages are never observable from inside this service) rather than a scrape or a sampler problem. A genuinely missing series — as opposed to a present series reading NaN — usually means the sample tick has not run yet since startup, or fcc_hazelcast_connected is 0.

PUT /FCC/item/import is refused and the operator asks what was already created

No longer possible. PUT /FCC/item/import creates nothing since DEV-1381 #13 — it parses the document and answers the campaign FORM, which the operator then saves — so a failed import leaves nothing behind and nothing has to be found and deleted before retrying. A document carrying more than one <test> is refused outright, and every other refusal is a plain 400 at any point in the parse; see The Import Parses and Returns.

PUT /FCC/item/import answers 500 rather than a 400 about the document

A code defect to report, not a document to fix. IllegalStateException from FccConfigRecordFields — an unreadable accessor, an uncallable canonical constructor, or a record component of a type the format cannot carry — means a programming error and not a caller’s mistake, so it reaches the 500 handler rather than being dressed up as a refusal about the document. It means a request record (FccTestActivationRequest, FccSpeedSettingRequest or FccLatencySettingRequest) gained a field of a type FccConfigRecordFields does not know how to convert — most plausibly a BigDecimal. Everything about the document itself is a 400, at whatever point in the parse it is noticed: an unreadable XML, a second <test>, a settings block that does not match the type, a serial that resolves to no visible device, an unknown element. There is no per-test results[] envelope any more — DEV-1381 #13 made this endpoint answer one campaign form or one error, and nothing in between.