Architecture

The Governing Decision

The ACS and ft-fcc-service do not know each other. There is no SOAP client, no REST call, and no address of one service in the configuration of the other — in either direction. The only channel between them is a fixed set of five Hazelcast queues, described in Hazelcast Queues.

This is deliberate, not an oversight:

  • Independent deploy and rollback. Each side ships on its own schedule. A queue message is the only coupling, and the DTO contract rule "fields are only ever added" (DTO Contract Rules) means an old consumer can read a new producer’s message and vice versa.

  • One transport, one message contract. A SOAP or REST integration would mean two API surfaces to version together (request/response shapes plus a synchronous failure mode). A queue message has exactly one shape per queue, defined once in com.friendly.fcc.dto and copied byte-for-byte into both repositories.

  • The new FCC path never touches Update Group. fcc-core is a separate ACS module; the legacy fcc/** code and ftdm-update-group are not modified by this work at all.

This decision has a real cost: there is no synchronous accept/reject. An ACTIVATE command is offered to a queue and the caller has no idea, at that instant, whether the ACS will ever see it. This is why:

  • every command carries an explicit ack, correlated by idempotencyKey and commandSeq (fccCommandAck, FccAckResultEnum) — see Matching an Ack;

  • an ack that does not arrive within ft.fcc.command-ack-timeout (default 60s) triggers republication with backoff, not a silent retry;

  • Hazelcast becomes a hard dependency for both sides: if the cluster is unreachable, neither side can move a campaign forward, and this service alone cannot tell whether the ACS is down or the cluster is.

See Command and Event Semantics for the full timeout and backoff ladder.

No Authentication by Design

ft-fcc-service has no authentication of its own — no Spring Security, no tokens, no user store. Two things protect it instead:

  • The port is never published outside the docker network. Dockerfile only `EXPOSE`s 8080 for the compose network; the compose/Helm topology never maps it to the host or to an ingress.

  • Angular reaches it only through ui-backend, which authenticates the portal user, resolves their domain scope, and injects the identity headers this service trusts unconditionally — see Identity Contract. ui-backend proxies iot-webservice/iotw/FCC/ to /FCC/ one to one and never rewrites a body.

This is why the identity filter (FccIdentityFilter) treats every header as authoritative and refuses the request (HTTP 403) rather than falling back to a default when X-FT-Domain-Scope is missing: a filter that answered 200 on a missing scope would not be an authorization boundary at all, it would be decoration.

There is also no role model for FCC: access is simply "the FCC menu item is visible in the portal", and once inside, the only further narrowing is the domain scope.

This is exactly why fcc_audit exists. Every caller ui-backend lets through may create, activate, stop, pause and delete any campaign of a domain they can see, so there is no permission that would have prevented an action; the only question an auditor can still be answered is who took it and when. The four audit columns on every table (created, creator, updated, updator) do not cover that — they show the last writer of a row, and a deleted campaign has no row left at all — so the lifecycle actions are recorded as an append-only trail of their own. FccAuditService is @Transactional(propagation = MANDATORY), which is what forces the trail into the same transaction as the change it describes: a trail that could record a stop which was then rolled back, or miss one that committed, is evidence nobody can rely on. A null user_id means the engine did it, not "unknown user" — the scheduled-campaign poller and the reactivation path create runs nobody asked for at that moment, and an auditor reading a daily campaign has to be able to tell the operator’s first activation from the six the service performed by itself. See fcc_audit.

Schema Ownership

Four schemas, four different ownership rules:

Schema Access Rule

ftfcc

Read-write

Owned by this service. Migrated by Liquibase, and by nothing else — see ftfcc Schema.

ftacs

Read-only

Owned by the ACS. A dedicated database account holds SELECT grants on a fixed table list (Startup Behaviour); the Hikari pool is additionally marked read-only in code (FtacsJdbcConfig, DataSourcesConfig), and every ftacsTransactionManager transaction must declare readOnly = true. There is no JPA here on purpose: entities invite lazy writes and dirty checking on a schema this service does not own.

ClickHouse ftacs_qoe_ui_data

Shared, this service’s tables only

Shared with QoE monitoring, which already owns diag_speed_test and diag_ip_ping there. This service owns fcc_result_speed / fcc_result_latency and migrates them with Flyway under a dedicated history table flyway_schema_history_fcc — see ClickHouse Schema.

iotw

Read-only, two tables, optional

Owned by ui-backend. Borrowed for one thing: the saved conditions a campaign configuration can name (iotw_view, iotw_view_column_condition — the grant). Read-only in the grant and in code, no JPA, no transaction manager and a lazy pool, so an installation that does not grant it starts normally and refuses only sourceType=Condition. The narrowest dependency in this service, and the only optional one.

Liquibase cannot be pointed at ftacs: it needs CREATE TABLE DATABASECHANGELOG/DATABASECHANGELOGLOCK and INSERT/UPDATE/DELETE on the lock table before it runs anything, and the read-only account has none of that. Instead, FtacsSchemaCheckRunner verifies at startup, with a single catalog query per vendor, that every table and column this service reads still exists and is visible to the account — see Startup Behaviour. IotwSchemaCheckRunner does the same for the two iotw tables and warns instead of failing, because a missing grant there costs one device source type and not the service — The iotw Grant Check.

The State Machine as the Single Decision Point

Every change of fcc_test.state and fcc_activation.state goes through one pure function. com.friendly.fcc.statemachine holds two of them — FccTestStateMachine.evaluate(from, event, context) and FccActivationStateMachine.evaluate(from, event, context) — and nothing in that package is a Spring bean, holds state, performs I/O or logs. The only dependency is com.friendly.fcc.domain.enums.

The transitions used to be inline if`s and `EnumSet guards spread over FccAckService, FccStalledActivationService, FccActivationHandler and FccTestValidator, where the full rule set could only be reconstructed by reading four classes and no two of them agreed on what a late event should do. The duplicated STARTABLE_STATES / OPEN_STATES / ACTIVATABLE_STATES sets are gone.

Each evaluate returns a sealed decision, which is what makes a caller’s switch exhaustive without a default branch — a third outcome added later becomes a compile error at every call site instead of falling silently into somebody’s else:

Outcome Meaning

Allowed(to, effects) — on the activation side Allowed(to, stopReason, effects)

The caller writes to into the state column, applies stopReason when it is non-null, and carries out every effect, in the same transaction. to may equal the old state: RUNNING + ACTIVATION_STARTED is an idempotent confirmation, not a no-op the caller may skip.

Denied(reason)

The event does not apply and nothing is written. The reason is human-readable ("activation state FINISHED does not accept EVENT_STARTED") and goes verbatim into the log line or the 409 message.

A Denied is a normal outcome, not an error. Acks and activation events reach this service over Hazelcast queues, which deliver at least once and do not order across producers. A republished command produces a second ack; an ACS that failed over answers about a run this service closed minutes ago; a late ACTIVATION_FINISHED arrives for a campaign the operator has already stopped. Applying one would resurrect a campaign nobody expects to be alive. A caller driven by a queue therefore logs the denial at DEBUG/INFO and moves on; only a caller driven by a user request turns it into an FccConflictException (HTTP 409), because there a denial is an answer the operator asked for.

An Effect Is a Declaration, Not an Action

The state machine performs no I/O, so it names the side effect and the caller — which owns the transaction, the repositories and the outbox — carries it out. That is what makes the rule set reviewable as a table.

Machine Effect What the caller must do

fcc_test

CLEAR_SCHEDULED_AT

Null fcc_test.scheduled_at. The campaign is running now, so a leftover instant would make the SCHEDULED poller claim it a second time when that instant arrives.

fcc_test

SET_SCHEDULED_AT

Write the requested instant. The instant is the caller’s — it comes from the request, not from the state machine.

fcc_test

PLAN_REACTIVATION

Create the next run as a PLANNED fcc_activation with no command attached (FccReactivationService → FccActivationHandler.planReactivation). Emitted only when the campaign still owes a run.

fcc_test

STOP_ACTIVE_ACTIVATION

Stop whatever run the campaign currently owns. Deliberately not applied by FccActivationTransitionService: the effect means "stop whatever run the campaign owns", which is a query against fcc_activation and may name a different row than the one whose event is being applied. Two callers carry it out, and both delegate to FccStopService so that the PLANNED-versus-live split has one implementation — see [superseded-run].

fcc_activation

SET_STARTED_AT

Write started_at once. The ack and ACTIVATION_STARTED travel on different queues and either may arrive first; the measurement window is anchored on this column.

fcc_activation

RECOMPUTE_WINDOW_END

Re-anchor window_end_at on the real started_at plus total_duration_h, so a command that waited in the outbox still gets its full window — see ACTIVATE for why this deliberately diverges from the window the ACS holds.

fcc_activation

SET_STOPPED_AT

Write stopped_at once. ft.fcc.stop-grace is measured from it, so a repeated stop report would keep pushing the deadline and a run that never drains would never be closed.

fcc_activation

SET_FINISHED_AT

Write finished_at, unconditionally — every transition carrying it lands in a terminal state, so it is written exactly once anyway.

fcc_activation

PUBLISH_STOP_COMMAND

Write a STOP to fcc_acs_outbox with a fresh idempotency key and a fresh command_seq — a new command, never a republish. One implementation only: FccStopCommandPublisher, shared by the stop/pause/delete path and by FccStalledActivationService. Two of them would be two chances to get the sequence wrong, which is the failure that makes the ACS discard the command as STALE_SEQ.

FccActivationTransitionService is the single bean that applies both machines' decisions, and it is @Transactional(propagation = MANDATORY): the watchdog’s callers claim their rows with FOR UPDATE SKIP LOCKED, whose locks live until the claiming transaction commits, so a nested or new transaction here would apply the transition outside the scope that holds the claim. MANDATORY turns "somebody forgot the transaction" into a call-site failure instead of a race nobody reproduces.

fcc_test Transition Table

This is the specification, reproduced from FccTestStateMachine’s class Javadoc. Anything not listed is `Denied.

From Event To Effects

NOT_ACTIVE, STOPPED, COMPLETED, ERROR

ACTIVATE_IMMEDIATELY

RUNNING

CLEAR_SCHEDULED_AT

ACTIVATE_SCHEDULED

SCHEDULED

SET_SCHEDULED_AT

PAUSED

ACTIVATE_IMMEDIATELY

RUNNING

CLEAR_SCHEDULED_AT

ACTIVATE_SCHEDULED

SCHEDULED

SET_SCHEDULED_AT

USER_STOP

NOT_ACTIVE

CLEAR_SCHEDULED_AT

SCHEDULED

ACTIVATE_IMMEDIATELY

RUNNING

CLEAR_SCHEDULED_AT

ACTIVATE_SCHEDULED

SCHEDULED

SET_SCHEDULED_AT

SCHEDULE_DUE

RUNNING

CLEAR_SCHEDULED_AT

USER_STOP

NOT_ACTIVE

CLEAR_SCHEDULED_AT

SCHEDULED_FOR_REACTIVATION

REACTIVATION_DUE

RUNNING

 — 

ACTIVATE_IMMEDIATELY

RUNNING

CLEAR_SCHEDULED_AT, STOP_ACTIVE_ACTIVATION

USER_STOP

NOT_ACTIVE

STOP_ACTIVE_ACTIVATION

USER_PAUSE

PAUSED

STOP_ACTIVE_ACTIVATION

RUNNING

ACTIVATION_STARTED

RUNNING

 — 

USER_PAUSE

PAUSED

STOP_ACTIVE_ACTIVATION

USER_STOP

NOT_ACTIVE

STOP_ACTIVE_ACTIVATION

ACTIVATION_FAILED

ERROR

 — 

ACTIVATION_STOPPED

STOPPED

 — 

ACTIVATION_FINISHED (more reactivations)

SCHEDULED_FOR_REACTIVATION

PLAN_REACTIVATION

ACTIVATION_FINISHED (none left)

COMPLETED

 — 

ARCHIVED

anything

denied

(terminal and unreachable)

Four rows are load-bearing and easy to "fix" wrongly:

  • RUNNING + ACTIVATION_STARTED → RUNNING is idempotent on purpose. The ACS confirms a run this service already believes is running — because it applied the ack first — and the confirmation may arrive twice. Answering Allowed keeps it out of the denial logs, where it would be noise nobody can act on.

  • USER_STOP → NOT_ACTIVE, and STOPPED is the ACS’s word rather than the operator’s. This follows Group Update, which has no "stopped" status at all: its stop returns the group to Created — the UI’s "Not active" — because that is what the group is afterwards. The same was always true here and only the label said otherwise. FccStopService.stop runs the identical code for a stop and a pause: both clear scheduled_at, stop every activation of the live run with a real STOP command, and close every PLANNED reactivation. There is no prev_state, no resume-from-position and no scheduled job to un-pause, so a later ACTIVATE_* starts a brand-new run from either state — a stopped campaign sits in exactly the condition one that never ran sits in. PAUSED survives because it still says the one thing a stop does not: the operator means to come back to it.

  • RUNNING + ACTIVATION_STOPPED → STOPPED is now what STOPPED means, and only that: the run was cut short and nobody asked for it. All three callers that could feed this event — the activation-event consumer, the device-state drain and the stop-grace watchdog — skip it when FccActivationTransitionService.isUserInitiatedStop holds, so an operator’s stop or pause never reaches it (see Activation Event Semantics). That guard was already load-bearing for the pause, because feeding the event on top of one is the single way a pause silently degrades into a stop; it is now also what keeps the two kinds of closed run apart in the campaign list. NOT_ACTIVE is "the operator stopped it", STOPPED is "the ACS or an outage stopped it" — before this change both read STOPPED and nothing distinguished them.

  • PAUSED + USER_STOP → NOT_ACTIVE, and PAUSED is the only closed state that accepts a stop. A paused campaign was stoppable nowhere before this row existed: PAUSED was evaluated by the branch that serves NOT_ACTIVE, STOPPED, COMPLETED and ERROR, which allows only the ACTIVATE_* events, so POST /FCC/stop answered 409 test state PAUSED does not accept USER_STOP and the only way to close the campaign was to activate it again — a whole new run on the fleet — and stop that. Two places told the operator to do the impossible: the stop endpoint, and FccTestValidator.validateTiming, which refuses a new booking with "test is PAUSED and its schedule cannot be changed by a save; stop it first, then book it again". It is routed separately rather than added to the four states above, deliberately: "stop something that is already closed" has to stay the 409 that names the state, because that is how an operator stopping forty campaigns learns that somebody else closed one of them a minute earlier. PAUSED is the one closed state where the answer is not "nothing would change" — it is the state an operator is meant to leave, and NOT_ACTIVE is a different fact about the campaign: nobody is coming back to it. The effect is CLEAR_SCHEDULED_AT and not STOP_ACTIVE_ACTIVATION, because the pause has already done the stopping — see [nothing-left-to-stop-after-a-pause]. A second USER_PAUSE stays denied: it would change nothing, and the 409 names the state the operator is looking at.

  • NOT_ACTIVE, PAUSED, STOPPED, ERROR and COMPLETED deny ACTIVATION_FINISHED / STOPPED / _FAILED outright, but still accept ACTIVATE*. Re-running a finished, stopped or paused campaign is a normal product operation — every run is its own fcc_activation row with its own settings snapshot. PAUSED’s denial of the three is untouched by the row above and has to stay: an `ACTIVATION_STOPPED applied on top of a pause is the single way a pause silently degrades into a stop. The new row gives PAUSED USER_STOP and nothing else.

Stopping a paused campaign carries CLEAR_SCHEDULED_AT only, because a pause leaves nothing on the ACS to stop. FccStopService.stop runs the identical code for a stop and a pause, so by the time a campaign is PAUSED every activation of its live run has either been closed locally (PLANNED — nothing was ever sent for it) or been sent a real STOP and moved to STOPPING, and every PLANNED reactivation has been closed with it. STOPPING is deliberately not in FccStopService.LIVE_STATES: that run is draining under ft.fcc.stop-grace with its STOP already on the ACS, and a second command would burn a command sequence to repeat an instruction that is being carried out. Nothing can re-open the gap either — SCHEDULE_DUE and REACTIVATION_DUE are denied from PAUSED, and an ACTIVATE_* that would create a new run leaves PAUSED in the same transaction — so the only thing this stop can still have to do is clear a pending instant. Declaring the other effect would be the table claiming that a paused campaign may own something live on the ACS when it may not.

Neither effect is what performs the run half, which is why this is a question about the specification rather than about behaviour: FccStopService queries fcc_activation unconditionally instead of gating on STOP_ACTIVE_ACTIVATION, so an exotic live row would be closed either way.

One consequence is load-bearing: the draining run keeps stop_reason = USER_PAUSE, because the stop finds nothing to re-stamp. That is what isUserInitiatedStop reads, so when the run finally drains, the ACTIVATION_STOPPED the ACS reports is still not fed to the campaign and a stopped-after-pause campaign cannot slide from NOT_ACTIVE into STOPPED — this product’s word for "the ACS stopped it and nobody asked".

SCHEDULED_FOR_REACTIVATION + ACTIVATE_IMMEDIATELY carries STOP_ACTIVE_ACTIVATION as well as CLEAR_SCHEDULED_AT, and that second effect is not tidiness. A campaign between daily runs already owns its next one as a PLANNED row, and the operator is asking for a different run right now. Leaving the old row behind orphans it permanently: the campaign moves to RUNNING, so FccActivationHandler.dispatch then evaluates RUNNING + REACTIVATION_DUE, is denied, and leaves the row PLANNED — which the reactivation poller re-claims every tick for the rest of the installation’s life, gaining one more orphan on every later PLAN_REACTIVATION. A row that had already reached DISPATCHING or RUNNING is worse: it stays live on the ACS and keeps measuring under a campaign that has moved to a different run.

FccActivationHandler.stopSupersededRun performs it, and delegates to FccStopService.stopLiveRun rather than reimplementing the stop, because the two cases a superseded run splits into are exactly the two that service already distinguishes: a PLANNED row is closed locally with no command — nothing was ever sent for it — while a DISPATCHING/RUNNING one takes the normal STOP_REQUESTED path with a STOP for the ACS. A second implementation of that split is a second chance to send the ACS a STOP for a run it has never heard of.

Two details of it are load-bearing:

  • It runs before fcc_test.command_seq is bumped for the new run. The superseded run’s STOP has to take the lower sequence and the new ACTIVATE the higher one, or the ACS discards the ACTIVATE as stale against a STOP that was meant to precede it.

  • The reason is USER_STOP, not ERROR. An operator pressing activate is what ended the old run; nothing failed. It is also what makes FccActivationTransitionService.isUserInitiatedStop true, so the ACTIVATION_STOPPED the ACS reports for the old run later does not drag the campaign back out of the RUNNING state this activation has just put it in.

This edge closes a bean cycle — FccActivationHandler → FccStopService → FccActivationTransitionService → FccReactivationService → FccActivationHandler — so FccStopService is injected into the handler as an ObjectProvider and resolved per call rather than at construction. The last edge is the one that cannot move: planning the next daily run is the handler’s job. FccResultDeadLetterService breaks its own cycle with the ingest service the same way, for the same reason.

ARCHIVED is unreachable in both directions. It is written only by the M4 legacy ETL, for campaigns that finished years ago in the legacy product and must never be re-run, re-scheduled or reported on as live. Nothing leads to it and nothing leads out of it; an event naming it is a bug in the caller and is denied like any other.

fcc_activation Transition Table

Reproduced from FccActivationStateMachine’s class Javadoc. A dash in the Stop reason column means `null, which means leave stop_reason as it is — not "clear it".

From Event To Stop reason Effects

PLANNED

DISPATCH

DISPATCHING

 — 

 — 

PLANNED, DISPATCHING

ACK_ACCEPTED

RUNNING

 — 

SET_STARTED_AT, RECOMPUTE_WINDOW_END

ACK_REJECTED

FAILED

ERROR

SET_FINISHED_AT

GIVE_UP

FAILED

ACS_UNAVAILABLE

SET_FINISHED_AT, PUBLISH_STOP_COMMAND

EVENT_STARTED

RUNNING

 — 

SET_STARTED_AT, RECOMPUTE_WINDOW_END

RUNNING

EVENT_STARTED

RUNNING

 — 

 — 

EVENT_FINISHED

FINISHED

 — 

SET_FINISHED_AT

EVENT_FAILED

FAILED

ERROR

SET_FINISHED_AT

WINDOW_EXPIRED

FINISHED

WATCHDOG

SET_FINISHED_AT

PLANNED, DISPATCHING, RUNNING

STOP_REQUESTED

STOPPING

the context’s

SET_STOPPED_AT, PUBLISH_STOP_COMMAND

RUNNING, STOPPING

EVENT_STOPPED, in flight null or > 0

STOPPING

 — 

SET_STOPPED_AT

EVENT_STOPPED, in flight 0

STOPPED

 — 

SET_STOPPED_AT, SET_FINISHED_AT

STOPPING

IN_FLIGHT_DRAINED, STOP_GRACE_EXPIRED, EVENT_FINISHED, WINDOW_EXPIRED

STOPPED

 — 

SET_FINISHED_AT

STOPPED, FINISHED, FAILED

anything

denied

(terminal)

Four rows carry reasoning that is not obvious from the table:

  • PLANNED/DISPATCHING + EVENT_STARTED → RUNNING exists because the ack and the activation event travel on different Hazelcast queues, which are not ordered against each other. The ACS can have started measuring and reported it before this service applied the ack that authorised it. Denying the report would strand the run in DISPATCHING until the watchdog killed a campaign that is live and streaming results, so the report is accepted and does exactly what the ack would have done, including re-anchoring the window.

  • EVENT_STOPPED splits on the in-flight count the ACS reports. A positive — or absent — count leaves the run in STOPPING, because those devices are still producing measurements that belong to this run; only a count of zero closes it immediately. An absent count is treated as positive on purpose: waiting out ft.fcc.stop-grace costs a delay, closing early costs measurements. Both branches stay Allowed rather than one being a no-op, so stopped_at is written even when the stop ack arrives before the stop was recorded locally.

  • STOPPING + EVENT_FINISHED → STOPPED, not FINISHED. A run that was stopped is reported as STOPPED even if the ACS managed to complete it afterwards: the operator’s decision is what the run history has to show, and a race between the stop and the last iteration must not be able to relabel it as a clean completion.

  • STOPPED, FINISHED and FAILED are terminal. Nothing leaves them — not a late event, not a second watchdog tick, not an ACS that comes back and re-reports. The run is over and its measurements have been counted; re-opening it would make a regulatory report disagree with itself.

The one place a caller deliberately diverges from this table is a PLANNED run being stopped. PLANNED means no command was ever sent — FccActivationHandler.dispatch writes the ACTIVATE and moves the row to DISPATCHING in the same transaction, which is also why a planned run’s command_seq is still the unassigned placeholder 0. Following PLANNED + STOP_REQUESTED → STOPPING would send the ACS a STOP for a run it has never heard of: it would consume a command sequence, it would be answered as an unknown activation, and the run would sit in STOPPING until the stop grace expired. FccStopService.closePlannedRun therefore closes it locally as STOPPED with no command at all, and the endpoint answers commandPublished: false — see POST /FCC/stop. The code marks this as an acknowledged follow-up: a dedicated PLANNED + CANCEL → STOPPED event carrying SET_STOPPED_AT, SET_FINISHED_AT and no PUBLISH_STOP_COMMAND would say the same thing in the one place that is meant to hold transition rules.

State Diagrams

fcc-state-machine-test
fcc-state-machine-activation

Why a Poller, Not Quartz

The scheduled work of this service — "activate a test whose scheduled_at has come due", "dispatch an outbox row", "check for an ack that never arrived", "close a run whose window has passed" — is expressed as @Scheduled pollers reading WHERE state = …​ AND …​ ⇐ now() …​ FOR UPDATE SKIP LOCKED, not as a Quartz job store.

Quartz would mean an additional dedicated database (see the ACS’s own Quartz Database in its architecture docs) and a cluster configuration, all to express what is, in substance, "periodically look at what has come due". A SKIP LOCKED poller needs neither:

  • poller state survives a restart because it lives directly in the business row (fcc_test.state / scheduled_at, fcc_activation.state / window_end_at, fcc_acs_outbox.state / next_attempt_at) — there is no separate job store to reconcile against the data after a crash;

  • FOR UPDATE SKIP LOCKED gives at-most-once dispatch across however many instances of this service are running, without a distributed lock.

The cost is that the claim is two statements — an ordered, capped, read-only select of candidate ids, then a FOR UPDATE SKIP LOCKED lock of those ids under the same predicate — and that only the first of the two needs a vendor-specific row cap (LIMIT ? on MySQL, ROWNUM outside an ordered inline view on Oracle). Both vendors accept FOR UPDATE SKIP LOCKED itself. See Vendor-Specific Claim SQL.

All five pollers belong to com.friendly.fcc.orchestration and are implemented and code-read as of this pass: the outbox dispatcher (FccOutboxDispatcher / FccOutboxPublisher), the ack-timeout poller (FccAckTimeoutPoller / FccStalledActivationService), the two due-work passes of FccActivationScheduler / FccActivationSchedulingService, and the two closing passes of FccActivationWatchdog / FccActivationClosingService. The indexes that back every one of their queries exist in the schema — fcc_activation(state, window_start_at) and fcc_activation(state, stopped_at) were added in M2, in their own changelog so the table changelog keeps its checksums. See ftfcc Schema for the exact index-to-query mapping and The Five Pollers for their cadences.

The Scheduler / Transaction Split

Every poller in this service is two beans, and the split is the same in all five. A @Component carrying the @Scheduled method holds no transaction and no business logic; a second @Component carries the @Transactional method that claims and applies. FccActivationScheduler / FccActivationSchedulingService and FccActivationWatchdog / FccActivationClosingService follow the shape FccOutboxDispatcher / FccOutboxPublisher established in M1.

Two independent reasons, and either one alone would be enough:

Reason What it prevents

A tick failure stays out of Spring’s scheduler log

The @Scheduled method catches, counts and throttles its own failures (FccFailureThrottle), logging the first occurrence of a signature in full and the repetitions at DEBUG. Left uncaught, Spring’s scheduler prints a full stack trace every tick — every 30 seconds for the scheduler, every 60 for the watchdog — for the entire duration of an outage that may last hours.

The Spring proxy stays in play

A @Transactional method invoked from another method of the same bean never reaches the proxy and therefore silently runs without a transaction. Here that is not a performance detail: the claims are FOR UPDATE SKIP LOCKED reads whose row locks live until the claiming transaction commits, so a self-invoked claim would release every lock the moment the statement returned, and a second service instance would immediately claim the same rows. Starting a campaign is not idempotent — it writes an ACTIVATE command and burns a command_seq.

The transactional bean therefore claims and applies in one scope, and each pass is its own transaction: a backlog of due campaigns and a backlog of due reactivations are independent, and one failing has no reason to discard the other’s work.

Inside a pass, a failure re-throws rather than being caught per row — FccActivationSchedulingService.startDueScheduledTests says so explicitly. FccActivationHandler.activate is @Transactional on another bean and therefore joins this transaction, and Spring marks a participating transaction rollback-only the moment the inner proxy sees an exception. Catching and continuing would start further campaigns, log them as started, and then have the commit throw UnexpectedRollbackException and discard all of them — a log that says a campaign started and a database that says it never did. Re-throwing keeps the tick honest: nothing was applied, nothing is claimed any more, and the rows are simply claimed again next tick.

Component View

fcc-architecture-components

Every component in the diagram is implemented and code-read for this pass. FccQueueListenerRunner starts a thread per registered FccQueueConsumer bean and all four inbound queues now have one: fccCommandAck, fccResultQueue, fccDeviceStateQueue and fccActivationEventQueue — see Messaging for the full specification.