Architecture
The Governing Decision
The ACS and ft-fcc-service do not know each other. There is no SOAP client, no REST call, and no address of one service in the configuration of the other — in either direction. The only channel between them is a fixed set of five Hazelcast queues, described in Hazelcast Queues.
This is deliberate, not an oversight:
-
Independent deploy and rollback. Each side ships on its own schedule. A queue message is the only coupling, and the DTO contract rule "fields are only ever added" (DTO Contract Rules) means an old consumer can read a new producer’s message and vice versa.
-
One transport, one message contract. A SOAP or REST integration would mean two API surfaces to version together (request/response shapes plus a synchronous failure mode). A queue message has exactly one shape per queue, defined once in
com.friendly.fcc.dtoand copied byte-for-byte into both repositories. -
The new FCC path never touches Update Group.
fcc-coreis a separate ACS module; the legacyfcc/**code andftdm-update-groupare not modified by this work at all.
|
This decision has a real cost: there is no synchronous accept/reject. An
See Command and Event Semantics for the full timeout and backoff ladder. |
No Authentication by Design
ft-fcc-service has no authentication of its own — no Spring Security, no tokens, no user store. Two things protect it instead:
-
The port is never published outside the docker network.
Dockerfileonly `EXPOSE`s 8080 for the compose network; the compose/Helm topology never maps it to the host or to an ingress. -
Angular reaches it only through
ui-backend, which authenticates the portal user, resolves their domain scope, and injects the identity headers this service trusts unconditionally — see Identity Contract.ui-backendproxiesiot-webservice/iotw/FCC/to/FCC/one to one and never rewrites a body.
This is why the identity filter (FccIdentityFilter) treats every header as authoritative and refuses the request (HTTP 403) rather than falling back to a default when X-FT-Domain-Scope is missing: a filter that answered 200 on a missing scope would not be an authorization boundary at all, it would be decoration.
There is also no role model for FCC: access is simply "the FCC menu item is visible in the portal", and once inside, the only further narrowing is the domain scope.
This is exactly why fcc_audit exists.
Every caller ui-backend lets through may create, activate, stop, pause and delete any campaign of a domain they can see, so there is no permission that would have prevented an action; the only question an auditor can still be answered is who took it and when.
The four audit columns on every table (created, creator, updated, updator) do not cover that — they show the last writer of a row, and a deleted campaign has no row left at all — so the lifecycle actions are recorded as an append-only trail of their own.
FccAuditService is @Transactional(propagation = MANDATORY), which is what forces the trail into the same transaction as the change it describes: a trail that could record a stop which was then rolled back, or miss one that committed, is evidence nobody can rely on.
A null user_id means the engine did it, not "unknown user" — the scheduled-campaign poller and the reactivation path create runs nobody asked for at that moment, and an auditor reading a daily campaign has to be able to tell the operator’s first activation from the six the service performed by itself.
See fcc_audit.
Schema Ownership
Four schemas, four different ownership rules:
| Schema | Access | Rule |
|---|---|---|
|
Read-write |
Owned by this service. Migrated by Liquibase, and by nothing else — see ftfcc Schema. |
|
Read-only |
Owned by the ACS. A dedicated database account holds |
ClickHouse |
Shared, this service’s tables only |
Shared with QoE monitoring, which already owns |
|
Read-only, two tables, optional |
Owned by |
Liquibase cannot be pointed at ftacs: it needs CREATE TABLE DATABASECHANGELOG/DATABASECHANGELOGLOCK and INSERT/UPDATE/DELETE on the lock table before it runs anything, and the read-only account has none of that. Instead, FtacsSchemaCheckRunner verifies at startup, with a single catalog query per vendor, that every table and column this service reads still exists and is visible to the account — see Startup Behaviour.
IotwSchemaCheckRunner does the same for the two iotw tables and warns instead of failing, because a missing grant there costs one device source type and not the service — The iotw Grant Check.
The State Machine as the Single Decision Point
Every change of fcc_test.state and fcc_activation.state goes through one pure function.
com.friendly.fcc.statemachine holds two of them — FccTestStateMachine.evaluate(from, event, context) and FccActivationStateMachine.evaluate(from, event, context) — and nothing in that package is a Spring bean, holds state, performs I/O or logs.
The only dependency is com.friendly.fcc.domain.enums.
The transitions used to be inline if`s and `EnumSet guards spread over FccAckService, FccStalledActivationService, FccActivationHandler and FccTestValidator, where the full rule set could only be reconstructed by reading four classes and no two of them agreed on what a late event should do.
The duplicated STARTABLE_STATES / OPEN_STATES / ACTIVATABLE_STATES sets are gone.
Each evaluate returns a sealed decision, which is what makes a caller’s switch exhaustive without a default branch — a third outcome added later becomes a compile error at every call site instead of falling silently into somebody’s else:
| Outcome | Meaning |
|---|---|
|
The caller writes |
|
The event does not apply and nothing is written. The |
|
A |
An Effect Is a Declaration, Not an Action
The state machine performs no I/O, so it names the side effect and the caller — which owns the transaction, the repositories and the outbox — carries it out. That is what makes the rule set reviewable as a table.
| Machine | Effect | What the caller must do |
|---|---|---|
|
|
Null |
|
|
Write the requested instant. The instant is the caller’s — it comes from the request, not from the state machine. |
|
|
Create the next run as a |
|
|
Stop whatever run the campaign currently owns. Deliberately not applied by |
|
|
Write |
|
|
Re-anchor |
|
|
Write |
|
|
Write |
|
|
Write a STOP to |
FccActivationTransitionService is the single bean that applies both machines' decisions, and it is @Transactional(propagation = MANDATORY): the watchdog’s callers claim their rows with FOR UPDATE SKIP LOCKED, whose locks live until the claiming transaction commits, so a nested or new transaction here would apply the transition outside the scope that holds the claim.
MANDATORY turns "somebody forgot the transaction" into a call-site failure instead of a race nobody reproduces.
fcc_test Transition Table
This is the specification, reproduced from FccTestStateMachine’s class Javadoc.
Anything not listed is `Denied.
| From | Event | To | Effects |
|---|---|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
— |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
— |
|
|
|
|
|
|
|
|
|
|
— |
|
|
|
— |
|
|
|
|
|
|
|
— |
|
|
anything |
denied |
(terminal and unreachable) |
Four rows are load-bearing and easy to "fix" wrongly:
-
RUNNING + ACTIVATION_STARTED → RUNNINGis idempotent on purpose. The ACS confirms a run this service already believes is running — because it applied the ack first — and the confirmation may arrive twice. AnsweringAllowedkeeps it out of the denial logs, where it would be noise nobody can act on. -
USER_STOP → NOT_ACTIVE, andSTOPPEDis the ACS’s word rather than the operator’s. This follows Group Update, which has no "stopped" status at all: its stop returns the group toCreated— the UI’s "Not active" — because that is what the group is afterwards. The same was always true here and only the label said otherwise.FccStopService.stopruns the identical code for a stop and a pause: both clearscheduled_at, stop every activation of the live run with a real STOP command, and close everyPLANNEDreactivation. There is noprev_state, no resume-from-position and no scheduled job to un-pause, so a laterACTIVATE_*starts a brand-new run from either state — a stopped campaign sits in exactly the condition one that never ran sits in.PAUSEDsurvives because it still says the one thing a stop does not: the operator means to come back to it. -
RUNNING + ACTIVATION_STOPPED → STOPPEDis now whatSTOPPEDmeans, and only that: the run was cut short and nobody asked for it. All three callers that could feed this event — the activation-event consumer, the device-state drain and the stop-grace watchdog — skip it whenFccActivationTransitionService.isUserInitiatedStopholds, so an operator’s stop or pause never reaches it (see Activation Event Semantics). That guard was already load-bearing for the pause, because feeding the event on top of one is the single way a pause silently degrades into a stop; it is now also what keeps the two kinds of closed run apart in the campaign list.NOT_ACTIVEis "the operator stopped it",STOPPEDis "the ACS or an outage stopped it" — before this change both readSTOPPEDand nothing distinguished them. -
PAUSED + USER_STOP → NOT_ACTIVE, andPAUSEDis the only closed state that accepts a stop. A paused campaign was stoppable nowhere before this row existed:PAUSEDwas evaluated by the branch that servesNOT_ACTIVE,STOPPED,COMPLETEDandERROR, which allows only theACTIVATE_*events, soPOST /FCC/stopanswered409 test state PAUSED does not accept USER_STOPand the only way to close the campaign was to activate it again — a whole new run on the fleet — and stop that. Two places told the operator to do the impossible: the stop endpoint, andFccTestValidator.validateTiming, which refuses a new booking with "test is PAUSED and its schedule cannot be changed by a save; stop it first, then book it again". It is routed separately rather than added to the four states above, deliberately: "stop something that is already closed" has to stay the 409 that names the state, because that is how an operator stopping forty campaigns learns that somebody else closed one of them a minute earlier.PAUSEDis the one closed state where the answer is not "nothing would change" — it is the state an operator is meant to leave, andNOT_ACTIVEis a different fact about the campaign: nobody is coming back to it. The effect isCLEAR_SCHEDULED_ATand notSTOP_ACTIVE_ACTIVATION, because the pause has already done the stopping — see [nothing-left-to-stop-after-a-pause]. A secondUSER_PAUSEstays denied: it would change nothing, and the 409 names the state the operator is looking at. -
NOT_ACTIVE,PAUSED,STOPPED,ERRORandCOMPLETEDdenyACTIVATION_FINISHED/STOPPED/_FAILEDoutright, but still acceptACTIVATE*. Re-running a finished, stopped or paused campaign is a normal product operation — every run is its ownfcc_activationrow with its own settings snapshot.PAUSED’s denial of the three is untouched by the row above and has to stay: an `ACTIVATION_STOPPEDapplied on top of a pause is the single way a pause silently degrades into a stop. The new row givesPAUSEDUSER_STOPand nothing else.
Stopping a paused campaign carries CLEAR_SCHEDULED_AT only, because a pause leaves nothing on the ACS to stop.
FccStopService.stop runs the identical code for a stop and a pause, so by the time a campaign is PAUSED every activation of its live run has either been closed locally (PLANNED — nothing was ever sent for it) or been sent a real STOP and moved to STOPPING, and every PLANNED reactivation has been closed with it.
STOPPING is deliberately not in FccStopService.LIVE_STATES: that run is draining under ft.fcc.stop-grace with its STOP already on the ACS, and a second command would burn a command sequence to repeat an instruction that is being carried out.
Nothing can re-open the gap either — SCHEDULE_DUE and REACTIVATION_DUE are denied from PAUSED, and an ACTIVATE_* that would create a new run leaves PAUSED in the same transaction — so the only thing this stop can still have to do is clear a pending instant.
Declaring the other effect would be the table claiming that a paused campaign may own something live on the ACS when it may not.
Neither effect is what performs the run half, which is why this is a question about the specification rather than about behaviour: FccStopService queries fcc_activation unconditionally instead of gating on STOP_ACTIVE_ACTIVATION, so an exotic live row would be closed either way.
One consequence is load-bearing: the draining run keeps stop_reason = USER_PAUSE, because the stop finds nothing to re-stamp.
That is what isUserInitiatedStop reads, so when the run finally drains, the ACTIVATION_STOPPED the ACS reports is still not fed to the campaign and a stopped-after-pause campaign cannot slide from NOT_ACTIVE into STOPPED — this product’s word for "the ACS stopped it and nobody asked".
SCHEDULED_FOR_REACTIVATION + ACTIVATE_IMMEDIATELY carries STOP_ACTIVE_ACTIVATION as well as CLEAR_SCHEDULED_AT, and that second effect is not tidiness.
A campaign between daily runs already owns its next one as a PLANNED row, and the operator is asking for a different run right now.
Leaving the old row behind orphans it permanently: the campaign moves to RUNNING, so FccActivationHandler.dispatch then evaluates RUNNING + REACTIVATION_DUE, is denied, and leaves the row PLANNED — which the reactivation poller re-claims every tick for the rest of the installation’s life, gaining one more orphan on every later PLAN_REACTIVATION.
A row that had already reached DISPATCHING or RUNNING is worse: it stays live on the ACS and keeps measuring under a campaign that has moved to a different run.
FccActivationHandler.stopSupersededRun performs it, and delegates to FccStopService.stopLiveRun rather than reimplementing the stop, because the two cases a superseded run splits into are exactly the two that service already distinguishes: a PLANNED row is closed locally with no command — nothing was ever sent for it — while a DISPATCHING/RUNNING one takes the normal STOP_REQUESTED path with a STOP for the ACS.
A second implementation of that split is a second chance to send the ACS a STOP for a run it has never heard of.
Two details of it are load-bearing:
-
It runs before
fcc_test.command_seqis bumped for the new run. The superseded run’s STOP has to take the lower sequence and the newACTIVATEthe higher one, or the ACS discards the ACTIVATE as stale against a STOP that was meant to precede it. -
The reason is
USER_STOP, notERROR. An operator pressing activate is what ended the old run; nothing failed. It is also what makesFccActivationTransitionService.isUserInitiatedStoptrue, so theACTIVATION_STOPPEDthe ACS reports for the old run later does not drag the campaign back out of theRUNNINGstate this activation has just put it in.
This edge closes a bean cycle — FccActivationHandler → FccStopService → FccActivationTransitionService → FccReactivationService → FccActivationHandler — so FccStopService is injected into the handler as an ObjectProvider and resolved per call rather than at construction.
The last edge is the one that cannot move: planning the next daily run is the handler’s job.
FccResultDeadLetterService breaks its own cycle with the ingest service the same way, for the same reason.
ARCHIVED is unreachable in both directions.
It is written only by the M4 legacy ETL, for campaigns that finished years ago in the legacy product and must never be re-run, re-scheduled or reported on as live.
Nothing leads to it and nothing leads out of it; an event naming it is a bug in the caller and is denied like any other.
fcc_activation Transition Table
Reproduced from FccActivationStateMachine’s class Javadoc.
A dash in the Stop reason column means `null, which means leave stop_reason as it is — not "clear it".
| From | Event | To | Stop reason | Effects |
|---|---|---|---|---|
|
|
|
— |
— |
|
|
|
— |
|
|
|
|
|
|
|
|
|
|
|
|
|
— |
|
|
|
|
|
— |
— |
|
|
— |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
the context’s |
|
|
|
|
— |
|
|
|
— |
|
|
|
|
|
— |
|
|
anything |
denied |
(terminal) |
Four rows carry reasoning that is not obvious from the table:
-
PLANNED/DISPATCHING+EVENT_STARTED → RUNNINGexists because the ack and the activation event travel on different Hazelcast queues, which are not ordered against each other. The ACS can have started measuring and reported it before this service applied the ack that authorised it. Denying the report would strand the run inDISPATCHINGuntil the watchdog killed a campaign that is live and streaming results, so the report is accepted and does exactly what the ack would have done, including re-anchoring the window. -
EVENT_STOPPEDsplits on the in-flight count the ACS reports. A positive — or absent — count leaves the run inSTOPPING, because those devices are still producing measurements that belong to this run; only a count of zero closes it immediately. An absent count is treated as positive on purpose: waiting outft.fcc.stop-gracecosts a delay, closing early costs measurements. Both branches stayAllowedrather than one being a no-op, sostopped_atis written even when the stop ack arrives before the stop was recorded locally. -
STOPPING + EVENT_FINISHED → STOPPED, notFINISHED. A run that was stopped is reported asSTOPPEDeven if the ACS managed to complete it afterwards: the operator’s decision is what the run history has to show, and a race between the stop and the last iteration must not be able to relabel it as a clean completion. -
STOPPED,FINISHEDandFAILEDare terminal. Nothing leaves them — not a late event, not a second watchdog tick, not an ACS that comes back and re-reports. The run is over and its measurements have been counted; re-opening it would make a regulatory report disagree with itself.
The one place a caller deliberately diverges from this table is a PLANNED run being stopped.
PLANNED means no command was ever sent — FccActivationHandler.dispatch writes the ACTIVATE and moves the row to DISPATCHING in the same transaction, which is also why a planned run’s command_seq is still the unassigned placeholder 0.
Following PLANNED + STOP_REQUESTED → STOPPING would send the ACS a STOP for a run it has never heard of: it would consume a command sequence, it would be answered as an unknown activation, and the run would sit in STOPPING until the stop grace expired.
FccStopService.closePlannedRun therefore closes it locally as STOPPED with no command at all, and the endpoint answers commandPublished: false — see POST /FCC/stop.
The code marks this as an acknowledged follow-up: a dedicated PLANNED + CANCEL → STOPPED event carrying SET_STOPPED_AT, SET_FINISHED_AT and no PUBLISH_STOP_COMMAND would say the same thing in the one place that is meant to hold transition rules.
Why a Poller, Not Quartz
The scheduled work of this service — "activate a test whose scheduled_at has come due", "dispatch an outbox row", "check for an ack that never arrived", "close a run whose window has passed" — is expressed as @Scheduled pollers reading WHERE state = … AND … ⇐ now() … FOR UPDATE SKIP LOCKED, not as a Quartz job store.
Quartz would mean an additional dedicated database (see the ACS’s own Quartz Database in its architecture docs) and a cluster configuration, all to express what is, in substance, "periodically look at what has come due". A SKIP LOCKED poller needs neither:
-
poller state survives a restart because it lives directly in the business row (
fcc_test.state/scheduled_at,fcc_activation.state/window_end_at,fcc_acs_outbox.state/next_attempt_at) — there is no separate job store to reconcile against the data after a crash; -
FOR UPDATE SKIP LOCKEDgives at-most-once dispatch across however many instances of this service are running, without a distributed lock.
The cost is that the claim is two statements — an ordered, capped, read-only select of candidate ids, then a FOR UPDATE SKIP LOCKED lock of those ids under the same predicate — and that only the first of the two needs a vendor-specific row cap (LIMIT ? on MySQL, ROWNUM outside an ordered inline view on Oracle). Both vendors accept FOR UPDATE SKIP LOCKED itself. See Vendor-Specific Claim SQL.
|
All five pollers belong to |
The Scheduler / Transaction Split
Every poller in this service is two beans, and the split is the same in all five.
A @Component carrying the @Scheduled method holds no transaction and no business logic; a second @Component carries the @Transactional method that claims and applies.
FccActivationScheduler / FccActivationSchedulingService and FccActivationWatchdog / FccActivationClosingService follow the shape FccOutboxDispatcher / FccOutboxPublisher established in M1.
Two independent reasons, and either one alone would be enough:
| Reason | What it prevents |
|---|---|
A tick failure stays out of Spring’s scheduler log |
The |
The Spring proxy stays in play |
A |
The transactional bean therefore claims and applies in one scope, and each pass is its own transaction: a backlog of due campaigns and a backlog of due reactivations are independent, and one failing has no reason to discard the other’s work.
Inside a pass, a failure re-throws rather than being caught per row — FccActivationSchedulingService.startDueScheduledTests says so explicitly.
FccActivationHandler.activate is @Transactional on another bean and therefore joins this transaction, and Spring marks a participating transaction rollback-only the moment the inner proxy sees an exception.
Catching and continuing would start further campaigns, log them as started, and then have the commit throw UnexpectedRollbackException and discard all of them — a log that says a campaign started and a database that says it never did.
Re-throwing keeps the tick honest: nothing was applied, nothing is claimed any more, and the rows are simply claimed again next tick.
Component View
Every component in the diagram is implemented and code-read for this pass.
FccQueueListenerRunner starts a thread per registered FccQueueConsumer bean and all four inbound queues now have one: fccCommandAck, fccResultQueue, fccDeviceStateQueue and fccActivationEventQueue — see Messaging for the full specification.