Metrics

M5 adds the queue-depth, queue-age, outbox-row, outbox-age, command-ack-latency, result-buffer-depth and Hazelcast-connectivity meters this page introduces below; every meter that already existed through M4 is listed here too, because a scrape configuration is written from this one table and not from the source. Endpoint: GET /actuator/prometheus, gated by management.endpoints.web.exposure.include: health,info,prometheus, on the same port 8080 as the REST API — there is no separate metrics port. See Deployment for the Helm chart’s two discovery styles (pod annotations and an optional ServiceMonitor).

Names below are the Prometheus-rendered names — what a scrape configuration and an alert rule actually match against — not the Micrometer names the source code registers. The convention throughout is fcc_<area>_<what>, from the Micrometer convention fcc.<area>.<what>.

How a Micrometer Name Becomes a Prometheus Name

Two independent transformations happen between Counter.builder("fcc.legacy.import.anchors.created") and the line a scrape sees, and both are worth knowing before writing an alert rule, not after one silently matches nothing.

  1. Dots become underscores. fcc.queue.depth renders as fcc_queue_depth. Mechanical, and the only step most engineers expect.

  2. A name ending in a reserved OpenMetrics suffix has that suffix stripped before exposition, and a Counter then has _total appended back on unconditionally. PrometheusNaming.sanitizeMetricName (io.prometheus.metrics.model.snapshots, the library Micrometer’s Prometheus registry is built on since Spring Boot 3.5’s Micrometer line) strips a trailing member of a fixed reserved-suffix list — _total, _created, _bucket, _info, _sum, _count among them — repeatedly, until none apply. fcc.legacy.import.anchors.created therefore loses its trailing _created during sanitisation, and the Prometheus Counter type then appends the _total every OpenMetrics counter is required to end in. The result is fcc_legacy_import_anchors_total, not fcc_legacy_import_anchors_created_total. The same happens to fcc.legacy.import.activations.created → fcc_legacy_import_activations_total.

    This was verified by decompiling the exact dependency versions this project resolves (micrometer-registry-prometheus and prometheus-metrics-model, both pulled transitively through spring-boot-starter-parent 3.5.6) rather than by observing a running scrape — PrometheusNamingConvention.name() calls PrometheusNaming.prometheusName() then PrometheusNaming.sanitizeMetricName(), and the latter’s bytecode is the loop described above, operating on the exact RESERVED_METRIC_NAME_SUFFIXES array quoted here. It was not additionally confirmed against a live /actuator/prometheus response from a running instance; if a scrape ever shows something different from this table, trust the scrape and re-open this note.

    No other meter name in the table below collides with a reserved suffix — fcc.legacy.import.rows.read / .written / .skipped, .batches, .devices.snapshotted / .unresolved, .passes.completed / .failed all render as plain _total counters with no stripping.

  3. A Gauge whose Micrometer name already ends in a word matching its baseUnit (fcc.queue.age.seconds with baseUnit("seconds"), fcc.outbox.age.seconds likewise) is not suffixed a second time — the convention only appends _<baseUnit> when the name does not already end with it, so these render exactly as spelled: fcc_queue_age_seconds, fcc_outbox_age_seconds.

  4. A Timer always renders as a family of samples with a _seconds base (Micrometer’s Prometheus registry standardises timers on seconds regardless of the unit passed to Duration), plus the OpenMetrics summary suffixes: _seconds_count, _seconds_sum, and _seconds_max (the rolling time-window maximum Micrometer publishes by default for every Timer, since no percentile histogram is configured on this one).

Queue Metrics

Tag queue, five values: fccCommandQueue, fccCommandAck, fccResultQueue, fccDeviceStateQueue, fccActivationEventQueue (FccQueueNames).

Name Type Tags Measures Alert?

fcc_queue_depth

Gauge

queue

IQueue.size() of the named Hazelcast queue, sampled every ft.fcc.metrics.sample-interval.

Yes — a sustained rise on fccCommandQueue or fccResultQueue means the ACS side (or the local take() consumer) has stopped draining it.

fcc_queue_age_seconds

Gauge

queue

Age of the head element: now minus the publish timestamp carried inside the message, taken from IQueue.peek(). 0 for an empty queue.

Yes, on the three queues where it is observable (see Where NaN Appears, and What It Means) — a rising age with a shallow fcc_queue_depth is a stuck consumer more than a backlog.

Outbox Metrics

Tag state. fcc_outbox_rows carries all five live fcc_acs_outbox.state values (FccOutboxStateEnum); fcc_outbox_age_seconds carries only PENDING and SENT — an ABANDONED row is not waiting for anything, so it has no age.

Name Type Tags Measures Alert?

fcc_outbox_rows

Gauge

state = PENDING | SENT | ABANDONED | (the other live FccOutboxStateEnum values)

COUNT(*) per state, each from its own index-only statement against fcc_acs_outbox — see Why the Gauges Are Sampled on a Timer, Never Computed on Scrape. PENDING and SENT refresh on every sample tick (15s); ABANDONED refreshes only every five minutes. ABANDONED is terminal and nothing purges it, so it grows for the life of the installation exactly like ACKED — counting a set that only grows on a 15-second tick would buy nothing, and five minutes is well inside any alert window that would fire on it. An operator watching this series and expecting it to move every tick is watching the wrong cadence, not a stuck sampler.

Yes on ABANDONED trending up — see Troubleshooting, "Outbox rows reaching ABANDONED".

fcc_outbox_age_seconds

Gauge

state = PENDING | SENT

Age of the oldest row in that state, measured entirely against this JVM’s own clock (created for PENDING, sent_at for SENT) — no cross-host clock skew possible.

Yes — this is the metric to alert on for the command path, not fcc_queue_age_seconds{queue="fccCommandQueue"}, which cannot be produced at all (Where NaN Appears, and What It Means).

Command Acknowledgement Latency

Name Type Tags Measures Alert?

fcc_command_ack_latency_seconds_count

Timer (count component)

result = ACCEPTED | REJECTED | DUPLICATE

Number of acks applied, tagged by the wire result — a republished command’s DUPLICATE ack keeps its own series rather than being merged into ACCEPTED.

Not on its own; read alongside _sum for a rate, or _max.

fcc_command_ack_latency_seconds_sum

Timer (sum component)

result

Cumulative seconds from offering the command to fccCommandQueue to applying its ack, per result. _sum / _count over a range gives the mean.

No.

fcc_command_ack_latency_seconds_max

Timer (rolling max component)

result

Rolling time-window maximum ack latency.

Yes on ACCEPTED — a rising max is the signal that ft.fcc.command-ack-timeout (default 60s) is no longer sized against reality.

fcc_command_ack_untimed_total

Counter

(none)

Acks applied that could not be timed because their outbox row’s sent_at is null — the row was written by something other than the dispatcher, so now - null is not a duration and is not fed into the distribution.

Yes if non-zero and rising — it means an ack is being applied for a row the normal outbox path did not produce.

None of the three tag values of fcc_command_ack_latency_seconds_* is pre-registered: a Timer with no observations is simply absent from the scrape (the normal behaviour for a timer), rather than a fake zero polluting a latency distribution an alert reads.

Recording this timer is now guarded in FccAckService, deliberately, because of where it sits rather than what it measures. The call happens inside the same @Transactional method that has just marked the outbox row ACKED and moved the activation on — and the ack itself is not replayable, because the ACS already considers it delivered. Before this guard, a RuntimeException escaping the meter call (a registry failure, however unlikely with a bounded tag cardinality of three FccAckResultEnum values) would have rolled back both writes, stranding the run in DISPATCHING for a reason that has nothing to do with the ack itself. Losing one observation of a distribution is a cost nobody can measure; losing an ack that the ACS will never resend is not.

Result Buffer and Hazelcast Connectivity

Name Type Tags Measures Alert?

fcc_result_buffer_depth

Gauge

(none)

Measurements currently held in the in-memory buffer that doubles as the ClickHouse retry queue (FccResultIngestService.bufferDepth()), bounded by ft.fcc.result.queue-capacity (default 10000).

Yes — approaching the capacity is the leading indicator before measurements start landing in fcc_result_dead_letter.

fcc_hazelcast_connected

Gauge

(none)

1 when the Hazelcast client has a live cluster connection (isRunning() and a non-empty member list — isRunning() alone is not connectivity, since an async-start client with ReconnectMode.ASYNC reports running for the whole of an outage), 0 otherwise.

Yes — and it is the metric that explains the ten fcc_queue_* NaN series when it drops to 0 (Where NaN Appears, and What It Means).

Where NaN Appears, and What It Means

Every gauge above is sampled on a timer, never computed on scrape (Why the Gauges Are Sampled on a Timer, Never Computed on Scrape), and an unknown value is reported as NaN, never 0 — 0 is itself a statement ("the queue is empty", "the outbox is drained"), the exact opposite of what is true when the sample could not be taken at all. A NaN on a dashboard therefore reads as a bug unless a reader already knows these four cases:

Series Why it is NaN

All ten fcc_queue_* series (fcc_queue_depth and fcc_queue_age_seconds, all five queues), whenever fcc_hazelcast_connected is 0

The whole sampler loop short-circuits on a disconnected client: IQueue.size() and IQueue.peek() would both throw the same offline exception, five times over, for no information gained. 0 is deliberately not used here either, for the reason above.

fcc_queue_age_seconds{queue="fccCommandQueue"} and {queue="fccCommandAck"}, always, even against a fully healthy cluster

Not observable from inside this service, and nothing substitutes for it. The age of a queue’s head is now minus the publish time carried inside the message, and FccCommandDto / FccCommandAckDto carry no publish timestamp at all — com.friendly.fcc.dto is a byte-identical, frozen copy of a cross-repo contract (DTO Contract Rules), and adding a field to it is not a decision this service gets to make alone. State this plainly rather than approximating it: fcc_outbox_age_seconds is the honest substitute for the command side, and it is in fact the better measurement of the two, because both of its ends — created and sent_at on fcc_acs_outbox — are read off this JVM’s own clock, with no ACS-to-service clock skew in it at all.

fcc_outbox_rows and fcc_outbox_age_seconds, every tag, whenever ftfcc cannot be read

An empty table (no rows in a state) reports 0 — that is real information, correctly zero. An unreachable database reports NaN on every tag of both meters, because "the table has no rows in that state" and "the query could not run" are different facts and collapsing them into the same 0 would make an outage indistinguishable from a healthy drained outbox.

(not a NaN case, but the paired fact) three of the five queue ages are observable, and carry clock skew

fcc_queue_age_seconds{queue="fccResultQueue"|"fccDeviceStateQueue"|"fccActivationEventQueue"} are computed from a publish timestamp the ACS stamps on its own host (sentAt / at on FccResultBatchDto, FccDeviceStateBatchDto, FccActivationEventDto), subtracted from this service’s own now. An ACS clock running behind inflates all three; one running ahead would produce a negative, which is clamped to 0 rather than reported, because a queue cannot have been drained before it was filled. Diagnostic: if these three rise together while fcc_outbox_age_seconds does not, suspect the ACS host’s clock before suspecting a stuck consumer — fcc_outbox_age_seconds has no ACS clock in it at all and would not move for a clock-skew cause.

Why the Gauges Are Sampled on a Timer, Never Computed on Scrape

A single @Scheduled sampler, FccMetricsSampler (ft.fcc.metrics.sample-interval, FCC_METRICS_SAMPLE_INTERVAL, default 15s, one of the eight @Scheduled ticks this service runs on virtual threads with no shared pool to starve — see the scheduling note), refreshes a set of in-memory holders the gauges read from. Every tag value is registered at construction time, not on first observation, so a drained queue or an empty outbox state still produces a series — a series that vanishes when a backlog clears is indistinguishable from a dead scrape target, which is precisely the failure mode a monitoring system exists to catch.

Two independent reasons this is not computed inline on every scrape:

  • IQueue.peek() deserializes the head element on every call, and the head of fccResultQueue in particular is a batch of up to a hundred measurements (FccResultBatchDto). Paying that deserialization cost on every Prometheus scrape — which may run far more often than the sample interval, or from more than one scraper — is a real, avoidable cost on a busy installation.

  • A scrape must never block. A disconnected Hazelcast client and a database in failover are both states this service is explicitly designed to survive without going DOWN (Health); a gauge that queried either of them inline would hang the whole /actuator/prometheus response behind a connection timeout, taking every other meter on the page down with it.

The sampler never throws: a sampler that died on a database outage would freeze every gauge at its last value, which is the one moment an operator is actually looking at them.

Since M5, not throwing is not enough on its own — the sampler also bounds how long each dependency is allowed to take, because exception isolation alone left a hung call free to freeze gauges that had nothing to do with it.

As the sampler first shipped, its outbox query carried no timeout of its own, and the Hazelcast client — present since long before the sampler, and shared with the queue consumers — had never had its invocation timeout set, so it sat at Hazelcast’s own default of 120 seconds; while either was hanging, every gauge in the tick — including the two that depend on neither — held its last value rather than reporting NaN, which is the one thing this class must never do (Where NaN Appears, and What It Means). Three changes close that:

  • Order. sample() now runs the two cheap, dependency-free gauges first — fcc_result_buffer_depth (an in-memory queue size) and fcc_hazelcast_connected (a local check of the client’s own member list) — before anything that can block, so a slow dependency delays only the gauges that depend on it, never the ones that do not.

  • The outbox query has a five-second deadline, on a JdbcTemplate of its own rather than the shared one: the shared template runs inside the dispatcher’s own transactions and must not carry a timeout meant for a query that never runs in one.

  • The Hazelcast client’s invocation timeout is five seconds, set explicitly (ClientProperty.INVOCATION_TIMEOUT_SECONDS) in HazelcastClientConfig rather than left at Hazelcast’s default of 120s. This is safe for the client’s other users too — the take() consumer loops and the outbox publisher’s offer() — because an invocation timeout bounds retrying a failed call, never a call legitimately waiting for one: a take() parked on an empty queue is untouched by it, however long it blocks, and a mutating call like offer() is non-retryable in the first place and already fails on the spot when the connection drops.

One correction worth stating plainly, because it is the one figure this page would otherwise get wrong by carrying over the pre-M5 number: 120 seconds was never the bound on a completely silent socket. It was the client’s retry ceiling. A member that stops responding without closing the connection is actually bounded by the client’s heartbeat timeout, which is 60 seconds and was deliberately not lowered alongside the invocation timeout — an aggressive heartbeat causes false reconnects on an ordinary GC pause, which is a worse failure than a slow sample. So the worst case for the three queue gauges that can still block is about one heartbeat interval, and what actually protects the other gauges from it is the ordering above, not the invocation timeout.

Everything Else on /actuator/prometheus

Every meter that existed before M5, listed so a scrape configuration is complete without reading the source.

Name Type Tags Measures

fcc_config_export_tests_total

Counter

(none)

M5. Campaigns written into a PUT /FCC/item/export document.

fcc_config_import_documents_total

Counter

(none)

M5. Configuration documents read into a campaign form by PUT /FCC/item/import. One per successful parse, with no outcome tag — it replaced a per-<test> counter tagged IMPORTED / FAILED, which could not survive DEV-1381 #13: the endpoint answers ONE campaign and creates nothing, so there are no per-element outcomes left to tag and a refusal is an HTTP 400 that the error metrics already carry. A rate on this counter is "how often an operator imports a configuration", which is the question the old one could not answer.

fcc_clickhouse_result_inserted_total

Counter

(none)

Speed result rows written to ClickHouse.

fcc_clickhouse_result_failed_total

Counter

(none)

Speed result rows whose ClickHouse insert failed.

fcc_clickhouse_latency_inserted_total

Counter

(none)

Latency result rows written to ClickHouse.

fcc_clickhouse_latency_failed_total

Counter

(none)

Latency result rows whose ClickHouse insert failed. Rising while fcc_clickhouse_result_inserted_total keeps advancing is the signature of a latency-only schema lag — see the V3 migration troubleshooting row.

fcc_queue_message_unreadable_total

Counter

(none)

Hazelcast queue messages skipped because they could not be deserialized as the expected DTO type.

fcc_queue_consumer_error_total

Counter

(none)

Failures of a queue consumer take() loop.

fcc_queue_message_lost_total

Counter

(none)

Queue messages given up on after the bounded consume retry.

fcc_watchdog_window_closed_total

Counter

(none)

Runs the watchdog closed as FINISHED(WATCHDOG) because their measurement window expired.

fcc_watchdog_stop_closed_total

Counter

(none)

Runs the watchdog force-closed to STOPPED because ft.fcc.stop-grace expired.

fcc_outbox_published_total

Counter

(none)

Commands accepted by fccCommandQueue.offer().

fcc_outbox_publish_failed_total

Counter

(none)

Publish attempts that failed and will be retried.

fcc_outbox_abandoned_total

Counter

(none)

Commands given up on after ft.fcc.outbox-max-attempts (default 10) — see Troubleshooting, "Outbox rows reaching ABANDONED".

fcc_activation_event_applied_total

Counter

(none)

fccActivationEventQueue events applied to a run.

fcc_activation_event_dropped_total

Counter

(none)

Activation events discarded because they name no run this service knows (the mirror of ACTIVATION_GONE).

fcc_reactivation_not_planned_total

Counter

(none)

Daily runs a campaign owed that were never created because its previous run never reached FINISHED.

fcc_reactivation_dispatch_skipped_total

Counter

(none)

Planned reactivation runs never sent because their campaign’s settings were no longer runnable at dispatch time.

fcc_device_state_applied_total

Counter

(none)

Device states written to fcc_device_state.

fcc_device_state_unroutable_total

Counter

(none)

Device states discarded because they name a run this service does not know.

fcc_device_state_invalid_total

Counter

(none)

Device states discarded because the message is missing a field it is keyed on — a malformed ACS message.

fcc_legacy_import_rows_read_total

Counter

(none)

Legacy FCC result rows read from ftacs.

fcc_legacy_import_rows_written_total

Counter

(none)

Legacy FCC result rows written to ClickHouse.

fcc_legacy_import_rows_skipped_total

Counter

(none)

Legacy rows skipped because crash recovery found them already in ClickHouse.

fcc_legacy_import_batches_total

Counter

(none)

Batches of legacy results committed.

fcc_legacy_import_anchors_total

Counter

(none)

Anchor campaigns (source=LEGACY) created for legacy update groups. Rendered name has its .created stripped — see How a Micrometer Name Becomes a Prometheus Name.

fcc_legacy_import_activations_total

Counter

(none)

Synthetic activations created for legacy day-windows. Same stripping as the row above.

fcc_legacy_import_devices_snapshotted_total

Counter

(none)

Devices added to the frozen fleet of an imported anchor campaign.

fcc_legacy_import_devices_unresolved_total

Counter

(none)

Legacy serials no live ftacs.cpe row matches any more.

fcc_legacy_import_passes_completed_total

Counter

(none)

Legacy import passes that reached the end of their result kind.

fcc_legacy_import_passes_failed_total

Counter

(none)

Legacy import passes that ended on an unexpected failure.

fcc_result_parked_total

Counter

(none)

Results written to fcc_result_dead_letter instead of ClickHouse.

fcc_result_lost_total

Counter

(none)

Results that could not even be parked — genuinely gone. The one counter here worth paging on.

fcc_result_resent_total

Counter

(none)

Results handed back from the dead letter table into the ingest path by POST /FCC/admin/result-dead-letter/resend.

fcc_result_discarded_total

Counter

(none)

Dead letter rows given up on because their payload could not be read back.

fcc_result_resend_skipped_late_total

Counter

(none)

Dead letter rows a resend could not use because they carry reason=LATE and were left PENDING by an earlier build — see the dead-letter troubleshooting row. The Micrometer name is fcc.result.resend.skipped.late, not fcc.result.deadletter.skipped.late — the two names have appeared inconsistently in older prose; this is the one the code actually registers.

fcc_result_dedup_skipped_total

Counter

(none)

Results already stored under their natural key — an at-least-once redelivery correctly discarded.

fcc_result_unroutable_total

Counter

(none)

Results discarded because they name no activation this service knows.

fcc_result_late_total

Counter

(none)

Results that arrived after their run’s stop grace and were excluded from the run on purpose — see The After-Stop Window.

Not Exposed Here

rowsLateSkipped and rowsInvalid of the legacy import status response have no Micrometer counter — they are visible on POST /FCC/admin/legacy-import/status and in the log only (Watch Progress). A scrape configuration built from this page alone will not see them; that is by design, not an omission this page should paper over.