JVM and connection-pool metric names: what each producer emits

Version 1.6.9 | Updated: September 04, 2026

JVM and connection-pool metric names: what each producer emits

Everything here was measured on 05.08.2026 against running systems; nothing is quoted from Micrometer or JMX exporter documentation. The hosts are named by role rather than by address: this page is published, and the numbers are what matters, not which machine produced them.

Where the numbers came from

  • JMX, WildFly/ACS 6.4 — an ACS application server, exporters :5556 (Hikari) and :5557 (JVM). 110 distinct names.

  • JMX, Hazelcast — same host, :9101, exporter_type=hazelcast. 208 java_lang_* lines plus 7176 com_hazelcast_*.

  • Micrometer, ACS 7.1.1 — an ACS 7.1 test container, :8080/actuator/prometheus. 98 names.

  • Micrometer, ACS 6.5.2 — the acs OKD cluster, via ft-prometheus-acs, job="ftacs". 167 series names.

6.5 and 7.1 are the same producer. The two sets are identical: 48 shared JVM/pool names, zero on either side alone. Two capture artefacts had to be removed before that was visible, and both are worth knowing if these numbers are ever re-taken:

  • # HELP names a summary once; /api/v1/series lists its _count/_sum/_bucket expansions. Comparing one against the other invents differences. Normalise those three suffixes.

  • _max must not be normalised away. In Micrometer it is a metric in its own right — hikaricp_connections_max is what the pool-exhaustion alert queries, and stripping it made this table briefly claim the metric does not exist, when it has 16 series in the cluster and 10 on-prem.

Three producer spellings, not two

This is the part that matters, and it is not what the plan assumed.

  1. JMX exporter, lowercase — java_lang_runtime_uptime, java_lang_classloading_loadedclasscount, java_lang_threading_threadcount. Emitted by the WildFly/ACS JVM exporter (:5557, exporter_type=jvm) on both production ACS hosts. 208 lines, 0 CamelCase.

  2. JMX exporter, CamelCase — java_lang_Runtime_Uptime, java_lang_Memory_HeapMemoryUsage_used. Emitted by the Hazelcast exporter (:9101, exporter_type=hazelcast). 208 lines, 0 lowercase.

  3. Micrometer — process_uptime_seconds, jvm_memory_used_bytes, hikaricp_connections_active. ACS 6.5+, every Spring Boot service, and Kubernetes.

The two JMX spellings are the same exporter with different config (lowercaseOutputName), and both are present in production Prometheus at the same time under the same instance label, separated only by exporter_type.

What this already costs us, measured

  • jmx-dashboard is empty for the JVM it is named after. Its expressions query CamelCase and are scoped by exporter_type=~"$app". On production: count(java_lang_Runtime_Uptime{exporter_type="jvm"}) = 0; count(java_lang_Runtime_Uptime{exporter_type="hazelcast"}) = 2; count(java_lang_runtime_uptime{exporter_type="jvm"}) = 2. So the dashboard shows something only when pointed at Hazelcast. The WildFly JVM data has been collected all along and read by nothing. This matches the note already in the test-monitoring skill — “the lowercase half has never been read by anything” — and is a defect in its own right, independent of this plan.

  • The ACS pool alert is blind on-prem and works in Kubernetes. hikaricp_connections_active (Micrometer) — on-prem 10 series, none of them ACS; in the cluster it covers ftacs itself (4 series). hikaricp_active_connections (JMX) — on-prem 6 series, all ACS; in the cluster 0. The alert queries the Micrometer spelling.

  • hikari.json is entirely empty in Kubernetes. Every expression in it queries the JMX spelling, which no producer there emits.

Canonical names

ft: prefix. One entry per concept: the canonical name, then the expression per producer. — means the producer does not carry it and the row is served by the other side (or by node-exporter, noted explicitly).

  • ft:jvm_uptime_seconds

    • jmx/lowercase: java_lang_runtime_uptime / 1000

    • jmx/CamelCase: java_lang_Runtime_Uptime / 1000

    • micrometer: process_uptime_seconds

  • ft:jvm_heap_used_bytes

    • jmx/lowercase: java_lang_memory_heapmemoryusage_used

    • jmx/CamelCase: java_lang_Memory_HeapMemoryUsage_used

    • micrometer: jvm_memory_used_bytes{area="heap"}

  • ft:jvm_heap_max_bytes

    • jmx/lowercase: java_lang_memory_heapmemoryusage_max

    • jmx/CamelCase: java_lang_Memory_HeapMemoryUsage_max

    • micrometer: jvm_memory_max_bytes{area="heap"}

  • ft:jvm_nonheap_used_bytes

    • jmx: java_lang_memory_nonheapmemoryusage_used / java_lang_Memory_NonHeapMemoryUsage_used

    • micrometer: jvm_memory_used_bytes{area="nonheap"}

  • ft:jvm_threads_live

    • jmx: java_lang_threading_threadcount / java_lang_Threading_ThreadCount

    • micrometer: jvm_threads_live_threads

  • ft:jvm_threads_daemon

    • jmx: java_lang_threading_daemonthreadcount / java_lang_Threading_DaemonThreadCount

    • micrometer: jvm_threads_daemon_threads

  • ft:jvm_threads_peak

    • jmx: java_lang_threading_peakthreadcount / java_lang_Threading_PeakThreadCount

    • micrometer: jvm_threads_peak_threads

  • ft:jvm_classes_loaded

    • jmx: java_lang_classloading_loadedclasscount / java_lang_ClassLoading_LoadedClassCount

    • micrometer: jvm_classes_loaded_classes

  • ft:jvm_gc_time_seconds_total

    • jmx: java_lang_garbagecollector_collectiontime / 1000 (both spellings)

    • micrometer: jvm_gc_pause_seconds_sum

  • ft:jvm_gc_count_total

    • jmx: java_lang_garbagecollector_collectioncount

    • micrometer: jvm_gc_pause_seconds_count

  • ft:jvm_cpu_process_ratio

    • jmx: java_lang_operatingsystem_processcpuload / java_lang_OperatingSystem_ProcessCpuLoad

    • micrometer: process_cpu_usage

  • ft:jvm_open_file_descriptors

    • jmx: java_lang_operatingsystem_openfiledescriptorcount

    • micrometer: process_files_open_files

  • ft:pool_connections_active

    • jmx: hikaricp_active_connections

    • micrometer: hikaricp_connections_active

  • ft:pool_connections_idle

    • jmx: hikaricp_idle_connections

    • micrometer: hikaricp_connections_idle

  • ft:pool_connections_total

    • jmx: hikaricp_total_connections

    • micrometer: hikaricp_connections

  • ft:pool_connections_max

    • jmx: hikaricp_maximum_pool_size

    • micrometer: hikaricp_connections_max

  • ft:pool_connections_min

    • jmx: hikaricp_minimum_idle

    • micrometer: hikaricp_connections_min

  • ft:pool_threads_awaiting

    • jmx: hikaricp_threads_awaiting_connection

    • micrometer: hikaricp_connections_pending

Deliberately not mapped

These are queried by jmx-dashboard today and get no canonical name. Each is a host-level reading that the JVM happens to report; node-exporter already scrapes the same host and is both more accurate and unaffected by which JVM is running.

  • java_lang_OperatingSystem_TotalPhysicalMemorySize → node_memory_MemTotal_bytes

  • java_lang_OperatingSystem_FreePhysicalMemorySize → node_memory_MemAvailable_bytes

  • java_lang_OperatingSystem_TotalSwapSpaceSize → node_memory_SwapTotal_bytes

  • java_lang_OperatingSystem_FreeSwapSpaceSize → node_memory_SwapFree_bytes

  • java_lang_OperatingSystem_CommittedVirtualMemorySize → drop (JVM address space, not useful next to the RSS that node-exporter and cAdvisor already show)

  • java_lang_OperatingSystem_SystemCpuLoad → node_cpu_seconds_total based panel

  • java_lang_OperatingSystem_AvailableProcessors → count(node_cpu_seconds_total) by (instance)

  • java_lang_OperatingSystem_ProcessCpuTime → covered by ft:jvm_cpu_process_ratio

  • java_lang_Runtime_StartTime → covered by ft:jvm_uptime_seconds

  • java_lang_Threading_CurrentThreadCpuTime → drop (per-thread sampling, no consumer)

  • java_lang_Threading_TotalStartedThreadCount → drop (no panel reads it meaningfully)

  • jvm_threads_deadlocked, jvm_threads_deadlocked_monitor → these come from the JMX exporter’s own JVM collector, which is not enabled on the production exporters (0 jvm_* names in the 110 captured). They have no producer today on either side; drop the panels rather than pretend.

Consequence for the plan

Task 2 must emit each canonical name from three groups, not two — lowercase JMX, CamelCase JMX, Micrometer. The alternative (normalising the two JMX spellings at the exporter by fixing lowercaseOutputName on one side) changes a producer on a production host and would silently break whatever currently reads the CamelCase Hazelcast series, so it is not part of this plan. It is worth a separate ticket.

Left on producer names on purpose (task 4b)

These eight still appear in jmx-dashboard.json. All are emitted only by the Hazelcast exporter (exporter_type=hazelcast) — measured on-prem; zero series in Kubernetes and on the qa stand — except jvm_info, which Micrometer also emits for every Spring Boot service.

  • jvm_memory_pool_bytes_used / _max / _committed — per memory pool. The canonical heap/nonheap names aggregate id away deliberately, so there is no equivalent. Canonicalising per-pool needs its own rules and its own measurement; it is not free and no consumer has asked for it.

  • jvm_memory_bytes_committed — committed heap, no consumer beyond this panel.

  • jvm_info — a label-only series; label_join on it drives the JDK panel.

  • jvm_memory_bytes_used — only inside label_values(…​, area), which builds the $memarea variable. The canonical names split heap from nonheap by name rather than by label, so no variable can be derived from them.

  • java_lang_ClassLoading_UnloadedClassCount, java_lang_Threading_TotalStartedThreadCount — extra series inside panels whose primary series is already canonical.

The consequence, stated so nobody rediscovers it: those panels show Hazelcast’s JVM on-prem and nothing anywhere else. That is a smaller version of the problem this whole exercise fixes, left open rather than half-fixed.

Labelled canonical names (task 30, closed)

Two dashboards needed a label the aggregate rules deliberately drop, so the set gained three names that keep exactly one label each and nothing more:

  • ft:jvm_gc_time_seconds_by_collector / ft:jvm_gc_count_by_collector — keep the collector. Every producer spells it differently, measured on production: WildFly puts it in name, the Hazelcast exporter and Micrometer in gc, and Micrometer adds action and cause which multiply the series. All three are normalised onto gc.

  • ft:jvm_memory_used_bytes_by_area — keeps area. Micrometer and the Hazelcast exporter carry that label; the WildFly JMX side has no such label at all, since heap and nonheap are separate metric names there, so it is synthesised.

The original reasoning, kept because it is why these names are narrow:

  • hazelcast-overview asks for GC per collector — jvm_gc_collection_seconds_* with gc="G1 Old Generation". Both canonical GC rules aggregate that away (sum without (action, cause, gc) on the Micrometer side), so the selector would match nothing. Needs ft:jvm_gc_time_seconds_by_collector or the panels rewritten to the aggregate.

  • ft-ui-services asks for memory by area — sum by (area) (jvm_memory_used_bytes{…​}). The canonical names split heap from nonheap by NAME rather than by label, so there is no area to group on. Needs either a labelled canonical name or two explicit series per panel.

Migrated in this pass: acs and system-dashboard-basic.

acs also gained an exporter_type="jvm" filter, and that is the substantive change. Its panels queried CamelCase java_lang_* filtered only by job="JmxExporter", and CamelCase is emitted solely by the Hazelcast exporter: measured on production, java_lang_Threading_DaemonThreadCount{exporter_type="jvm"} returns 0 series and the same query with exporter_type="hazelcast" returns 2. The ACS dashboard has been showing Hazelcast’s JVM all along. The canonical series spans both exporters, so without the filter it would now show both.

The other half of the contract: consumers (task 27, closed)

Everything above is the producer side — what the exporters emit. The mapping is only worth something if the queries actually name those metrics, and until this pass nothing checked that they did.

misc/testing/fixtures/metric-inventory.txt is the producer side written down: 10322 metric names captured 05.08.2026 from the live Prometheus of all three environments, each prefixed by which of them emits it. Regenerate it with misc/testing/scripts/capture-inventory.sh — never by hand, because a name added to silence a finding removes the only reason the check means anything.

misc/testing/scripts/expr-coverage.sh extracts the metric names from every dashboard expr and every alert expr (1127 expressions, including the Helm chart’s rules when that repository is checked out alongside) and reports any name with no producer. validate-all Test 40 runs it.

What it found

Five live queries named metrics that do not exist anywhere:

  • process_resident_memory_max_bytes, in nodes-monitoring and system-dashboard. Producers emit process_resident_memory_bytes and process_virtual_memory_max_bytes; the crossed name has never existed. Both panels also carried a duplicated target pair, and the broken query was the mangled half of one — the working copy sat right beside it, so the panel drew a line and looked fine. The duplicate pair is removed.

  • windows_cs_physical_memory_bytes, windows_os_physical_memory_free_bytes and windows_os_virtual_memory_bytes. windows_exporter dropped the cs and os collector prefixes in 0.20; the monitored Windows host emits windows_memory_physical_total_bytes, windows_memory_physical_free_bytes and windows_memory_commit_limit. Verified against on-prem: each replacement returns 1 series, each old name returns 0.

And two alert families that could never fire, both left in place because fixing them is a separate change:

  • hz_cluster_size — no producer, and no recording rule defines it either. Both prometheus-grafana-stack/prometheus/hazelcast_rules.yml and the chart’s hazelcast.yml alert on it.

  • probe_ssl_earliest_cert_expiry — the blackbox exporter is not deployed in any environment, so the certificate-expiry alerts in prometheus_rules.yml and the chart’s platform.yml are decorative.

Why Hazelcast could not be fixed by renaming a query (task 32, closed)

Hazelcast embeds its randomly generated instance name in the metric name rather than a label. The same three metrics appear under six different names across the three environments — com_hazelcast_thirsty_margulis_memberCount, com_hazelcast_laughing_lamport_memberCount, com_hazelcast_compassionate_wilbur_memberCount and so on — and the name changes on every restart, so no query could be written that survives one.

The canonical_hazelcast group in canonical_rules.yml extracts the instance from name into an hz_instance label:

  • ft:hazelcast_cluster_members

  • ft:hazelcast_running

  • ft:hazelcast_cluster_safe

  • ft:hazelcast_local_member_safe

  • ft:hazelcast_executor_queue_size

Two details in those rules are load-bearing. The type matcher is what keeps com_hazelcast_Metrics_queueSize out — an unrelated per-executor metric that matches the same name regex, with 23 series on the QA stand and hundreds in Kubernetes, every one of which would arrive labelled hz_instance="Metrics". And the instance is extracted from the metric name rather than read from the name label, because name means the instance on HazelcastInstance and HazelcastInstance.PartitionServiceMBean but the EXECUTOR on HazelcastInstance.ManagedExecutorService.

All four Hazelcast alerts were inert before this, each differently, and each convincing to read:

  • HazelcastClusterSizeChanged read hz_cluster_size, which nothing defines.

  • HazelcastNodeDown used count(…​) < 1. On an empty selector count() returns no samples rather than 0, so the comparison filtered nothing in exactly the situation the alert existed for. It now uses absent().

  • HazelcastHighPendingOps read com_hazelcast_Metrics_completedPacketCount, which counts completed packets rather than pending ones and is flat at zero on production (98 series, max 0 over seven days). Had it ever moved, being cumulative, it would have fired permanently. It now reads the managed-executor queue depth. The chart had already spotted the first problem and replaced the metric with com_hazelcast_Metrics_operationExecutorQueueSize, which no producer emits either.

  • isClusterSafe — the reason this stack scrapes JMX over the native endpoint — was present on every instance and readable by nothing. HazelcastClusterUnsafe now alerts on it.

hazelcast_canonical_test.yml and hazelcast_alert_test.yml cover both halves, feeding the instance names measured in each environment. The chart’s hazelcast.yml is now byte-identical to hazelcast_rules.yml and the same tests pass against it, which matters because the two copies had already drifted into different broken states.

The remaining 24 absences

misc/testing/fixtures/metric-inventory-exceptions.txt lists every name a query asks for that no producer emits, each with the reason. The largest group is mysql.json: eleven query-cache and InnoDB-memory counters that MySQL 8 removed outright, inherited from the upstream Percona dashboard written for MySQL 5.x. Those panels are permanently empty on every database we monitor.

One dashboard set, two destinations (task 31, closed)

The dashboards shipped twice — from this repo for on-prem, and from ft-charts-library/charts/ft-monitoring/dashboards for Kubernetes — and nothing kept the two in step. Measured 05.08.2026: seven files with different queries, drifting in both directions. Neither copy was simply the older one.

  • The chart was behind on the canonical names. acs, hikari, hazelcast-overview and jmx-dashboard still queried one producer’s spelling, so the Kubernetes dashboards were blind to exactly what the ft: names exist to fix.

  • The chart was ahead on mysql: six panels this repo lacked (InnoDB buffer pool, query throughput, full-scan and expensive-query signals, a performance_schema digest table) and the two Query Cache panels correctly deleted, MySQL 8 having removed the query cache. That version is now the one here, which also retired six entries from the coverage exceptions file.

misc/testing/scripts/sync-dashboards.sh makes this repo the source and copies into the chart; --check reports drift and exits non-zero. validate-all Test 42 runs the check when an ft-charts-library checkout is present and only ever warns — a suite that fails on someone else’s branch state is worse than the drift it catches.

There was no environment reason for the copies to differ at all. Both sides provision a Prometheus datasource with uid prometheus_uid, so the files can be byte-identical; the ${datasource} references the chart had accumulated were dangling in the eight dashboards that define no such variable.

The two that stay forked

Extended/ft-ui-services.json and system-dashboard-basic.json are skipped by the sync. Their panels do not query different spellings of one metric — they query different producers. On-prem reads host processes from process-exporter (namedprocess_namegroup_*); the chart reads containers from cAdvisor (container_*{namespace,pod}). A canonical name cannot reconcile those, because a process on a host and a container in a pod are not the same measurement.

Logs/logs-overview.json exists only in the chart: it reads Loki, which on-prem does not run.

Not taken back

jmx-dashboard has eight panels in the chart that this repo does not: Threads current and Classes Loaded (three each — value, increase, rate), plus Start time and Cpu Time. The first six duplicate panels already here on canonical names, and all eight read a single producer spelling. Importing them would reintroduce the blindness, and validate-all Test 35 would reject them.

tag 1.6.9 · commit a4c0a5e133c2cba0a8b461039e9634af33e3dd9b · page generated September 04, 2026