JVM and connection-pool metric names: what each producer emits

Version 1.7.1 | Updated: October 06, 2026

JVM and connection-pool metric names: what each producer emits

Everything here was measured on 05.08.2026 against running systems; nothing is quoted from Micrometer or JMX exporter documentation. The hosts are named by role rather than by address: this page is published, and the numbers are what matters, not which machine produced them.

Where the numbers came from

  • JMX, WildFly/ACS 6.4 — an ACS application server, exporters :5556 (Hikari) and :5557 (JVM). 110 distinct names.

  • JMX, Hazelcast — same host, :9101, exporter_type=hazelcast. 208 java_lang_* lines plus 7176 com_hazelcast_*.

  • Micrometer, ACS 7.1.1 — an ACS 7.1 test container, :8080/actuator/prometheus. 98 names.

  • Micrometer, ACS 6.5.2 — the acs OKD cluster, via ft-prometheus-acs, job="ftacs". 167 series names.

6.5 and 7.1 are the same producer. The two sets are identical: 48 shared JVM/pool names, zero on either side alone. Two capture artefacts had to be removed before that was visible, and both are worth knowing if these numbers are ever re-taken:

  • # HELP names a summary once; /api/v1/series lists its _count/_sum/_bucket expansions. Comparing one against the other invents differences. Normalise those three suffixes.

  • _max must not be normalised away. In Micrometer it is a metric in its own right — hikaricp_connections_max is what the pool-exhaustion alert queries, and stripping it made this table briefly claim the metric does not exist, when it has 16 series in the cluster and 10 on-prem.

Three producer spellings, not two

This is the part that matters, and it is not what the plan assumed.

  1. JMX exporter, lowercase — java_lang_runtime_uptime, java_lang_classloading_loadedclasscount, java_lang_threading_threadcount. Emitted by the WildFly/ACS JVM exporter (:5557, exporter_type=jvm) on both production ACS hosts. 208 lines, 0 CamelCase.

  2. JMX exporter, CamelCase — java_lang_Runtime_Uptime, java_lang_Memory_HeapMemoryUsage_used. Emitted by the Hazelcast exporter (:9101, exporter_type=hazelcast). 208 lines, 0 lowercase.

  3. Micrometer — process_uptime_seconds, jvm_memory_used_bytes, hikaricp_connections_active. ACS 6.5+, every Spring Boot service, and Kubernetes.

The two JMX spellings are the same exporter with different config (lowercaseOutputName), and both are present in production Prometheus at the same time under the same instance label, separated only by exporter_type.

What this already costs us, measured

  • jmx-dashboard is empty for the JVM it is named after. Its expressions query CamelCase and are scoped by exporter_type=~"$app". On production: count(java_lang_Runtime_Uptime{exporter_type="jvm"}) = 0; count(java_lang_Runtime_Uptime{exporter_type="hazelcast"}) = 2; count(java_lang_runtime_uptime{exporter_type="jvm"}) = 2. So the dashboard shows something only when pointed at Hazelcast. The WildFly JVM data has been collected all along and read by nothing. This matches the note already in the test-monitoring skill — “the lowercase half has never been read by anything” — and is a defect in its own right, independent of this plan.

  • The ACS pool alert is blind on-prem and works in Kubernetes. hikaricp_connections_active (Micrometer) — on-prem 10 series, none of them ACS; in the cluster it covers ftacs itself (4 series). hikaricp_active_connections (JMX) — on-prem 6 series, all ACS; in the cluster 0. The alert queries the Micrometer spelling.

  • hikari.json is entirely empty in Kubernetes. Every expression in it queries the JMX spelling, which no producer there emits.

Canonical names

ft: prefix. One entry per concept: the canonical name, then the expression per producer. — means the producer does not carry it and the row is served by the other side (or by node-exporter, noted explicitly).

  • ft:jvm_uptime_seconds

    • jmx/lowercase: java_lang_runtime_uptime / 1000

    • jmx/CamelCase: java_lang_Runtime_Uptime / 1000

    • micrometer: process_uptime_seconds

  • ft:jvm_heap_used_bytes

    • jmx/lowercase: java_lang_memory_heapmemoryusage_used

    • jmx/CamelCase: java_lang_Memory_HeapMemoryUsage_used

    • micrometer: jvm_memory_used_bytes{area="heap"}

  • ft:jvm_heap_max_bytes

    • jmx/lowercase: java_lang_memory_heapmemoryusage_max

    • jmx/CamelCase: java_lang_Memory_HeapMemoryUsage_max

    • micrometer: jvm_memory_max_bytes{area="heap"}

  • ft:jvm_nonheap_used_bytes

    • jmx: java_lang_memory_nonheapmemoryusage_used / java_lang_Memory_NonHeapMemoryUsage_used

    • micrometer: jvm_memory_used_bytes{area="nonheap"}

  • ft:jvm_threads_live

    • jmx: java_lang_threading_threadcount / java_lang_Threading_ThreadCount

    • micrometer: jvm_threads_live_threads

  • ft:jvm_threads_daemon

    • jmx: java_lang_threading_daemonthreadcount / java_lang_Threading_DaemonThreadCount

    • micrometer: jvm_threads_daemon_threads

  • ft:jvm_threads_peak

    • jmx: java_lang_threading_peakthreadcount / java_lang_Threading_PeakThreadCount

    • micrometer: jvm_threads_peak_threads

  • ft:jvm_classes_loaded

    • jmx: java_lang_classloading_loadedclasscount / java_lang_ClassLoading_LoadedClassCount

    • micrometer: jvm_classes_loaded_classes

  • ft:jvm_gc_time_seconds_total

    • jmx: java_lang_garbagecollector_collectiontime / 1000 (both spellings)

    • micrometer: jvm_gc_pause_seconds_sum

  • ft:jvm_gc_count_total

    • jmx: java_lang_garbagecollector_collectioncount

    • micrometer: jvm_gc_pause_seconds_count

  • ft:jvm_cpu_process_ratio

    • jmx: java_lang_operatingsystem_processcpuload / java_lang_OperatingSystem_ProcessCpuLoad

    • micrometer: process_cpu_usage

  • ft:jvm_open_file_descriptors

    • jmx: java_lang_operatingsystem_openfiledescriptorcount

    • micrometer: process_files_open_files

  • ft:pool_connections_active

    • jmx: hikaricp_active_connections

    • micrometer: hikaricp_connections_active

  • ft:pool_connections_idle

    • jmx: hikaricp_idle_connections

    • micrometer: hikaricp_connections_idle

  • ft:pool_connections_total

    • jmx: hikaricp_total_connections

    • micrometer: hikaricp_connections

  • ft:pool_connections_max

    • jmx: hikaricp_maximum_pool_size

    • micrometer: hikaricp_connections_max

  • ft:pool_connections_min

    • jmx: hikaricp_minimum_idle

    • micrometer: hikaricp_connections_min

  • ft:pool_threads_awaiting

    • jmx: hikaricp_threads_awaiting_connection

    • micrometer: hikaricp_connections_pending

  • ft:jvm_threads_started_total

    • jmx: java_lang_threading_totalstartedthreadcount / java_lang_Threading_TotalStartedThreadCount

    • micrometer: jvm_threads_started_threads_total

  • ft:jvm_classes_unloaded_total

    • jmx: java_lang_classloading_unloadedclasscount / java_lang_ClassLoading_UnloadedClassCount

    • micrometer: jvm_classes_unloaded_classes_total

Deliberately not mapped

These are queried by jmx-dashboard today and get no canonical name. Each is a host-level reading that the JVM happens to report; node-exporter already scrapes the same host and is both more accurate and unaffected by which JVM is running.

  • java_lang_OperatingSystem_TotalPhysicalMemorySize → node_memory_MemTotal_bytes

  • java_lang_OperatingSystem_FreePhysicalMemorySize → node_memory_MemAvailable_bytes

  • java_lang_OperatingSystem_TotalSwapSpaceSize → node_memory_SwapTotal_bytes

  • java_lang_OperatingSystem_FreeSwapSpaceSize → node_memory_SwapFree_bytes

  • java_lang_OperatingSystem_CommittedVirtualMemorySize → drop (JVM address space, not useful next to the RSS that node-exporter and cAdvisor already show)

  • java_lang_OperatingSystem_SystemCpuLoad → node_cpu_seconds_total based panel

  • java_lang_OperatingSystem_AvailableProcessors → count(node_cpu_seconds_total) by (instance)

  • java_lang_OperatingSystem_ProcessCpuTime → covered by ft:jvm_cpu_process_ratio

  • java_lang_Runtime_StartTime → covered by ft:jvm_uptime_seconds

  • java_lang_Threading_CurrentThreadCpuTime → drop (per-thread sampling, no consumer)

  • java_lang_Threading_TotalStartedThreadCount → ft:jvm_threads_started_total (was “drop, no panel reads it meaningfully” — acs.json and jmx-dashboard both read it, and on a Micrometer-only installation both were blind)

  • jvm_threads_deadlocked, jvm_threads_deadlocked_monitor → these come from the JMX exporter’s own JVM collector, which is not enabled on the production exporters (0 jvm_* names in the 110 captured). They have no producer today on either side; drop the panels rather than pretend.

Consequence for the plan

Task 2 must emit each canonical name from three groups, not two — lowercase JMX, CamelCase JMX, Micrometer. The alternative (normalising the two JMX spellings at the exporter by fixing lowercaseOutputName on one side) changes a producer on a production host and would silently break whatever currently reads the CamelCase Hazelcast series, so it is not part of this plan. It is worth a separate ticket.

Left on producer names on purpose (task 4b) — mostly closed, 19.09.2026

The deferral below said per-pool canonicalisation was “not free and no consumer has asked for it”. A consumer asked: measured 19.09.2026 on the OpenShift acs namespace and on grafrd, these panels are blind on every installation whose ACS is a container.

  • jvm_memory_pool_bytes_used / max / _committed → ft:jvm_memory_used_bytes_by_pool, ft:jvm_memory_max_bytes_by_pool, ft:jvm_memory_committed_bytes_by_pool. Three producers, three spellings of the pool label, normalised onto pool: java-client pool=, WildFly JMX java_lang_memorypool_usage*{name=}, Micrometer jvm_memory_*_bytes{id=}.

  • jvm_memory_bytes_committed → ft:jvm_memory_committed_bytes_by_area, the same shape as the used-memory name beside it.

  • java_lang_ClassLoading_UnloadedClassCount, java_lang_Threading_TotalStartedThreadCount → canonical, see the list above.

Still on producer names, and still deliberately:

  • jvm_info — a label-only series driving the JDK panel. Both the JMX exporter and Micrometer emit it with the same vendor/runtime/version labels (8 series in Kubernetes, measured), so label_join already works on both and a canonical name would add nothing.

  • java_lang_OperatingSystem_CommittedVirtualMemorySize — the “Physical memory” panel. Micrometer publishes no equivalent in any spelling, so the panel is JMX-only and now says so in its description.

What made this worth doing, stated once: the canonical rules are producer-agnostic, but the dashboards were not. acs.json selected instances with label_values(ft:jvm_heap_used_bytes{job="JmxExporter"}, instance) and filtered every panel by the same job, and jmx-dashboard built its job list from jvm_info{job="JmxExporter"} — so on the OpenShift cluster, where the ACS publishes Micrometer under job="ftacs", the ACS dashboard offered 24 instances and not one of them was an ACS. Measured through that cluster’s own Grafana on 19.09.2026, with the data present all along: count(ft:jvm_heap_used_bytes{job="ftacs"}) = 2.

jmx-dashboard: the panels that never had data (DEV-3042, 02.10.2026)

Once the job list stopped excluding Micrometer, the panels that had always been empty became visible. None of them lost data; their queries and the rules behind them were the same in 1.6.9. Measured with every panel run through Grafana’s /api/ds/query, before and after:

  • grafrd, ACS 6.4 (JMX exporter): 8 empty panels → 0.

  • grafrd, Spring services: 9 → 1 (Process Memory).

  • acs (OpenShift), ftacs and Spring services: 20 → 2 (Process Memory; SWAP Utilization, because the nodes have no swap).

  • acs, Hazelcast: 21 → 5 (JDK, Process Memory, JVM Memory committed, Memory pool, which its exporter does not publish on Kubernetes; and SWAP Utilization).

The causes, and what each panel reads now:

  • Host memory and CPU (Total RAM, Total SWAP, Memory and SWAP Utilization, CPU Cores, both System Load panels, and the host lines of Physical memory) asked for node_memory_*{job="$job"} and node_cpu_seconds_total{instance="$server"}. The first exists only under job="node-exporter", and on Kubernetes the second never matches, because $server is a pod address. They now read node-exporter on the JVM’s host: the one with the same instance (bare metal, Deployment Manager), or on Kubernetes the pod’s node through kube_pod_info. This is the same join the ACS dashboard uses.

  • JVM Memory Used [heap]/[nonheap] filtered ft:jvm_heap_used_bytes on an area label that no producer sets on that name. It now reads ft:jvm_memory_used_bytes_by_area. Where an exporter has no per-area breakdown (Hazelcast on Kubernetes), it falls back to ft:jvm_heap_used_bytes / ft:jvm_nonheap_used_bytes. $memarea is a fixed list (heap, nonheap), and the Max line is drawn for heap only.

  • JVM Memory Usage is now one panel, JVM Heap Usage: heap used as a share of max heap. Non-heap has no meaningful maximum (Micrometer reports -1).

  • Open File Descriptors read the JMX exporter’s process_open_fds. It now reads ft:jvm_open_file_descriptors, which all three producers feed.

  • Every rate/increase panel used [$interval] or [$rate_interval]. The ft: series are recording rules evaluated every 60 s, so on a 30 s scrape those windows held one sample and drew nothing. Every such panel now uses a fixed [5m], as the ACS dashboard already does.

  • JVM Threading Threads no longer plots Started threads. Since the panel moved to the canonical name, that cumulative counter (1.46 M on an ftacs pod) is present for every producer, and it flattened Current/Daemon/Peak (~1.6 K) onto the floor. Started stays in the rate panel.

  • Process Memory stays JMX-only and says so: Micrometer publishes no resident or virtual memory.

Labelled canonical names (task 30, closed)

Two dashboards needed a label the aggregate rules deliberately drop, so the set gained three names that keep exactly one label each and nothing more:

  • ft:jvm_gc_time_seconds_by_collector / ft:jvm_gc_count_by_collector — keep the collector. Every producer spells it differently, measured on production: WildFly puts it in name, the Hazelcast exporter and Micrometer in gc, and Micrometer adds action and cause which multiply the series. All three are normalised onto gc.

  • ft:jvm_memory_used_bytes_by_area — keeps area. Micrometer and the Hazelcast exporter carry that label; the WildFly JMX side has no such label at all, since heap and nonheap are separate metric names there, so it is synthesised.

The original reasoning, kept because it is why these names are narrow:

  • hazelcast-overview asks for GC per collector — jvm_gc_collection_seconds_* with gc="G1 Old Generation". Both canonical GC rules aggregate that away (sum without (action, cause, gc) on the Micrometer side), so the selector would match nothing. Needs ft:jvm_gc_time_seconds_by_collector or the panels rewritten to the aggregate.

  • ft-ui-services asks for memory by area — sum by (area) (jvm_memory_used_bytes{…​}). The canonical names split heap from nonheap by NAME rather than by label, so there is no area to group on. Needs either a labelled canonical name or two explicit series per panel.

Migrated in this pass: acs and system-dashboard-basic.

acs also selects the ACS JVM and nothing else, and that is the substantive change. Its panels queried CamelCase java_lang_* filtered only by job="JmxExporter", which on most hosts is the Hazelcast exporter: the ACS dashboard showed Hazelcast’s JVM. The canonical series span every producer — the ACS, Hazelcast and every Spring service — so the dashboard now requires the ACS job (ftacs, acs-metrics or JmxExporter) with exporter_type jvm, acs-metrics or none (Kubernetes sets none), and leaves out the Hazelcast pods. CamelCase is not Hazelcast’s alone: some 6.4 hosts’ ACS exporter emits it too, which is why ft:jvm_gc_*by_collector has a CamelCase source guarded by unless against Hazelcast’s jvm_gc_collection_seconds*.

The other half of the contract: consumers (task 27, closed)

Everything above is the producer side — what the exporters emit. The mapping is only worth something if the queries actually name those metrics, and until this pass nothing checked that they did.

misc/testing/fixtures/metric-inventory.txt is the producer side written down: 10322 metric names captured 05.08.2026 from the live Prometheus of all three environments, each prefixed by which of them emits it. Regenerate it with misc/testing/scripts/capture-inventory.sh — never by hand, because a name added to silence a finding removes the only reason the check means anything.

misc/testing/scripts/expr-coverage.sh extracts the metric names from every dashboard expr and every alert expr (1127 expressions, including the Helm chart’s rules when that repository is checked out alongside) and reports any name with no producer. validate-all Test 40 runs it.

What it found

Five live queries named metrics that do not exist anywhere:

  • process_resident_memory_max_bytes, in nodes-monitoring and system-dashboard. Producers emit process_resident_memory_bytes and process_virtual_memory_max_bytes; the crossed name has never existed. Both panels also carried a duplicated target pair, and the broken query was the mangled half of one — the working copy sat right beside it, so the panel drew a line and looked fine. The duplicate pair is removed.

  • windows_cs_physical_memory_bytes, windows_os_physical_memory_free_bytes and windows_os_virtual_memory_bytes. windows_exporter dropped the cs and os collector prefixes in 0.20; the monitored Windows host emits windows_memory_physical_total_bytes, windows_memory_physical_free_bytes and windows_memory_commit_limit. Verified against on-prem: each replacement returns 1 series, each old name returns 0.

And two alert families that could never fire, both left in place because fixing them is a separate change:

  • hz_cluster_size — no producer, and no recording rule defines it either. Both prometheus-grafana-stack/prometheus/hazelcast_rules.yml and the chart’s hazelcast.yml alert on it.

  • probe_ssl_earliest_cert_expiry — the blackbox exporter is not deployed in any environment, so the certificate-expiry alerts in prometheus_rules.yml and the chart’s platform.yml are decorative.

Why Hazelcast could not be fixed by renaming a query (task 32, closed)

Hazelcast embeds its randomly generated instance name in the metric name rather than a label. The same three metrics appear under six different names across the three environments — com_hazelcast_thirsty_margulis_memberCount, com_hazelcast_laughing_lamport_memberCount, com_hazelcast_compassionate_wilbur_memberCount and so on — and the name changes on every restart, so no query could be written that survives one.

The canonical_hazelcast group in canonical_rules.yml extracts the instance from name into an hz_instance label:

  • ft:hazelcast_cluster_members

  • ft:hazelcast_running

  • ft:hazelcast_cluster_safe

  • ft:hazelcast_local_member_safe

  • ft:hazelcast_executor_queue_size

Two details in those rules are load-bearing. The type matcher is what keeps com_hazelcast_Metrics_queueSize out — an unrelated per-executor metric that matches the same name regex, with 23 series on the QA stand and hundreds in Kubernetes, every one of which would arrive labelled hz_instance="Metrics". And the instance is extracted from the metric name rather than read from the name label, because name means the instance on HazelcastInstance and HazelcastInstance.PartitionServiceMBean but the EXECUTOR on HazelcastInstance.ManagedExecutorService.

All four Hazelcast alerts were inert before this, each differently, and each convincing to read:

  • HazelcastClusterSizeChanged read hz_cluster_size, which nothing defines.

  • HazelcastNodeDown used count(…​) < 1. On an empty selector count() returns no samples rather than 0, so the comparison filtered nothing in exactly the situation the alert existed for. It now uses absent().

  • HazelcastHighPendingOps read com_hazelcast_Metrics_completedPacketCount, which counts completed packets rather than pending ones and is flat at zero on production (98 series, max 0 over seven days). Had it ever moved, being cumulative, it would have fired permanently. It now reads the managed-executor queue depth. The chart had already spotted the first problem and replaced the metric with com_hazelcast_Metrics_operationExecutorQueueSize, which no producer emits either.

  • isClusterSafe — the reason this stack scrapes JMX over the native endpoint — was present on every instance and readable by nothing. HazelcastClusterUnsafe now alerts on it.

hazelcast_canonical_test.yml and hazelcast_alert_test.yml cover both halves, feeding the instance names measured in each environment. The chart’s hazelcast.yml is now byte-identical to hazelcast_rules.yml and the same tests pass against it, which matters because the two copies had already drifted into different broken states.

The remaining 24 absences

misc/testing/fixtures/metric-inventory-exceptions.txt lists every name a query asks for that no producer emits, each with the reason. The largest group is mysql.json: eleven query-cache and InnoDB-memory counters that MySQL 8 removed outright, inherited from the upstream Percona dashboard written for MySQL 5.x. Those panels are permanently empty on every database we monitor.

One dashboard set, two destinations (task 31, closed)

The dashboards shipped twice — from this repo for on-prem, and from ft-charts-library/charts/ft-monitoring/dashboards for Kubernetes — and nothing kept the two in step. Measured 05.08.2026: seven files with different queries, drifting in both directions. Neither copy was simply the older one.

  • The chart was behind on the canonical names. acs, hikari, hazelcast-overview and jmx-dashboard still queried one producer’s spelling, so the Kubernetes dashboards were blind to exactly what the ft: names exist to fix.

  • The chart was ahead on mysql: six panels this repo lacked (InnoDB buffer pool, query throughput, full-scan and expensive-query signals, a performance_schema digest table) and the two Query Cache panels correctly deleted, MySQL 8 having removed the query cache. That version is now the one here, which also retired six entries from the coverage exceptions file.

misc/testing/scripts/sync-dashboards.sh makes this repo the source and copies into the chart; --check reports drift and exits non-zero. validate-all Test 42 runs the check when an ft-charts-library checkout is present and only ever warns — a suite that fails on someone else’s branch state is worse than the drift it catches.

There was no environment reason for the copies to differ at all. Both sides provision a Prometheus datasource with uid prometheus_uid, so the files can be byte-identical; the ${datasource} references the chart had accumulated were dangling in the eight dashboards that define no such variable.

The two that stay forked

Extended/ft-ui-services.json and system-dashboard-basic.json are skipped by the sync. Their panels do not query different spellings of one metric — they query different producers. On-prem reads host processes from process-exporter (namedprocess_namegroup_*); the chart reads containers from cAdvisor (container_*{namespace,pod}). A canonical name cannot reconcile those, because a process on a host and a container in a pod are not the same measurement.

Logs/logs-overview.json exists only in the chart: it reads Loki, which on-prem does not run.

Not taken back

jmx-dashboard has eight panels in the chart that this repo does not: Threads current and Classes Loaded (three each — value, increase, rate), plus Start time and Cpu Time. The first six duplicate panels already here on canonical names, and all eight read a single producer spelling. Importing them would reintroduce the blindness, and validate-all Test 35 would reject them.

branch release/1.7.1 · commit e70021b6498caa7f94c439e9d43703e4496f41e2 · page generated October 06, 2026