Metrics & Monitoring

1. Overview

FT System Metrics exports metrics in Prometheus format via the Spring Boot Actuator endpoint /actuator/prometheus.

Metrics collection is triggered by PrometheusMetricsFilter on each scrape request from Prometheus. Three specialized collectors aggregate data from the database and register Gauge/Counter metrics in the Micrometer Registry.

1.1. Endpoint

GET /actuator/prometheus

Returns all metrics in Prometheus text format. Available Actuator endpoints:

Endpoint Description

/actuator/health

Application status (including database connection state)

/actuator/info

Application information

/actuator/prometheus

All metrics in Prometheus format

/actuator/metrics

List of available metrics (JSON)

/actuator/metrics/{name}

Details for a specific metric (JSON)

2. Metrics Collectors

2.1. Collection Trigger

PrometheusMetricsFilter (extends OncePerRequestFilter) intercepts every HTTP request. If the URI equals /actuator/prometheus, the filter sequentially invokes three collectors:

if (request.getRequestURI().equals("/actuator/prometheus")) {
    cpeMetricsCollector.collect();
    cpeTaskMetricsCollector.collectTaskMetrics();
    cpeEventCollector.collectEventMetrics();
}

2.2. Gauge Registration Pattern

All collectors use a ConcurrentHashMap to store Gauge metric values. On each collection cycle the map is cleared and populated with fresh values:

private final Map<String, Number> gaugeValues = new ConcurrentHashMap<>();

// Registration pattern
String gaugeId = String.format("metric_%s_%s", tenantId, dimension);
gaugeValues.put(gaugeId, value);
Gauge.builder("metric.name", gaugeValues, map -> map.get(gaugeId).doubleValue())
    .tags("tenant", tenantName, "dimension", dimensionValue)
    .description("Metric description")
    .register(registry);

2.3. Tenant Tagging

All metrics are tagged with the tenant (ISP) name. IspService maps ISP IDs to names from the database:

Map<Integer, String> ispMap = ispService.getIspMap(projections);
// Tags: tenant=<isp_name>, protocol=TR069, status=online, ...

3. Device Metrics (CpeMetricsCollector)

3.1. cpe.devices.total

Total number of registered devices broken down by protocol and tenant.

Property Value

Type

Gauge

Tags

protocol (TR069, LWM2M, MQTT, USP, …​), tenant (ISP name)

Source

cpe table, GROUP BY protocol_id, location_id

Update

On each Prometheus scrape

Example output:

cpe_devices_total{application="ft-system-metrics",protocol="TR069",tenant="Default"} 15234.0
cpe_devices_total{application="ft-system-metrics",protocol="LWM2M",tenant="ISP-A"} 892.0

3.2. cpe.devices.online_status

Number of online and offline devices broken down by protocol and tenant.

Property Value

Type

Gauge

Tags

protocol, status (online / offline), tenant

Source

Non-TR069: is_online=1 field in the cpe table. TR069: record in cpe_next_session_time with check_time >= now

Update

On each Prometheus scrape

Online detection logic:

  • TR069 — a device is online if cpe_next_session_time.check_time >= LocalDateTime.now() + 10s

  • All other protocols — a device is online if cpe.is_online = 1

  • Offline count = total - online

Example output:

cpe_devices_online_status{protocol="TR069",status="online",tenant="Default"} 12100.0
cpe_devices_online_status{protocol="TR069",status="offline",tenant="Default"} 3134.0

3.3. tenant_list

A marker counter that enumerates all tenants in the system.

Property Value

Type

Counter

Tags

tenant (ISP name)

Source

Unique ISPs from device query results

4. Task Metrics (CpeTaskMetricsCollector)

4.1. cpe.tasks.status

Task distribution by status for the current and previous day.

Property Value

Type

Gauge

Tags

day (YYYY-MM-DD), status (Completed / Pending / Failed / Rejected / Sent), tenant

Source

UNION ALL across cpe_completed_task, cpe_failed_task, cpe_rejected_task, cpe_pending_task

Time window

From the start of the previous day (TimeUtil.getPreviousDay()) to the current moment

Update

On each Prometheus scrape

Task status definitions:

Status Source

Completed

cpe_completed_task

Failed

cpe_failed_task

Rejected

cpe_rejected_task

Pending

cpe_pending_task WHERE repeats = 0

Sent

cpe_pending_task WHERE repeats > 0

Example output:

cpe_tasks_status{day="2026-02-17",status="Completed",tenant="Default"} 4521.0
cpe_tasks_status{day="2026-02-17",status="Failed",tenant="Default"} 23.0
cpe_tasks_status{day="2026-02-18",status="Pending",tenant="Default"} 156.0

4.2. cpe.tasks.type

Task distribution by type for the last 24 hours grouped by hour.

Property Value

Type

Gauge

Tags

hour (YYYY-MM-DD HH:00:00), type (task type display name), tenant

Source

UNION ALL across all four task tables

Time window

From the start of the previous hour (TimeUtil.getPreviousHourStart()) to the current moment

Update

On each Prometheus scrape

Task types (subset):

ID Name ID Name

0

GetRPCMethods

2

Reboot

4

SetParameterValues

5

GetParameterNames

9

FactoryReset

10

ScheduleInform

29

Download

36

Upload

37

CustomRPC

…​

…​

Example output:

cpe_tasks_type{hour="2026-02-18 14:00:00",type="SetParameterValues",tenant="Default"} 89.0
cpe_tasks_type{hour="2026-02-18 14:00:00",type="Reboot",tenant="ISP-A"} 12.0

5. Event Metrics (CpeEventCollector)

Event metrics are collected only during the first minute of each hour (TimeUtil.isFirstMinuteOfHour()). At all other times collectEventMetrics() returns without executing any queries.

5.1. cpe.events.type

CPE event distribution by type for the previous hour.

Property Value

Type

Gauge

Tags

hour (YYYY-MM-DD HH:00:00), type (event name), tenant

Source

cpe_log JOIN cpe_log_event_name, GROUP BY event name and hour

Time window

Previous hour

Update

Once per hour (first minute)

Example output:

cpe_events_type{hour="2026-02-18 14:00:00",type="0 BOOTSTRAP",tenant="Default"} 342.0
cpe_events_type{hour="2026-02-18 14:00:00",type="1 BOOT",tenant="Default"} 1205.0

5.2. cpe.events.total.last_hour

Total number of CPE events for the previous hour by tenant.

Property Value

Type

Gauge

Tags

hour (HH format), tenant

Source

cpe_log COUNT(*) GROUP BY location_id

Time window

Previous hour

Update

Once per hour (first minute)

Example output:

cpe_events_total_last_hour{hour="14",tenant="Default"} 8923.0

6. Metrics Reference

6.1. Complete Metrics Table

Metric Name Type Labels Description

cpe.devices.total

Gauge

protocol, tenant

Total registered devices by protocol and tenant

cpe.devices.online_status

Gauge

protocol, status, tenant

Online/offline device count

tenant_list

Counter

tenant

All tenants in the system

cpe.tasks.status

Gauge

day, status, tenant

Task count by status and date

cpe.tasks.type

Gauge

hour, type, tenant

Task count by type and hour

cpe.events.type

Gauge

hour, type, tenant

Event count by type and hour

cpe.events.total.last_hour

Gauge

hour, tenant

Total event count in the previous hour

6.2. Standard Micrometer Metrics

In addition to custom metrics, the application automatically exports standard Micrometer metrics:

  • http.server.requests — HTTP requests with histogram and SLA buckets

  • jvm.* — JVM metrics (memory, threads, GC)

  • process.* — process metrics (CPU, uptime)

  • system.* — system metrics (CPU load)

  • hikaricp.* — connection pool metrics

  • spring.data.repository.* — Spring Data metrics

7. Prometheus Integration

7.1. Scrape Configuration

Prometheus is configured via the prometheus.yml file:

global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'spring-boot-app'
    metrics_path: '/actuator/prometheus'
    static_configs:
      - targets: ['ft-system-metrics:8080']

7.2. Useful PromQL Queries

Total devices by tenant:

sum by (tenant) (cpe_devices_total)

Online ratio by protocol:

cpe_devices_online_status{status="online"}
  /
cpe_devices_total

Task failure rate by tenant (today):

cpe_tasks_status{status="Failed", day="2026-02-18"}
  /
(cpe_tasks_status{status="Completed", day="2026-02-18"} + cpe_tasks_status{status="Failed", day="2026-02-18"})

Events per hour trend:

cpe_events_total_last_hour

8. Grafana Dashboards

8.1. Data Source Setup

  1. Open Grafana: http://localhost:3000

  2. Log in: admin / admin (or the GRAFANA_PASSWORD value)

  3. Configuration → Data Sources → Add data source

  4. Select Prometheus

  5. URL: http://prometheus:9090 (Docker Compose service name)

  6. Click Save & Test

8.2. Recommended Dashboard Panels

Panel Type PromQL

Total Devices

Stat

sum(cpe_devices_total)

Devices by Protocol

Pie Chart

sum by (protocol) (cpe_devices_total)

Online vs Offline

Bar Gauge

sum by (status) (cpe_devices_online_status)

Online Ratio

Gauge (%)

sum(cpe_devices_online_status{status="online"}) / sum(cpe_devices_total) * 100

Task Status Distribution

Stacked Bar

cpe_tasks_status

Task Types (Last Hour)

Table

cpe_tasks_type

Events per Hour

Time Series

cpe_events_total_last_hour

Event Type Distribution

Pie Chart

sum by (type) (cpe_events_type)

On first launch, Grafana data will appear after the first successful scrape from Prometheus (approximately 15 seconds after the application starts).