Monitoring Exporters and Grafana Installation Guide

Minimum per exporter node:

Prometheus/Grafana server:

Documentation:

Before starting, gather the following information:

In this example, Prometheus is installed on the same server as the DB. You need to know its IP address during setup to whitelist it on the servers where the exporters are installed:

Use the automated installation script:

The script automatically:

Expected output (versions may vary):

Manual Installation (if script fails)

Use the automated deployment script with service auto-detection:

Service Auto-Detection: The script automatically detects running services and deploys appropriate exporters:

The script automatically:

Manual Installation (click to expand)

If you prefer manual installation or need to customize the setup:

All exporters are configured to run in network_mode: host (they bind directly to host network). By default, they use these ports:

If you need to override these ports (e.g., due to conflicts or multi-instance deployment):

Three ways, in the order you should prefer them. All of them must give you the 1.6.9 branch — that is the version this page describes.

After downloading or cloning, you should have the following file structure:

The deployment automatically detects running services and deploys appropriate exporters:

For manual deployment, navigate to the relevant exporter directory:

Important: JMX ports are service-specific — do NOT mix them:

JMX Exporters (ports 5556/5557) connect to FTACS JMX on port 9999. For standalone Hazelcast, the jmx-hazelcast exporter connects to Hazelcast JMX on :9110 and serves Prometheus on :9101. Embedded-in-ACS Hazelcast uses its native :9101 endpoint (no JMX connection needed).

Manual: Enable JMX on WildFly/ACS (port 9999)

Before JMX exporters can collect metrics, WildFly/ACS (NOT Hazelcast) must be configured to expose JMX on port 9999. One-time configuration on each ACS server.

Step 1: Add to /usr/local/FTACS/bin/standalone.conf:

Step 2: Enable HikariCP MBeans in /usr/local/FTACS/standalone/configuration/ftacs_hikari.properties:

Step 3: Restart ACS:

Verify: ss -tlnp | grep 9999

Manual: Enable Prometheus on Hazelcast

Hazelcast uses native Prometheus endpoint on port 9101. The quick-deploy.sh script can configure this automatically for native installations.

Native installation (systemd/init.d):

Step 1: Add to /usr/local/hazelcast-5.0/bin/hz-start:

Step 2: Restart Hazelcast: systemctl restart hazelcast5

Hazelcast in Docker:

If Hazelcast runs in a Docker container, add the following to your docker run script:

Then restart the container.

Verify: curl -s http://localhost:9101/metrics | head

Use the automated deployment script:

Important: Before running the script, create the MySQL exporter user (see step 1 below).

Manual Installation (click to expand)

Run the following commands in the shell console on the target server:

By default, MySQL Exporter listens on port 9104. To override it (e.g., due to a conflict or multiple exporters):

Use the automated deployment script:

Note: Bootstrap and quick-deploy will prompt for Oracle credentials interactively.

Manual Installation (click to expand)

Run the following commands in the shell console on the target server:

If your database runs on a separate server (e.g., Windows server, cloud RDS, or a different network segment) where you cannot install Docker exporters, you can run the exporter on any Linux server that has network access to the database.

Step 1: On the Linux server (where exporter will run), set the host to the remote database IP:

Step 2: Generate config and start:

Step 3: Verify exporter connects to remote database:

Step 4: Add to Prometheus targets. Since the exporter runs locally, use 127.0.0.1:

When some monitored services run on the same server as Grafana/Prometheus but in a separate Docker Compose stack, those containers may be in a different Docker network. Prometheus runs in network_mode: host and cannot always reach containers in bridge networks through docker-proxy (connections hang/timeout).

Services on other servers are not affected — standard port mapping works fine over the network.

How to identify the problem:

Solution: Switch only the affected service to network_mode: host. Services on other servers keep their normal networking.

Example: Hazelcast on same server as Grafana

Before (bridge network — metrics not accessible from Prometheus):

After (host network — metrics accessible on localhost):

Important changes when switching a service to network_mode: host:

Prometheus target configuration:

For services on the same server — use 127.0.0.1:

For services on other servers — use their IP as usual:

Verify:

The bootstrap script automatically detects FT UI services when run on HC1:

The script will automatically deploy:

Native endpoints (no exporter needed):

Nginx Exporter (for Angular UI portals)

Step 1: Enable nginx stub_status in portals container

The portals nginx must expose /nginx_status endpoint. Add the following block to the nginx config inside the portals container:

How to find and edit the config:

Verify stub_status is working:

Step 2: Deploy nginx-exporter

Update:

Step 3: Start exporter

Step 4: Configure firewall

Verify:

ClickHouse metrics (native Prometheus endpoint)

ClickHouse exposes Prometheus metrics natively — no exporter container. The old f1yegor/clickhouse-exporter is not used: it crash-loops on modern ClickHouse (21.8+) because it does not sanitise block-device async metric names (clickhouse_block_discard_ops_dm-4 is not a valid Prometheus metric name).

Step 1: Enable the native endpoint on the ClickHouse host (port 9363)

This drops clickhouse-prometheus.xml into /etc/clickhouse-server/config.d/ and restarts clickhouse-server. Omit --restart to install the config and restart the DB yourself later. The endpoint binds to ClickHouse’s configured <listen_host> — if ClickHouse already answers on the server IP (HTTP 8123), 9363 binds there too.

Step 2: Configure firewall

Step 3: Add to the Prometheus server’s servers.env, then run generate-targets.sh

Verify (native metric names, not clickhouse_*):

PostgreSQL Exporter

Step 1: Create exporter user in PostgreSQL

Step 2: Let the exporter reach PostgreSQL — bind to localhost, never to all interfaces

The exporter runs with host networking, so binding to loopback is enough:

If the database must stay reachable from other hosts (published as 5432:5432 for other services), do not rely on UFW. Restrict it in the DOCKER-USER chain, which Docker does not overwrite:

Preferred alternative — publish nothing at all. Attach the exporter to the database’s own Docker network; it then talks to PostgreSQL container-to-container and no port is exposed anywhere:

This is also the only variant that works on a hardened host that drops outbound traffic to private ranges: a rule such as OUTPUT -d 172.16.0.0/12 -j DROP covers every default Docker subnet, so a host-networked exporter cannot reach the container even by IP, while a container on the same bridge is unaffected.

Step 3: Deploy postgres-exporter

Update:

Step 4: Start exporter

Step 5: Configure firewall

Verify:

These services expose Prometheus metrics natively - no separate exporter needed:

FT Device Network Service (Spring Boot Actuator)

Port: 8383 Path: /api/actuator/prometheus Type: Spring Boot Actuator

Metrics are enabled by default with micrometer-prometheus dependency.

Verify:

Add to Prometheus (servers.env):

Configure firewall:

AI Agent (Python/FastAPI)

Port: 8084 Path: /metrics Type: FastAPI with prometheus_client middleware

Metrics are exposed using prometheus_client library.

Expose the metrics port — the default ft-ui-services compose does not publish the AI Agent port to the host. Add a port mapping to the ui-ai-agent service in the ft-ui-services compose file and recreate the container:

Verify:

Add to Prometheus (servers.env):

Configure firewall:

UI Backend (Spring Boot Actuator) - Optional

Port: 8881 Path: /actuator/prometheus Type: Spring Boot Actuator

Prerequisites:

Add to Spring Security configuration:

Or update application.properties:

Verify:

Add to Prometheus (servers.env):

Configure firewall:

The FT UI Services dashboard (ft-ui-services.json) provides monitoring for all components:

Dashboard is auto-provisioned at: Dashboards → Extended → FT UI Services

Test: http://<server-ip>:9100/metrics

After Windows Exporter is running, add the server to servers.env on the Prometheus server:

Then regenerate targets:

Use the one-command bootstrap:

On the interactive path, bootstrap.sh stack opens a full-screen arrow-key configuration menu (↑↓ + Enter to navigate, Esc to go back, per-item help panel, masked passwords; nothing is written until Apply):

After Apply the script deploys the stack (containers up), auto-registers this host’s exporters in servers.env and generates targets, creates the read-only Grafana service token, and — on an existing install — changes the admin password via grafana cli if you set a new one (no current password needed). Values live in /opt/grafana/prometheus-grafana-stack/.env.

Run with -y to skip the menu entirely (non-interactive: domain = server IP, random passwords, self-signed cert).

After deployment:

Manual Installation (click to expand)

+ For a complete list of all variables, see .env.example in each directory.

+ Grafana Internal Environment Variables Reference:

+ These variables are set automatically by docker-compose based on your .env file:

+

+ . Prepare generated files (the bootstrap script creates these automatically — a manual/offline installation must create them before the first start, otherwise Docker turns the missing bind-mount files into directories and the containers fail with “Are you trying to mount a directory onto a file”):

+

Also make sure GRAFANA_DOMAIN in .env is set to the server’s domain or IP — nginx serves Grafana only for that hostname. . Start the Prometheus-Grafana stack:

+

After deployment, the monitoring stack provides two web interfaces accessible via nginx reverse proxy:

How it works:

All external traffic goes through nginx reverse proxy (ports 80/443). Nginx handles SSL, authentication, and forwards requests to internal services:

Grafana and Prometheus listen only on 127.0.0.1 (localhost) — they are not reachable directly from outside. This is why only ports 80 and 443 need to be opened in the firewall, not 3100 or 9090.

How to find your passwords:

Grafana UI pages:

Prometheus UI pages:

After the stack is running, configure which servers Prometheus should monitor.

Step 1: Edit the server inventory

Format: TYPE IP:PORT INSTANCE [EXTRA_LABELS]

EXTRA_LABELS are additional labels attached to all metrics from this target. Useful for:

Instance Naming Convention (recommended):

Instance names are flexible - you can use any naming scheme that works for your organization. The recommended format is {Environment}-{ServerType}{Number}:

Which exporters for which server type:

Minimal example (one server of each type):

Full example with labels and API endpoints

Step 2: Generate Prometheus target files

The script automatically:

Adding new servers later:

Benefits:

Supported target types (TYPE field reference)

The TYPE field in servers.env determines which Prometheus job will scrape the target. Each type maps to a specific exporter and port:

Parking a target that stopped answering. generate-targets.sh probes every row and reports the ones that did not respond. Two flags act on that list — neither ever deletes a line, because a row records an address somebody added for a reason and deleting it takes the reason with it:

A parked row looks like this, and comes back by deleting the three #:

Parking never happens on its own. A target that is merely restarting reads as unreachable too, so --park has to be typed by someone who has looked at the dry-run list. A row that answered is never touched.

Metrics paths. Most targets are scraped on /metrics, but the Spring Boot services are not, and the path is not guessable from the service name. These are the paths the generator writes into metrics_path — use them when probing a target by hand:

So a target that looks down is worth re-checking on its own path before anything else:

How IP addresses appear in Grafana dashboards

No configuration needed — prometheus.yml already includes relabel_configs that extract IP addresses from target addresses and store them in the node_ip label.

Grafana dashboards use this label to display server IPs in tables and filters.

How it works (FYI):

Enable a read-only reverse proxy for embedding Grafana dashboards in external applications (e.g., Angular UI portal).

The /grafana-ro/ endpoint requires two things configured in .env:

Step 1: Configure allowed portal origins

Set UI_PORTAL_ORIGINS in .env to the URLs of applications that will embed Grafana:

Without this setting, /grafana-ro/ returns 403 Forbidden for all external requests.

Step 2: Verify service token

Check that GRAFANA_SERVICE_TOKEN is set (not CHANGE_ME):

If it shows CHANGE_ME, generate a token — see “Manual token generation” below.

Step 3: Restart containers

Test the Nginx reverse proxy:

Open https://grafana_domain/grafana-ro/ — you should see Grafana dashboards without needing to log in, with read-only permissions.

Manual token generation (if bootstrap failed or manual installation)

Run the automated script:

The script creates a Grafana service account (ReadOnlySA) with Viewer role, generates a token, and updates .env automatically.

If the script fails, generate the token manually:

How nginx proxy works

The nginx container uses a template file (nginx.conf.template) with ${GRAFANA_SERVICE_TOKEN} placeholder. On startup, the entrypoint script:

All services run in network_mode: host. Nginx connects to Grafana via http://127.0.0.1:3100 and serves HTTPS on ports 80/443.

The /grafana-ro/ path provides read-only access without authentication, making it perfect for embedding dashboards in external applications.

How to get dashboard URL for embedding:

Example iframe code:

Available kiosk modes:

Common parameters:

Important notes:

Verify nginx proxy is working:

Grafana will only load dashboards present in the definitions folder during startup.

If later you decide to enable some dashboards:

Tip: You don’t need to restart the whole stack, only the Grafana container.

Alertmanager is configured during stack deployment:

The bootstrap script will:

Edit .env and restart:

During bootstrap, you have 3 options for Email configuration:

Testing with MailPit (fake SMTP):

During bootstrap, you have 2 options for SNMP configuration:

Testing with SNMP Trap Catcher:

SNMP Documentation:

If you selected testing options during bootstrap, these UIs are available:

When you send a test alert to Alertmanager, it routes to ALL configured channels:

To install a specific version, use the --branch parameter:

If you experience issues after downgrade or need a guaranteed clean state, perform a full reset. This involves two parts:

Symptoms: Target is DOWN in Prometheus UI, curl hangs

Step 1: Is exporter listening?

Step 2: Is container running?

Step 3: Is exporter accessible from Prometheus?

If connection hangs — check firewall allows access from Prometheus host.

Symptom: Grafana login page loads but admin password is rejected (Invalid username or password)

Cause: Grafana stores the admin password in its SQLite database on first startup. Changing GRAFANA_ADMIN_PASSWORD in .env and restarting the container does not update the existing password — it only applies when creating a new admin user.

Solution: Reset password via CLI:

Wrong: MYSQL_USER = exporter (spaces around =)

Correct: MYSQL_USER=exporter (no spaces)

Symptom: Containers start but use default/empty values

Solution: Always run cp .env.example .env before docker compose up

Problem: Rules disappear after reboot

Solution: Our scripts auto-save rules. Manual save:

Symptom: Prometheus shows targets as “DOWN”

Solution: Verify Prometheus IP and update firewall rules.

Three of them, each with a symptom that names the wrong thing: containers that cannot reach the network (a rebuilt FORWARD chain), SSH reporting No route to host on a host that is up (a connection limit on port 22), and a host exporter that cannot reach a healthy container (an OUTPUT drop covering the Docker bridge).

They are written up once, with the checks that tell them apart and the fix for each, in the Docker + UFW guide. They used to be repeated here as well; two copies of the same three faults is two copies to keep in step, and this is the one that was going to fall behind.

sudo bash /opt/grafana/misc/diagnose.sh reports all three under “Traps that present as something else”.

Two of the stack’s images — snmp-catcher and alertmanager-config-ui — are ours and live in a private project on the hub. Every other image is public and needs nothing.

The same missing credential can also arrive worded as a missing tag. On a host with no login, docker manifest inspect reports

for an image that exists — measured, and it sends people to check the version number instead of the login. docker pull on the same reference says unauthorized. misc/diagnose.sh reports either wording as a probable missing login.

On an online install this is expected the first time: log in once on that host and re-run. Bootstrap checks for it before starting the stack and stops with the same instruction, so you should see it before Docker’s own error.

On an offline install this must never happen, and if it does it is telling you the images are not on the host — nothing there should be reaching a registry at all. In order of likelihood:

An air-gapped host cannot log in to anything, so a login prompt there is always a missing-image problem, never a credentials problem.

Problem: “The container name is already in use”

Solution:

Solution:

Solution:

Symptom: cAdvisor returns only root container (id="/"), no individual Docker containers visible. System Dashboard Basic shows no FT UI Services. cAdvisor logs show:

Cause: Storage Driver: overlayfs means Docker uses the containerd image store — the default for fresh installs of Docker Engine ≥ 29 (opt-in on 28, and on Debian 13 / RHEL 9 static-tarball installs). There is no /var/lib/docker/image//layerdb in that mode, and *cAdvisor before v0.54.0 cannot resolve a container’s read-write layer: it skips every container and reports only the root cgroup, so all container_* series for that host are lost.

Upstream fixed this in cAdvisor v0.54.0 (PR #3709, released 2025-12-02) — it reads the container rootfs from the containerd spec instead of the graphdriver metadata. This repo pins a newer build, so the fix is the cAdvisor version, not the Docker storage driver.

Two conditions, not one. Because the new code talks to containerd, cAdvisor’s docker factory needs the socket dockerd actually uses. With a wrong path the container starts, the read-write layer ID spam stops and everything looks fixed, but the log says Registration of the docker container factory failed: …​ dial unix /run/containerd/containerd.sock and every container_* series arrives without name/container_label_* — only raw cgroup ids (verified on grafrd: 13 named containers with a reachable socket, 0 without). cAdvisor’s default is /run/containerd/containerd.sock, correct for package installs; the offline static-tarball install (dockerd spawns its own containerd) keeps it under /run/docker/containerd/. Hence CONTAINERD_SOCK in exporters/base/.env, detected by detect_containerd_sock() (misc/lib/container-helpers.sh) on every quick-deploy run.

Check:

Solution — upgrade cAdvisor and pin the socket. No Docker restart, no downtime for other containers:

Air-gapped host — load the image first. A ready archive is published on the hub, together with a step-by-step page for customers: https://hub.friendly-tech.com/docs/pdf/grafana-cadvisor-fix.html

Or build it yourself on any machine with internet access:

Verify:

Last resort — switch Docker back to the classic store. Only for hosts that must keep an old cAdvisor build. Graph drivers are deprecated in Docker 29, this restarts Docker, and images already written to the containerd store disappear from docker images afterwards (they are not deleted — reverting daemon.json brings them back):

On a fresh Docker 29 install this may not take effect at all — the daemon keeps the containerd store while its state exists, and a real switch means wiping /var/lib/docker. Upgrade cAdvisor instead.

Solution:

The UI runs as uid 1000 while bootstrap.sh runs as root, so the files it edits must be group-owned by the host docker group — the same group that lets it restart containers. Re-running bootstrap.sh stack sets this up (DOCKER_GID in .env, plus 0660 on .env and 0664 on the templates and prometheus_rules.yml, which other containers still have to read).

bootstrap.sh stack records two host-specific values in .env, and the UI needs both:

A stale or missing value here fails quietly: the container is Up and healthy, and only a save shows that nothing was applied. misc/diagnose.sh reports the mismatch under “Traps that present as something else”.

Check what the container actually sees:

A related symptom is “Configuration saved” with nothing changing: the write succeeded but the restart that applies it did not, because neither Alertmanager nor snmp-notifier re-reads .env while running. The UI then names the service that is still on the old settings. Applying it by hand:

If the UI keeps behaving like an older build after an upgrade, check which image tag is running. Since 1.6.9 it is pulled from our registry rather than built on the host, so an upgrade means pulling a new tag — docker compose pull alertmanager-config-ui && docker compose up -d alertmanager-config-ui (see prometheus-grafana-stack/alertmanager-config-ui/README.md).

This repository automatically maintains Antora-formatted documentation alongside the Markdown README.

Every image this stack runs exists in our own registry, scanned, at the version we ship. Not for convenience: an install that pulls from six third-party registries is only as available as the least reliable of them, and versioned tags do disappear — Bitnami deleted every one of jmx-exporter’s, which is why that exporter is pinned by digest.

Read-only account for the private project:

It is guest on both projects — it can pull and nothing else; a push is rejected. Ask for the password rather than reusing a personal account.

How the mirror stays in step with what we install. The list is never maintained by hand. misc/list-stack-images.sh computes it from the same .env.example and compose files the runtime reads, and the Mirror stack images workflow (mirror-images.yml) runs on any change to those files — so bumping a version in .env.example is what syncs it. It copies with skopeo --all, keeping multi-arch images multi-arch, and fails the run rather than leaving the mirror partial. A weekly run catches anything lost to a registry cleanup.

What redirects a deployment to the mirror is a line in .env, not the compose file. The compose files keep naming the UPSTREAM image as their default — that is what tells the mirror workflow what to copy — and .env.example carries NAME_IMAGE=hub.friendly-tech.com/mirror/… for each of them. Since .env.example is only copied when there is no .env, a host installed before the mirror existed would keep asking for upstream names that the offline bundle does not contain — an air-gapped upgrade reaching for Docker Hub, on exactly the hosts that cannot reach it. So bootstrap and quick-deploy re-apply the pins on every run, the same way they refresh DOCKER_GID and CONTAINERD_SOCK. A pin already in .env is never touched, including a commented-out one — that is how you keep a single image on upstream.

The two images we build are published by their own workflows, which fire when their sources change (prometheus-grafana-stack/snmp-catcher/ and …/alertmanager-config-ui/).

Six workflows live in .github/workflows/:

Two things about triggers are worth knowing before adding a workflow here.

schedule runs on the default branch only. GitHub takes the workflow from the latest commit on main regardless of where it was developed, so a weekly job that lives only on a version branch never fires. That is why fixture-drift.yml is on main as well as on the development line.

workflow_dispatch is not limited to the default branch — it dispatches against any ref, and the version that runs is the one on that ref. Measured 05.08.2026: gh workflow run fixture-drift.yml --ref 1.6.9 ran seven hours before the file existed on main at all.

What does trip people up is registration, which is a different thing: GitHub does not know a workflow exists until some event has caused it to read the file. A workflow whose only triggers are schedule and workflow_dispatch gives it no such event on a version branch, so the API reports it as non-existent and there is nothing to dispatch. Push the file with any trigger that fires once — a temporary push: is enough — and dispatch works from then on, on any branch.

The project uses version branches for development and releases. This describes what misc/release.sh does today.

A trunk-based model was agreed instead (DEV-1872, 23.07.2026) and is written up in docs/RELEASE-WORKFLOW.md — but the tooling has not been changed to match, so that document is the target and this section is the practice. Do not follow it expecting release.sh to behave that way.

Workflow:

Two steps in there surprise people, so they are spelled out rather than left in the script:

Version branch naming:

Using specific versions: