Monitoring Exporters and Grafana Installation Guide
- Monitoring Exporters and Grafana Installation Guide
- Table of Contents
- Introduction
- Quick Start
- 1. Install Docker and Docker Compose on All Linux Servers
- 2. Install Node Exporter and Process Exporter
- 3. Install JMX Exporter (WildFly/ACS Servers)
- 4. Install MySQL Exporter
- 5. Install Oracle DB Exporter
- 6. Install Windows Exporter
- 7. Configure and Deploy Prometheus with Grafana
- 8. Managing Grafana Dashboards Provisioning
- Troubleshooting
- Operations
- For Developers
Version 1.6.5 | Updated: March 10, 2026
Monitoring Exporters and Grafana Installation Guide
Introduction
This guide provides step-by-step instructions for setting up monitoring exporters and Grafana for observability across various servers. The setup includes Node Exporter, Process Exporter, cAdvisor, MySQL Exporter, Oracle DB Exporter, Windows Exporter, JMX Exporter for WildFly/ACS, and a Prometheus-Grafana stack deployment. Hazelcast uses its native Prometheus endpoint (no separate exporter needed). Following this guide will ensure proper monitoring and visualization of system metrics, database performance, and application health.
System Requirements
Minimum per exporter node:
-
OS: Ubuntu 20.04+, Debian 11+, CentOS 7+, Rocky Linux 8+
-
RAM: 2GB (4GB recommended)
-
Disk: 10GB free space
-
Network: Outbound internet access (for Docker image downloads)
Prometheus/Grafana server:
-
CPU: 4 vCPU
-
RAM: 16GB
-
Disk: 100GB+ (see
sizing.mdfor capacity planning)
Documentation:
-
Architecture Overview - Diagrams, server types, and port reference
-
Sizing Guide - Capacity planning for Prometheus/Grafana
Quick Start
ONE-COMMAND INSTALL (Recommended):
For monitored servers (exporters):
# Auto-detect services and deploy appropriate exporters:
curl -fsSL https://hub.friendly-tech.com/bootstrap/install.sh | sudo bash -s -- PROMETHEUS_IP
# Example:
curl -fsSL https://hub.friendly-tech.com/bootstrap/install.sh | sudo bash -s -- 65.108.67.167
# With custom instance prefix (default: DevOps):
curl -fsSL https://hub.friendly-tech.com/bootstrap/install.sh | sudo bash -s -- 65.108.67.167 --prefix Prod
For Prometheus/Grafana server (monitoring stack):
# Deploy Prometheus + Grafana + Nginx proxy with interactive configuration:
curl -fsSL https://hub.friendly-tech.com/bootstrap/install.sh | sudo bash -s -- stack
Bootstrap options:
| Option | Description | Default | Example |
|---|---|---|---|
|
Instance name prefix for servers.env output |
|
|
|
GitHub token for private repository access |
- |
|
|
Use specific version branch or tag |
|
|
|
Show all available versions (tags + branches) |
- |
|
|
Rollback deployment changes |
latest |
|
|
List available rollback manifests |
- |
|
|
Show bootstrap version |
- |
|
|
Show documentation links |
- |
|
Examples:
# List all available versions (releases and dev branches)
curl -fsSL https://hub.friendly-tech.com/bootstrap/install.sh | bash -s -- --version-list
# Show documentation links
curl -fsSL https://hub.friendly-tech.com/bootstrap/install.sh | bash -s -- --docs
# Install specific version
curl -fsSL https://hub.friendly-tech.com/bootstrap/install.sh | sudo bash -s -- stack --branch 1.6.4
The script will interactively prompt for all configuration parameters:
-
Server IP (auto-detected)
-
Prometheus/Grafana/Nginx ports
-
Grafana admin credentials
-
SMTP settings for alerts
-
And all other
.envparameters
Press Enter to keep current values, or type new values to change them.
What bootstrap does (exporters):
-
Installs Docker and Docker Compose (if not present)
-
Clones this repository to
/opt/grafana(or updates existing) -
Stashes local changes before update (with restore instructions)
-
Auto-detects running services (Hazelcast, WildFly, MySQL, Oracle, etc.)
-
Shows deployment summary with 20-second countdown
-
Prompts for database credentials interactively (MySQL/Oracle)
-
Deploys appropriate exporters based on detected services
-
Configures firewall rules for Prometheus access
Auto-detected services:
| Service | Exporter Directory | Ports |
|---|---|---|
WildFly/FTACS |
|
5556, 5557 |
Hazelcast |
(native prometheus) |
9101 |
MySQL/MariaDB |
|
9104 |
Oracle DB |
|
9161 |
(always) |
|
9100, 9256, 9183 |
Base exporters include:
-
node-exporter(9100) - system metrics -
process-exporter(9256) - host process metrics -
cadvisor(9183) - Docker container metrics
Note: Bootstrap does not support Windows. For Windows servers, install Windows Exporter manually (see section 6).
Manual/Offline Installation (air-gapped servers)
For servers without internet access. Requires pre-downloaded files.
-
Download repository as ZIP from GitHub (on machine with internet)
-
Download Docker images:
bash misc/download-docker-images.sh -
Transfer files to server:
scp grafana.zip docker-images-bundle.tar user@server:/tmp/ -
On target server:
# Load Docker images sudo docker load -i /tmp/docker-images-bundle.tar # Extract repository and install cd /opt && sudo unzip /tmp/grafana.zip && sudo mv grafana-* grafana cd grafana && sudo bash misc/install-docker.sh sudo bash misc/quick-deploy.sh PROMETHEUS_IP # Auto-detects OS
What this does:
-
Loads pre-downloaded Docker images into local registry
-
Installs Docker and Docker Compose (if not present)
-
Deploys exporters from local images (no internet required)
-
Configures firewall rules for Prometheus access
Tips:
-
TROUBLESHOOTING: Run
sudo bash misc/diagnose.shto automatically check Docker, containers, ports, firewall, and Prometheus targets -
Environment Configuration: All deployments use environment variables (
.envfiles) for secure and flexible configuration
Pre-Deployment Checklist
Before starting, gather the following information:
-
Prometheus Server IP - Will be used in firewall rules on all nodes
-
SMTP Credentials (optional) - For Grafana alerting (host, port, user, password)
-
MySQL Credentials - If deploying MySQL exporter (user, password)
-
Oracle Credentials - If deploying Oracle exporter (user, password)
-
Server IPs - All servers that will be monitored
For clarity in these instructions, we use the following servers:
-
acs1,acs2,hc1,hc2,db,iis
In this example, Prometheus is installed on the same server as the DB. You need to know its IP address during setup to whitelist it on the servers where the exporters are installed:
ACS1 - 65.109.58.164 ACS2 - 65.109.58.100 HC1 - 65.109.24.154 HC2 - 65.108.67.167 DB - 65.109.58.165 <- PROMETHEUS_IP IIS - 65.109.49.150
1. Install Docker and Docker Compose on All Linux Servers
Target servers: acs1, acs2, hc1, hc2, db, *-api servers
Before installing exporters, ensure that Docker and Docker Compose are installed on all target servers (except Windows-based servers).
Automated Installation (Recommended)
Use the automated installation script:
# If repository is already cloned:
cd /opt/grafana
sudo bash misc/install-docker.sh
# Or via one-command bootstrap (installs Docker automatically):
curl -fsSL https://hub.friendly-tech.com/bootstrap/install.sh | sudo bash -s -- PROMETHEUS_IP
The script automatically:
-
✅ Detects your OS and package manager
-
✅ Fixes EOL repos (CentOS 7/8)
-
✅ Installs Docker CE from official repository
-
✅ Installs Docker Compose plugin (falls back to static binary if needed)
-
✅ Enables and starts Docker service
-
✅ Verifies installation
Verify Installation
docker -v
docker compose version
Expected output (versions may vary):
Docker version 28.0.1, build 068a01e Docker Compose version v2.33.1
Manual Installation (if script fails)
Ubuntu/Debian:
sudo apt-get update
sudo apt-get install -y ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update && sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-compose-plugin
sudo systemctl enable --now docker
2. Install Node Exporter and Process Exporter
Target servers: acs1, acs2, hc1, hc2, db, *-api servers
Quick Installation (Recommended)
Use the automated deployment script with service auto-detection:
# Auto-detect services and deploy (recommended):
cd /opt/grafana
sudo bash misc/quick-deploy.sh PROMETHEUS_IP
# Examples:
sudo bash misc/quick-deploy.sh 65.108.67.167
# Or deploy specific exporters manually (--exporter flag):
sudo bash misc/quick-deploy.sh 65.108.67.167 --exporter base --exporter jmx-wildfly
sudo bash misc/quick-deploy.sh 65.108.67.167 --exporter base --exporter mysql
# Or use one-command bootstrap (installs Docker, clones repo, deploys):
curl -fsSL https://hub.friendly-tech.com/bootstrap/install.sh | sudo bash -s -- 65.108.67.167
--exporterflag: Specifies which exporters to deploy fromexporters/folder. Usebase(always required) plus service-specific:jmx-wildfly,mysql,oracle. Note: Hazelcast uses native Prometheus endpoint (port 9101) - no separate exporter needed.
Service Auto-Detection: The script automatically detects running services and deploys appropriate exporters:
| Detected Service | Exporter Directory | Detection Method |
|---|---|---|
WildFly/FTACS |
|
|
Hazelcast |
(native prometheus) |
|
MySQL/MariaDB |
|
|
Oracle DB |
|
|
(always) |
|
Always deployed (node + process + cadvisor) |
The script automatically:
-
✅ Detects running services on the server
-
✅ Shows deployment summary with confirmation prompt
-
✅ Prompts for database credentials if needed (MySQL/Oracle)
-
✅ Deploys Node Exporter (port 9100) - system metrics
-
✅ Deploys Process Exporter (port 9256) - per-process metrics
-
✅ Deploys service-specific exporters based on detection
-
✅ Configures firewall rules with persistence
-
✅ Verifies deployment
Manual Installation (click to expand)
If you prefer manual installation or need to customize the setup:
2.0 Optional: Change Exporter Ports
All exporters are configured to run in network_mode: host (they bind directly to host network). By default, they use these ports:
| Exporter | Default Port | Purpose |
|---|---|---|
Node Exporter |
|
System metrics |
Process Exporter |
|
Host process metrics |
cAdvisor |
|
Docker container metrics |
MySQL Exporter |
|
MySQL database metrics |
Oracle DB Exporter |
|
Oracle database metrics |
JMX Exporter |
|
JMX metrics (Hikari) |
JMX Exporter |
|
JVM metrics |
If you need to override these ports (e.g., due to conflicts or multi-instance deployment):
-
Uncomment the relevant
--web.listen-address=…orcommand: […]line indocker-compose.yml. -
Update firewall and Prometheus scrape configs accordingly.
2.1 Get the Repository
We have two options for obtaining the necessary files: download the repository or clone it.
Option A: Download ZIP
-
Open the repo Friendly-Technologies/grafana in your browser.
-
Click Code and select Download ZIP.
-
Copy
grafana-main.zipto the/optdirectory on the server and extract it:
cd /opt && unzip grafana-main.zip -d /opt/grafana
2.2 File Structure
After downloading or cloning, you should have the following file structure:
/opt
└── grafana
├── exporters/ # Modular exporter directories
│ ├── base/ # Node + Process exporters (all servers)
│ │ ├── docker-compose.yml
│ │ ├── process-exporter.yml
│ │ └── .env.example
│ ├── jmx-wildfly/ # JMX exporter for WildFly/FTACS
│ │ ├── docker-compose.yml
│ │ ├── config-hikari.yml
│ │ ├── config-jvm.yml
│ │ └── .env.example
│ ├── mysql/ # MySQL/MariaDB exporter
│ │ ├── docker-compose.yml
│ │ └── .env.example
│ └── oracle/ # Oracle DB exporter
│ ├── docker-compose.yml
│ ├── config.yaml.template
│ ├── generate-config.sh
│ └── .env.example
├── prometheus-grafana-stack/ # Monitoring stack
│ ├── docker-compose.yml
│ ├── .env.example
│ └── prometheus/
│ ├── prometheus.yml
│ ├── scripts/
│ │ └── generate-targets.sh
│ └── targets/
│ └── servers.env.example
├── docs/
│ └── architecture.md # Architecture diagrams
├── misc/
│ ├── bootstrap.sh # One-command installer
│ ├── quick-deploy.sh # Auto-detect and deploy exporters
│ ├── detect-services.sh # Service auto-detection
│ ├── diagnose.sh # Troubleshooting tool
│ └── install-docker.sh # Docker installer
├── README.md
└── sizing.md # Capacity planning guide
2.3 Deploy Exporters
The deployment automatically detects running services and deploys appropriate exporters:
# Auto-detect services and deploy (recommended)
cd /opt/grafana
sudo bash misc/quick-deploy.sh 65.109.58.165
For manual deployment, navigate to the relevant exporter directory:
# Base exporters (Node + Process) - all servers
cd /opt/grafana/exporters/base
# Hazelcast: No exporter needed - uses native Prometheus endpoint (port 9101)
# See section "Prerequisites: Enable Prometheus on Hazelcast Servers"
# JMX exporters for WildFly/FTACS
cd /opt/grafana/exporters/jmx-wildfly
-
Configure environment variables: Copy the example environment file and edit it with your configuration:
cp .env.example .env vi .env
The .env file contains configuration for exporter versions, ports, and other settings. Most values have sensible defaults and can be left as-is. Key variables:
-
NODE_EXPORTER_PORT- Default: 9100 -
PROCESS_EXPORTER_PORT- Default: 9256 -
JMX_HIKARI_PORT- Default: 5556 (for ACS servers) -
JMX_JVM_PORT- Default: 5557 (for ACS servers) Note: If you don’t create a.envfile, the default values will be used automatically.-
Start the exporters using Docker Compose:
-
docker compose up -d
Hint: You can check the logs of the started containers:
docker compose logs -f
-
Enable Prometheus to access exporter ports:
iptables -A INPUT -p tcp -s $PROMETHEUS_IP --dport 9100 -j ACCEPT # node-exporter iptables -A INPUT -p tcp -s $PROMETHEUS_IP --dport 9256 -j ACCEPT # process-exporter iptables -A INPUT -p tcp -s $PROMETHEUS_IP --dport 9183 -j ACCEPT # cadvisor # JMX ports (if WildFly detected): iptables -A INPUT -p tcp -s $PROMETHEUS_IP --dport 5556 -j ACCEPT # jmx-hikari iptables -A INPUT -p tcp -s $PROMETHEUS_IP --dport 5557 -j ACCEPT # jmx-jvm # Hazelcast native Prometheus (if Hazelcast detected): iptables -A INPUT -p tcp -s $PROMETHEUS_IP --dport 9101 -j ACCEPT # hazelcast native
In our example we took ip address of prometheus and commands should be looking like that:
iptables -A INPUT -p tcp -s 65.109.58.165 --dport 9100 -j ACCEPT # node-exporter
iptables -A INPUT -p tcp -s 65.109.58.165 --dport 9256 -j ACCEPT # process-exporter
iptables -A INPUT -p tcp -s 65.109.58.165 --dport 9183 -j ACCEPT # cadvisor
# JMX ports (if WildFly detected):
iptables -A INPUT -p tcp -s 65.109.58.165 --dport 5556 -j ACCEPT # jmx-hikari
iptables -A INPUT -p tcp -s 65.109.58.165 --dport 5557 -j ACCEPT # jmx-jvm
# Hazelcast native Prometheus (if Hazelcast detected):
iptables -A INPUT -p tcp -s 65.109.58.165 --dport 9101 -j ACCEPT # hazelcast native
3. Install JMX Exporter (WildFly/ACS Servers)
Target servers:
-
acs1,acs2- WildFly/FTACS application servers
Note: Hazelcast servers use native Prometheus endpoint (port 9101) - no JMX exporter needed. See “Manual: Enable Prometheus on Hazelcast” below.
Skip this section if: Auto-detection already deployed JMX exporters.
JMX Exporter Types
| Server Type | Exporter Directory | Ports | Metrics |
|---|---|---|---|
WildFly/FTACS |
|
5556, 5557 |
HikariCP pool, JVM heap/GC/threads |
Hazelcast |
(native prometheus) |
9101 |
Cluster members, partitions, maps, JVM |
Quick Installation
cd /opt/grafana
# Auto-detect (recommended) - deploys appropriate exporters based on detected services
sudo bash misc/quick-deploy.sh 65.108.67.167
# Or deploy specific exporters manually:
# For WildFly/FTACS servers:
sudo bash misc/quick-deploy.sh 65.108.67.167 --exporter base --exporter jmx-wildfly
Note: The
--exporterflag specifies which exporters to deploy fromexporters/folder.
base- always required (node-exporter + process-exporter + cAdvisor)
jmx-wildfly,mysql,oracle- service-specific exportersHazelcast uses native Prometheus endpoint (port 9101) - no separate exporter needed
Verify JMX Exporters
# Check containers are running
docker ps | grep jmx
# Test metrics endpoints
# WildFly:
curl -s http://localhost:5556/metrics | head # HikariCP
curl -s http://localhost:5557/metrics | head # JVM
# Hazelcast:
curl -s http://localhost:9101/metrics | head # Hazelcast
Note: JMX Exporters connect to WildFly JMX port on localhost (port 9999). Hazelcast uses native Prometheus endpoint (port 9101) - no JMX connection needed.
Manual: Enable JMX on WildFly/ACS
Before JMX exporters can collect metrics, WildFly/ACS must be configured to expose JMX on port 9999. One-time configuration on each ACS server.
Step 1: Add to /usr/local/FTACS/bin/standalone.conf:
JAVA_OPTS="$JAVA_OPTS \
-Dcom.sun.management.jmxremote \
-Dcom.sun.management.jmxremote.port=9999 \
-Dcom.sun.management.jmxremote.rmi.port=9999 \
-Dcom.sun.management.jmxremote.ssl=false \
-Dcom.sun.management.jmxremote.authenticate=false \
-Djava.rmi.server.hostname=127.0.0.1"
Step 2: Enable HikariCP MBeans in /usr/local/FTACS/standalone/configuration/ftacs_hikari.properties:
registerMbeans=true
Step 3: Restart ACS:
# For MySQL:
service jboss restart
# For Oracle:
service jboss restartoracle
Tip: Check current DB type:
ps aux | grep -o '\-Dftacs.usedDBMS=[^ ]*'
Verify: ss -tlnp | grep 9999
Manual: Enable Prometheus on Hazelcast
Hazelcast uses native Prometheus endpoint on port 9101. The quick-deploy.sh script can configure this automatically.
Manual configuration:
Step 1: Add to /usr/local/hazelcast-5.0/bin/hz-start:
export PROMETHEUS_PORT=9101
Step 2: Restart Hazelcast: systemctl restart hazelcast5
Verify: curl -s http://localhost:9101/metrics | head
4. Install MySQL Exporter
Target server: db
Quick Installation (Recommended)
Use the automated deployment script:
# If you already have Docker installed and repository cloned:
cd /opt/grafana
sudo bash misc/quick-deploy.sh 65.109.58.165 --exporter base --exporter mysql
# Or use one-command bootstrap (auto-detects MySQL):
curl -fsSL https://hub.friendly-tech.com/bootstrap/install.sh | sudo bash -s -- 65.109.58.165
Important: Before running the script, create the MySQL exporter user (see step 1 below).
Manual Installation (click to expand)
Run the following commands in the shell console on the target server:
4.0 Optional: Change Exporter Port
By default, MySQL Exporter listens on port 9104.
To override it (e.g., due to a conflict or multiple exporters):
-
Open
/opt/grafana/exporters/mysql/docker-compose.yml -
Uncomment and modify the
--web.listen-address=:PORTparameter in thecommand:block:
# Uncomment to override default port (9104)
# - '--web.listen-address=:19104'
4.1 Setup Steps
-
Create an exporter user in MySQL: Log in as
rootoradminand run the following commands (generate a secure password and replaceStR0nG_PassW0rD_hErEaccordingly):
CREATE USER IF NOT EXISTS 'exporter'@'localhost' IDENTIFIED BY 'StR0nG_PassW0rD_hErE' WITH MAX_USER_CONNECTIONS 3; GRANT PROCESS, REPLICATION CLIENT, SELECT ON *.* TO 'exporter'@'localhost';
-
Clone or download this repo into
/opt/grafana, in the same way as described in step 2. -
Configure environment variables: Navigate to the MySQL exporter directory and create the
.envfile:
cd /opt/grafana/exporters/mysql/ cp .env.example .env vi .env
Update the following variables with your MySQL credentials:
# MySQL Connection Details (REQUIRED)
MYSQL_HOST=localhost
MYSQL_PORT=3306
MYSQL_USER=exporter
MYSQL_PASSWORD=StR0nG_PassW0rD_hErE # Replace with your password
Optional variables (can be left as default):
-
MYSQLD_EXPORTER_VERSION- Default: latest -
MYSQLD_EXPORTER_PORT- Default: 9104-
Start the exporter using Docker Compose:
-
docker compose up -d
-
Enable Prometheus to access exporter ports:
iptables -A INPUT -p tcp -s $PROMETHEUS_IP --dport 9104 -j ACCEPT
5. Install Oracle DB Exporter
Target server: oracle-db or any server with Oracle Database access
Quick Installation (Recommended)
Use the automated deployment script:
# If you already have Docker installed and repository cloned:
cd /opt/grafana
sudo bash misc/quick-deploy.sh 65.109.58.165 --exporter base --exporter oracle
# Or use one-command bootstrap (auto-detects Oracle):
curl -fsSL https://hub.friendly-tech.com/bootstrap/install.sh | sudo bash -s -- 65.109.58.165
Note: Bootstrap and quick-deploy will prompt for Oracle credentials interactively.
Manual Installation (click to expand)
Run the following commands in the shell console on the target server:
5.1 Setup Oracle DB Exporter
-
Clone or download this repo into
/opt/grafana, in the same way as described in step 2. -
Navigate to the Oracle exporter directory:
cd /opt/grafana/exporters/oracle/
-
Configure environment variables: Create the
.envfile with your Oracle Database credentials:
cp .env.example .env vi .env
Update the following variables:
# Oracle Database Connection (REQUIRED)
ORACLE_HOST=localhost
ORACLE_PORT=1521
ORACLE_SERVICE_NAME=ORCL
ORACLE_USER=system
ORACLE_PASSWORD=your_oracle_password
Optional variables (can be left as default):
-
ORACLE_EXPORTER_VERSION- Default: 2.2.0 -
ORACLE_EXPORTER_PORT- Default: 9161-
Generate config.yaml from template:
-
./generate-config.sh
This script reads .env and generates config.yaml with substituted values.
. Start the exporter using Docker Compose:
docker compose up -d
Hint: You can check the logs of the started container:
docker compose logs -f
-
Verify the exporter is working:
curl http://localhost:9161/metrics
You should see Oracle Database metrics including:
-
oracledb_sessions_value- Database sessions -
oracledb_tablespace_bytes- Tablespace usage -
oracledb_activity_*- Database activity metrics -
oracledb_resource_*- Resource utilization-
Enable Prometheus to access exporter port:
-
iptables -A INPUT -p tcp -s $PROMETHEUS_IP --dport 9161 -j ACCEPT
In our example:
iptables -A INPUT -p tcp -s 65.109.58.165 --dport 9161 -j ACCEPT
6. Install Windows Exporter
Target server: iis
Note: Windows Exporter installation is manual only (no automated script available).
Download and Install
-
Download from Windows Exporter Releases
-
Run in Command Prompt (as Administrator):
lodctr.exe /E:Lsa lodctr.exe /E:PerfProc lodctr.exe /R msiexec /i "C:\Users\Administrator\windows_exporter-0.31.3-amd64.msi" ENABLED_COLLECTORS=ad,adfs,cache,cpu,cpu_info,container,dfsr,dhcp,dns,fsrmquota,iis,logical_disk,memory,msmq,mssql,net,os,process,remote_fx,service,tcp,time,vmware LISTEN_PORT=9100 EXTRA_FLAGS="--collector.process.include=""(w3wp|java.*)"" --collector.process.iis --collector.process.counter-version=2"
Configure Firewall
netsh advfirewall firewall add rule name="Allow Windows Exporter" dir=in action=allow protocol=TCP localport=9100
Add to Monitoring
After Windows Exporter is running, add the server to servers.env on the Prometheus server:
# === IIS SERVERS ===
NODE <server-ip>:9100 <Prefix>-IIS service=IIS,role=webserver,node_ip=<server-ip>
Then regenerate targets:
cd /opt/grafana/prometheus-grafana-stack/prometheus
./scripts/generate-targets.sh
Note: Windows Exporter uses the
NODEtype since it exposes metrics on the same port (9100) in a compatible format. Theservice=IISlabel is used by the Windows/IIS Grafana dashboard for filtering.
7. Configure and Deploy Prometheus with Grafana
Target server: db (or separate server dedicated for Grafana only)
In this example, we are using the db server to host Grafana and Prometheus, but there is no strict requirement—it can be any server that suits your environment.
Hardware requirements: For small-scale setups (up to 10 server nodes), 4 vCPU and 8 GB RAM is sufficient. For larger deployments, see Grafana sizing guidelines.
Quick Installation (Recommended)
Use the one-command bootstrap:
curl -fsSL https://hub.friendly-tech.com/bootstrap/install.sh | sudo bash -s -- stack
The script will interactively prompt for all configuration parameters:
| Section | Parameters |
|---|---|
Server |
IP, Prometheus port, Grafana port, Nginx port |
Prometheus |
Data retention period |
Grafana |
Admin user, password, plugins |
SMTP |
Enabled, host, user, password |
Service Account |
Token (can be added after setup) |
-
Press Enter to keep current/default value
-
Type new value to change it
-
All parameters are read from
/opt/grafana/prometheus-grafana-stack/.env
The script automatically:
-
✅ Installs Docker if not present
-
✅ Clones/updates repository to
/opt/grafana -
✅ Creates
.envfrom.env.exampleif missing -
✅ Configures all parameters interactively
-
✅ Deploys Prometheus, Grafana, and Nginx via Docker Compose
-
✅ Shows access URLs and credentials
After deployment:
-
Configure Prometheus targets (see section 7.1 below)
-
Generate Grafana service token for nginx (see section 7.2 below)
Manual Installation (click to expand)
-
Clone or download this repo into
/opt/grafana, as described previously. -
Configure environment variables: Navigate to the Prometheus-Grafana stack directory and create the
.envfile:
cd /opt/grafana/prometheus-grafana-stack cp .env.example .env vi .env
Update the following REQUIRED variables:
# Server Configuration (REQUIRED)
SERVER_IP=65.109.58.165 # Replace with your server's IP
# Grafana Admin Credentials (REQUIRED)
# NOTE: Password in .env is used only on FIRST deployment.
# After that, database is the source of truth.
# To change password later, use: docker exec grafana grafana cli admin reset-admin-password NEW_PASS
GRAFANA_ADMIN_USER=admin
GRAFANA_ADMIN_PASSWORD=your_secure_password_here
# SMTP Configuration (REQUIRED for alerting)
SMTP_ENABLED=true
SMTP_HOST=smtp.example.com:587
SMTP_USER=your_smtp_user
SMTP_PASSWORD=your_smtp_password
# Grafana Service Token (REQUIRED for nginx proxy)
GRAFANA_SERVICE_TOKEN=your_token_here # Will be generated in section 7.2
Optional variables (can be left as default):
-
PROMETHEUS_VERSION,GRAFANA_VERSION,NGINX_VERSION -
PROMETHEUS_PORT=9090,GRAFANA_PORT=3000,NGINX_PORT=3030 -
PROMETHEUS_RETENTION=90d -
GRAFANA_PLUGINS- Comma-separated list of plugins Full environment variables reference
Variable |
Description |
Required |
|
Your server’s external IP address or hostname |
✅ Yes |
|
Admin username for Grafana authentication |
✅ Yes |
|
Admin password for Grafana authentication |
✅ Yes |
|
SMTP server address and port for email alerts (e.g., smtp.gmail.com:587) |
✅ Yes |
|
SMTP username (usually an email address) |
✅ Yes |
|
SMTP password or API key |
✅ Yes |
|
Service account token for nginx proxy (generated in section 7.2) |
✅ Yes |
|
Data retention period (default: 90d) |
❌ No |
|
Prometheus port (default: 9090) |
❌ No |
|
Grafana port (default: 3000) |
❌ No |
|
Nginx proxy port (default: 3030) |
❌ No |
For a complete list of all variables, see .env.example in each directory.
Grafana Internal Environment Variables Reference:
These variables are set automatically by docker-compose based on your .env file:
| Variable | Description |
|---|---|
|
Root URL of Grafana instance, used to generate correct links in the UI |
|
Allowed origin for CORS (Cross-Origin Resource Sharing) |
|
Allowed origins for WebSocket connections (Grafana Live) |
|
Admin username for Grafana authentication |
|
Admin password for Grafana authentication |
|
Enables SMTP email notifications ( |
|
SMTP server address and port for sending alerts |
|
SMTP username (usually an email address) |
|
SMTP password or API key |
|
Allows embedding dashboards in iframes ( |
|
Enables JSON-based dashboard provisioning |
|
Custom CSS file path for UI styling |
-
Start the Prometheus-Grafana stack:
cd /opt/grafana/prometheus-grafana-stack docker compose up -d
-
Enable access to Grafana and Prometheus ports:
iptables -A INPUT -p tcp --dport 9090 -j ACCEPT iptables -A INPUT -p tcp --dport 3000 -j ACCEPT iptables -A INPUT -p tcp --dport 3030 -j ACCEPT
-
Test the setup:
-
Prometheus: http://grafana_ip:9090/targets?search= — all endpoints should be in UP state
-
Grafana: http://grafana_ip:3000/dashboards
-
-
Configure Alerting (optional): Update the email in
/opt/grafana/prometheus-grafana-stack/grafana/provisioning/alerting/alert_resources.yamlwith your email(s). Multiple emails can be separated by;,\n, or,.
7.1 Configure Prometheus Targets
After the stack is running, configure which servers Prometheus should monitor.
Tip: When you deploy exporters using
quick-deploy.shorbootstrap.sh, the script outputs ready-to-use configuration lines forservers.env. Just copy and paste them.
Step 1: Edit the server inventory
cd /opt/grafana/prometheus-grafana-stack/prometheus/targets
vi servers.env
First time setup:
Run
./scripts/generate-targets.sh— createsservers.envfrom templateEdit
servers.env— add your serversRun
./scripts/generate-targets.shagain — generates JSON target filesYour
servers.envis git-ignored and won’t be overwritten by repository updates.
Format: TYPE IP:PORT INSTANCE [EXTRA_LABELS]
| Field | Description | Example |
|---|---|---|
|
Exporter type (see supported types below) |
|
|
Server IP and exporter port |
|
|
Unique instance name for Grafana |
|
|
Optional comma-separated Prometheus labels |
|
EXTRA_LABELS are additional labels attached to all metrics from this target. Useful for:
-
Filtering in Grafana dashboards (
service=mysql) -
Grouping servers by role (
role=database,role=application) -
Environment tagging (
env=prod,env=staging)
Instance Naming Convention (recommended):
Instance names are flexible - you can use any naming scheme that works for your organization. The recommended format is {Environment}-{ServerType}{Number}:
| Prefix | Server Role | Examples |
|---|---|---|
|
Application servers (WildFly/FTACS) |
|
|
Database servers (MySQL, Oracle) |
|
|
Hazelcast cache servers (standalone) |
|
|
Microservices (Provision, Northbound, Service, Subscription) |
|
|
Windows/IIS web servers |
|
💡 Tip: Use the
rolelabel for server grouping:role=database,role=application,role=cache,role=api,role=webserver,role=monitoring
Which exporters for which server type:
| Server Type | Required Exporters | Optional Exporters |
|---|---|---|
Any server |
|
- |
MySQL DB |
|
- |
Oracle DB |
|
- |
WildFly/ACS |
|
|
Hazelcast (standalone) |
|
- |
API servers |
|
- |
Windows/IIS |
|
- |
Minimal example (one server of each type):
# MySQL database server
NODE 10.0.0.10:9100 Prod-DB
PROCESS 10.0.0.10:9256 Prod-DB
CADVISOR 10.0.0.10:9183 Prod-DB
MYSQL 10.0.0.10:9104 Prod-DB
# WildFly/ACS application server
NODE 10.0.0.20:9100 Prod-ACS1
PROCESS 10.0.0.20:9256 Prod-ACS1
CADVISOR 10.0.0.20:9183 Prod-ACS1
JMX_HIKARI 10.0.0.20:5556 Prod-ACS1
JMX_JVM 10.0.0.20:5557 Prod-ACS1
# Hazelcast cache server (standalone)
NODE 10.0.0.30:9100 Prod-HC1 role=cache
PROCESS 10.0.0.30:9256 Prod-HC1
CADVISOR 10.0.0.30:9183 Prod-HC1
JMX_HC 10.0.0.30:9101 Prod-HC1
# API server (microservices)
NODE 10.0.0.35:9100 Prod-API1 role=api
PROCESS 10.0.0.35:9256 Prod-API1
CADVISOR 10.0.0.35:9183 Prod-API1
PROVISION_API 10.0.0.35:8091 Prod-API1
NORTHBOUND_API 10.0.0.35:9880 Prod-API1
SERVICE_API 10.0.0.35:8085 Prod-API1
SUBSCRIPTION_API 10.0.0.35:8080 Prod-API1
# Oracle database server
NODE 10.0.0.40:9100 Prod-OracleDB
PROCESS 10.0.0.40:9256 Prod-OracleDB
CADVISOR 10.0.0.40:9183 Prod-OracleDB
ORACLE 10.0.0.40:9161 Prod-OracleDB
# Windows/IIS web server
NODE 10.0.0.50:9100 Prod-IIS service=IIS,role=webserver
Full example with labels and API endpoints
# === DATABASE SERVERS ===
NODE 65.109.58.165:9100 DevOps-DB service=mysql,role=database
PROCESS 65.109.58.165:9256 DevOps-DB
CADVISOR 65.109.58.165:9183 DevOps-DB
MYSQL 65.109.58.165:9104 DevOps-DB
# === ACS SERVERS ===
NODE 65.109.58.164:9100 DevOps-ACS1 service=ACS,role=application
PROCESS 65.109.58.164:9256 DevOps-ACS1
CADVISOR 65.109.58.164:9183 DevOps-ACS1
JMX_HIKARI 65.109.58.164:5556 DevOps-ACS1
JMX_JVM 65.109.58.164:5557 DevOps-ACS1
# Optional ACS endpoints:
WILDFLY 65.109.58.164:8080 DevOps-ACS1
FT_SYSTEM 65.109.58.164:8090 DevOps-ACS1
ACS_METRICS 65.109.58.164:8080 DevOps-ACS1
JMX_HC 65.109.58.164:9101 DevOps-ACS1
# === HAZELCAST SERVERS (standalone) ===
NODE 65.109.24.154:9100 DevOps-HC1 service=hazelcast,role=cache
PROCESS 65.109.24.154:9256 DevOps-HC1
CADVISOR 65.109.24.154:9183 DevOps-HC1
JMX_HC 65.109.24.154:9101 DevOps-HC1
# === API SERVERS (microservices) ===
NODE 65.109.24.155:9100 DevOps-API1 service=api,role=api
PROCESS 65.109.24.155:9256 DevOps-API1
CADVISOR 65.109.24.155:9183 DevOps-API1
PROVISION_API 65.109.24.155:8091 DevOps-API1
NORTHBOUND_API 65.109.24.155:9880 DevOps-API1
SERVICE_API 65.109.24.155:8085 DevOps-API1
SUBSCRIPTION_API 65.109.24.155:8080 DevOps-API1
# === ORACLE SERVERS ===
NODE 65.109.20.174:9100 DevOps-OracleDB service=oracle,role=database
PROCESS 65.109.20.174:9256 DevOps-OracleDB
CADVISOR 65.109.20.174:9183 DevOps-OracleDB
ORACLE 65.109.20.174:9161 DevOps-OracleDB
# === IIS SERVERS ===
NODE 65.109.49.150:9100 devops-ui.friendly-tech.com service=IIS,role=webserver,node_ip=65.109.49.150
Step 2: Generate Prometheus target files
cd /opt/grafana/prometheus-grafana-stack/prometheus
./scripts/generate-targets.sh
The script automatically:
-
✅ Parses
servers.env -
✅ Generates JSON files in
targets/folder (one per exporter type) -
✅ Adds
exporter_typelabel to each target (e.g.,node,hazelcast,hikari) -
✅ Validates JSON syntax
-
✅ Checks target reachability
-
✅ Shows summary of generated targets
How it works:
prometheus.ymlusesfile_sd_configsto read JSON files fromtargets/folder. Prometheus auto-reloads within 30 seconds — no restart needed!
Adding new servers later:
-
Edit
servers.env— add new server lines -
Run
./scripts/generate-targets.sh -
Done! Prometheus picks up changes automatically
Benefits:
-
✅ Zero downtime - No Prometheus restart required
-
🛡️ Error-proof - Cannot break YAML syntax
-
📝 Simple format - Easy to read and edit
-
🔄 Auto-reload - Prometheus picks up changes automatically
-
🔒 Safe updates - Your
servers.envis git-ignored and won’t be overwritten
Supported target types (TYPE field reference)
The TYPE field in servers.env determines which Prometheus job will scrape the target. Each type maps to a specific exporter and port:
| TYPE | Port | Description | Prometheus Job | exporter_type |
|---|---|---|---|---|
|
9100 |
System metrics (CPU, memory, disk) |
|
|
|
9256 |
Host process metrics |
|
|
|
9183 |
Docker container metrics |
|
|
|
9104 |
MySQL/MariaDB database metrics |
|
|
|
9161 |
Oracle database metrics |
|
|
|
5556 |
HikariCP connection pool (WildFly) |
|
|
|
5557 |
JVM heap, GC, threads (WildFly) |
|
|
|
9101 |
Hazelcast native Prometheus |
|
|
|
8080 |
WildFly server metrics |
|
|
|
8090 |
FT System Spring Boot metrics |
|
|
|
8080 |
ACS application metrics |
|
|
|
8091 |
Provision Portal API metrics |
|
|
|
9880 |
Northbound API metrics |
|
|
|
8085 |
Service API metrics |
|
|
|
8080 |
Subscription API metrics |
|
|
Note: The port in the table is the default. Always specify the actual port in your
servers.envline:TYPE IP:PORT INSTANCE
How IP addresses appear in Grafana dashboards
No configuration needed — prometheus.yml already includes relabel_configs that extract IP addresses from target addresses and store them in the node_ip label.
Grafana dashboards use this label to display server IPs in tables and filters.
How it works (FYI):
# prometheus.yml already has this for each job:
relabel_configs:
- source_labels: [__address__]
target_label: node_ip
regex: (.+):.* # extracts IP from "IP:PORT"
replacement: $1
7.2 Configure Nginx Proxy for Embedding
Enable a read-only reverse proxy for embedding Grafana dashboards in external applications.
Note: If you deployed using bootstrap (
curl … | sudo bash -s — stack), the service token is generated automatically. Skip to “Test the Nginx reverse proxy” below.
Test the Nginx reverse proxy:
Open http://grafana_ip:3030/ — you should see Grafana dashboards without needing to log in, with read-only permissions.
Manual token generation (if bootstrap failed or manual installation)
Run the automated script:
cd /opt/grafana
sudo bash misc/generate-grafana-token.sh --restart
The script creates a Grafana service account (ReadOnlySA) with Viewer role, generates a token, and updates .env automatically.
If the script fails, generate the token manually:
-
Open Grafana: http://your_server_ip:3000
-
Go to: Administration → Users and access → Service accounts
-
Click “Add service account” → Name:
ReadOnlySA, Role:Viewer→ Add -
Click “Add service account token” → Name:
nginx_ro→ Copy the token -
Update
.env:
cd /opt/grafana/prometheus-grafana-stack vi .env # Set: GRAFANA_SERVICE_TOKEN=glsa_YourActualTokenHere_12345678
-
Restart:
docker compose down && docker compose up -d
How nginx proxy works
The nginx container uses a template file (nginx.conf.template) with ${GRAFANA_SERVICE_TOKEN} placeholder. On startup, the entrypoint script:
-
Reads the template
-
Replaces placeholder with value from
.env -
Generates final
nginx.conf -
Starts nginx
All services run in network_mode: host. Nginx connects to Grafana via http://127.0.0.1:3000 and listens on port 3030.
7.3 Embedding Dashboards in iframe
The nginx proxy on port 3030 provides read-only access without authentication, making it perfect for embedding dashboards in external applications.
How to get dashboard URL for embedding:
-
Open the dashboard in Grafana (via port 3000 with authentication)
-
Copy the dashboard path (e.g.,
/d/abc123/my-dashboard) -
Add parameters:
-
?orgId=1- Organization ID (required) -
&kiosk- Kiosk mode (hides Grafana menus) -
&refresh=30s- Auto-refresh interval (optional)
-
Example iframe code:
<iframe
src="/d/business-metrics/business-metrics?orgId=1&kiosk&refresh=30s"
width="100%"
height="600"
frameborder="0">
</iframe>
Available kiosk modes:
-
&kiosk- Full kiosk mode (no top nav, no side menu) -
&kiosk=tv- TV mode (hides top nav, shows only dashboard)
Common parameters:
-
&from=now-6h&to=now- Time range -
&var-instance=ServerName- Dashboard variable values -
&theme=dark- Dark theme -
&theme=light- Light theme
Important notes:
-
Do NOT use “Share externally” button in Grafana UI (requires additional permissions)
-
Service account with Viewer role cannot create shared dashboards
-
Always use direct dashboard URLs through port 3030
-
Test the URL in browser before embedding to ensure it works
Verify nginx proxy is working:
# Should return 200 OK without authentication
curl -I http://YOUR_SERVER_IP:3030/
8. Managing Grafana Dashboards Provisioning
By default, Grafana automatically provisions all dashboards located under:
/opt/grafana/prometheus-grafana-stack/grafana/provisioning/dashboards/definitions/
Sometimes it is not necessary (or desirable) to install all dashboards at once. You have two options:
8.1 Skip Dashboards During Installation
-
Before running the stack, remove or move unwanted dashboard JSON files from the definitions folder:
mkdir -p /opt/grafana/prometheus-grafana-stack/grafana/provisioning/dashboards/disabled mv /opt/grafana/prometheus-grafana-stack/grafana/provisioning/dashboards/definitions/<dashboard-to-skip>.json \ /opt/grafana/prometheus-grafana-stack/grafana/provisioning/dashboards/disabled/
Grafana will only load dashboards present in the definitions folder during startup.
-
Start the stack as usual:
cd /opt/grafana/prometheus-grafana-stack docker compose up -d
8.2 Add Dashboards Later
If later you decide to enable some dashboards:
-
Move the desired JSON file(s) back into the definitions folder:
mv /opt/grafana/prometheus-grafana-stack/grafana/provisioning/dashboards/disabled/<dashboard>.json \ /opt/grafana/prometheus-grafana-stack/grafana/provisioning/dashboards/definitions/
-
Restart Grafana container to apply changes:
cd /opt/grafana/prometheus-grafana-stack docker compose restart grafana
Tip: You don’t need to restart the whole stack, only the Grafana container.
Troubleshooting
Quick Diagnostics: Run sudo bash misc/diagnose.sh to automatically check Docker, containers, ports, firewall, and Prometheus targets.
Common Issues and Solutions
Prometheus cannot connect to exporter
Symptoms: Target is DOWN in Prometheus UI, curl hangs
Step 1: Is exporter listening?
ss -ltpn | grep <port>
Step 2: Is container running?
docker ps
docker logs -f <CONTAINER_ID>
Step 3: Is exporter accessible from Prometheus?
docker exec $(docker ps | grep prom/prometheus | awk '{print $1}') \
wget http://<EXPORTER_IP>:<PORT>/metrics -O /tmp/test
If connection hangs — check firewall allows access from Prometheus host.
Spaces in .env Files
Wrong: MYSQL_USER = exporter (spaces around =)
Correct: MYSQL_USER=exporter (no spaces)
Forgetting to Create .env File
Symptom: Containers start but use default/empty values
Solution: Always run cp .env.example .env before docker compose up
Firewall Rules Not Persistent
Problem: Rules disappear after reboot
Solution: Our scripts auto-save rules. Manual save:
# Ubuntu/Debian
netfilter-persistent save
# CentOS/Rocky
service iptables save
Wrong Prometheus IP in Firewall
Symptom: Prometheus shows targets as “DOWN”
Solution: Verify Prometheus IP and update firewall rules.
Conflicting Container Names
Problem: “The container name is already in use”
Solution:
docker compose down
docker compose up -d
MySQL exporter shows mysql_up 0
Solution:
systemctl status mysql
mysql -u exporter -p -e "SHOW GRANTS;"
Operations
Updating Monitoring Stack
curl -fsSL https://hub.friendly-tech.com/bootstrap/install.sh | sudo bash -s -- stack
Select “Keep current configuration” to preserve your .env settings.
For exporters:
cd /opt/grafana
sudo bash misc/quick-deploy.sh PROMETHEUS_IP
Updating Configuration
vi .env
docker compose up -d
docker compose restart nginx # if nginx config changed
Updating Docker Images
vi .env # Change version, e.g., PROMETHEUS_VERSION=v3.6.0
docker compose pull prometheus
docker compose up -d prometheus
Rollback Deployment
The deployment script creates a manifest tracking all changes. To rollback:
# Rollback to previous state (latest manifest)
curl -fsSL https://hub.friendly-tech.com/bootstrap/install.sh | sudo bash -s -- --rollback
# Or directly:
sudo /opt/grafana/misc/quick-deploy.sh --rollback
# List available manifests
sudo /opt/grafana/misc/quick-deploy.sh --list-manifests
# Rollback specific deployment
sudo /opt/grafana/misc/quick-deploy.sh --rollback /opt/grafana/.deploy-manifests/manifest-20260201-143022
What gets rolled back:
-
Stops deployed containers
-
Removes added firewall rules
-
Restores original configuration files (WildFly, Hazelcast)
Note: Services like WildFly/ACS may need manual restart after rollback.
Offline/Air-gapped Installation
Prepare (machine with internet):
bash misc/download-docker-images.sh
scp grafana.zip docker-images-bundle.tar user@server:/tmp/
Install (air-gapped server):
sudo docker load -i /tmp/docker-images-bundle.tar
cd /opt && sudo unzip /tmp/grafana.zip && sudo mv grafana-* grafana
sudo bash misc/quick-deploy.sh PROMETHEUS_IP
Security Best Practices
-
Never commit
.envfiles to git -
Use strong passwords (16+ characters)
-
Restrict permissions:
chmod 600 .env -
Rotate passwords every 90 days
-
Use dedicated users with minimal grants
Deployment Checklist
Before Starting:
-
Docker installed, repository cloned to /opt/grafana
-
Prometheus IP address known
-
Credentials gathered (MySQL/Oracle/SMTP)
For Each Server:
-
.envconfigured, containers deployed -
Metrics endpoints working (
curl localhost:PORT/metrics) -
Firewall rules saved
On Prometheus/Grafana:
-
Targets show “UP” in Prometheus UI
-
Dashboards loading, alerts configured
-
Nginx proxy working (port 3030)
For Developers
Documentation
This repository automatically maintains Antora-formatted documentation alongside the Markdown README.
Automatic Conversion
When you commit changes to README.md, a pre-commit hook automatically converts it to AsciiDoc format for Antora documentation. The converted file is placed in docs/modules/ROOT/pages/index.adoc and staged for the commit.
Setting Up Git Hooks
After cloning this repository, install the Git hooks:
./misc/install-hooks.sh
This installs a pre-commit hook that:
-
Detects when
README.mdis staged for commit -
Converts it to AsciiDoc using
pandoc -
Stages the converted
index.adocfile automatically
Requirements:
-
pandocmust be installed:
# macOS brew install pandoc # Ubuntu/Debian sudo apt-get install pandoc # CentOS/RHEL sudo yum install pandoc
Development & Releases
The project uses version branches for development and releases.
Workflow:
1. Work on version branch (e.g., 1.6.5) 2. When ready: ./misc/release.sh [patch|minor|major] → Merges to main → Creates GitHub Release with changelog → Creates next version branch (e.g., 1.6.6) 3. Continue development on new branch
Version branch naming:
-
Branches:
1.6.5(withoutvprefix) -
Tags:
1.6.5(withoutvprefix) -
Releases:
1.6.5(withoutvprefix)
Using specific versions:
# List available versions
curl -fsSL https://hub.friendly-tech.com/bootstrap/install.sh | sudo bash -s -- --version-list
# Install specific version
curl -fsSL https://hub.friendly-tech.com/bootstrap/install.sh | sudo bash -s -- stack --branch 1.6.4