Prometheus, Grafana, and Dashboards

Version 1.6.9-option-2 | Updated: August 07, 2026

Prometheus, Grafana, and Dashboards

8. Configure and Deploy Prometheus with Grafana

Target server: db (or separate server dedicated for Grafana only)

In this example, we are using the db server to host Grafana and Prometheus, but there is no strict requirement—it can be any server that suits your environment.

Hardware requirements: For small-scale setups (up to 10 server nodes), 4 vCPU and 8 GB RAM is sufficient. For larger deployments, see Grafana sizing guidelines.

Quick Installation (Recommended)

Use the one-command bootstrap:

curl -fsSL https://hub.friendly-tech.com/bootstrap/install.sh | sudo bash -s -- stack

On the interactive path, bootstrap.sh stack opens a full-screen arrow-key configuration menu (↑↓ + Enter to navigate, Esc to go back, per-item help panel, masked passwords; nothing is written until Apply):

Menu section What it configures

Server

SERVER_IP, access domain (GRAFANA_DOMAIN — also the TLS cert CN)

Prometheus

port, data retention, basic-auth user/password (protects the Prometheus/Alertmanager UIs)

Grafana

port, admin user/password, and the shared SMTP (host/from/user/password) used by both Grafana and alert email

FTACS Database

MySQL/Oracle type + host/port/user/password/name for the Business dashboard (empty host = skip)

SSL / HTTPS

self-signed (validity/org), Let’s Encrypt (email), or an existing certificate (paths shown)

Alerting

drill into any channel to configure it — Email, Telegram (live chat auto-detect), Slack, Teams, Webhook, SMS, SNMP (v2c/v3)

UI Portal integration

UI_PORTAL_ORIGINS + extra allowed networks for /grafana-ro/ embedding

Review & apply

shows every value, then writes .env and applies

After Apply the script deploys the stack (containers up), auto-registers this host’s exporters in servers.env and generates targets, creates the read-only Grafana service token, and — on an existing install — changes the admin password via grafana cli if you set a new one (no current password needed). Values live in /opt/grafana/prometheus-grafana-stack/.env.

Run with -y to skip the menu entirely (non-interactive: domain = server IP, random passwords, self-signed cert).

After deployment:

  1. Configure Prometheus targets (see section 8.1 below)

  2. Verify /grafana-ro/ works if you configured UI Portal Origins in phase 5

Manual Installation (click to expand)

  1. Clone or download this repo into /opt/grafana, as described previously.

  2. Configure environment variables: Navigate to the Prometheus-Grafana stack directory and create the .env file:

cd /opt/grafana/prometheus-grafana-stack
cp .env.example .env
vi .env

Update the following REQUIRED variables:

# Server Configuration (REQUIRED)
SERVER_IP=65.109.58.165  # Replace with your server's IP

# Grafana Admin Credentials (REQUIRED)
# NOTE: Password in .env is used only on FIRST deployment.
# After that, database is the source of truth.
# To change password later, use: docker exec grafana grafana cli admin reset-admin-password NEW_PASS
GRAFANA_ADMIN_USER=admin
GRAFANA_ADMIN_PASSWORD=your_secure_password_here

# SMTP Configuration (REQUIRED for alerting)
SMTP_ENABLED=true
SMTP_HOST=smtp.example.com:587
SMTP_USER=your_smtp_user
SMTP_PASSWORD=your_smtp_password

# Grafana Service Token (REQUIRED for nginx proxy)
GRAFANA_SERVICE_TOKEN=your_token_here  # Will be generated in section 8.2

Optional variables (can be left as default):

  • PROMETHEUS_VERSION, GRAFANA_VERSION, NGINX_VERSION

  • PROMETHEUS_PORT=9090 — Prometheus’s own listen port. GRAFANA_PORT=443 — the external HTTPS port nginx listens on for Grafana, not Grafana’s own port (Grafana listens on 127.0.0.1:3100, hardcoded in docker-compose.yml; there is no variable for it)

  • PROMETHEUS_RETENTION=90d

  • GRAFANA_PLUGINS - Comma-separated list of plugins Full environment variables reference

Variable

Description

Required

SERVER_IP

Your server’s external IP address or hostname

✅ Yes

GRAFANA_ADMIN_USER

Admin username for Grafana authentication

✅ Yes

GRAFANA_ADMIN_PASSWORD

Admin password for Grafana authentication

✅ Yes

SMTP_HOST

SMTP server address and port for email alerts (e.g., smtp.gmail.com:587)

✅ Yes

SMTP_USER

SMTP username (usually an email address)

✅ Yes

SMTP_PASSWORD

SMTP password or API key

✅ Yes

GRAFANA_SERVICE_TOKEN

Service account token for nginx proxy (generated in section 8.2)

✅ Yes

PROMETHEUS_RETENTION

Data retention period (default: 90d)

❌ No

PROMETHEUS_PORT

Prometheus port (default: 9090)

❌ No

GRAFANA_PORT

External HTTPS port nginx listens on for Grafana (default: 443) — not Grafana’s own port, which is hardcoded to 127.0.0.1:3100 and has no variable

❌ No

For a complete list of all variables, see .env.example in each directory. Grafana Internal Environment Variables Reference: These variables are set automatically by docker-compose based on your .env file:

Variable Description

GF_SERVER_ROOT_URL

Root URL of Grafana instance, used to generate correct links in the UI

GF_SERVER_ALLOW_ORIGIN

Allowed origin for CORS (Cross-Origin Resource Sharing)

GF_LIVE_ALLOWED_ORIGINS

Allowed origins for WebSocket connections (Grafana Live)

GF_SECURITY_ADMIN_USER

Admin username for Grafana authentication

GF_SECURITY_ADMIN_PASSWORD

Admin password for Grafana authentication

GF_SMTP_ENABLED

Enables SMTP email notifications (true = enabled)

GF_SMTP_HOST

SMTP server address and port for sending alerts

GF_SMTP_USER

SMTP username (usually an email address)

GF_SMTP_PASSWORD

SMTP password or API key

GF_SECURITY_ALLOW_EMBEDDING

Allows embedding dashboards in iframes (true = enabled)

GF_DASHBOARDS_JSON_ENABLED

Enables JSON-based dashboard provisioning

GF_DASHBOARDS_CUSTOM_CSS

Custom CSS file path for UI styling

  1. Prepare generated files (the bootstrap script creates these automatically — a manual/offline installation must create them before the first start, otherwise Docker turns the missing bind-mount files into directories and the containers fail with “Are you trying to mount a directory onto a file”):

cd /opt/grafana/prometheus-grafana-stack

# Alertmanager config placeholder (regenerated by the container entrypoint)
cat > alertmanager/alertmanager.yml << 'EOF'
# Placeholder - will be regenerated by alertmanager entrypoint.sh
global:
  resolve_timeout: 5m
route:
  receiver: 'default'
receivers:
  - name: 'default'
EOF
chmod 666 alertmanager/alertmanager.yml

# TLS certificate for the nginx proxy (self-signed; replace with a real
# certificate later if you have one — files: ssl/certificate.crt, ssl/private.key).
# ssl/ is git-ignored, so it is NOT in the ZIP — create it first.
mkdir -p ssl
openssl req -x509 -nodes -days 365 -newkey rsa:2048 \
  -keyout ssl/private.key -out ssl/certificate.crt \
  -subj "/C=US/ST=State/L=City/O=FriendlyTech/CN=${GRAFANA_DOMAIN:-$(hostname -f)}"

# Prometheus basic-auth file for nginx (user: prometheus). Also git-ignored, so
# nginx would otherwise mount a directory onto it and fail. htpasswd may be
# absent on RHEL — fall back to openssl apr1; write the password back into .env.
PROM_PASS=$(openssl rand -base64 12 | tr -dc 'a-zA-Z0-9' | head -c16)
if command -v htpasswd >/dev/null 2>&1; then
  htpasswd -bc .htpasswd prometheus "$PROM_PASS"
else
  echo "prometheus:$(openssl passwd -apr1 "$PROM_PASS")" > .htpasswd
fi
grep -q '^PROMETHEUS_PASSWORD=' .env \
  && sed -i "s/^PROMETHEUS_PASSWORD=.*/PROMETHEUS_PASSWORD=$PROM_PASS/" .env \
  || echo "PROMETHEUS_PASSWORD=$PROM_PASS" >> .env
echo "Prometheus UI login: prometheus / $PROM_PASS"

# nginx ACME webroot (bind-mounted; keep it a directory)
mkdir -p acme-challenge

Also make sure GRAFANA_DOMAIN in .env is set to the server’s domain or IP — nginx serves Grafana only for that hostname. . Start the Prometheus-Grafana stack:

cd /opt/grafana/prometheus-grafana-stack
docker compose up -d

  1. Enable access to Grafana and Prometheus ports:

iptables -A INPUT -p tcp --dport 80 -j ACCEPT
iptables -A INPUT -p tcp --dport 443 -j ACCEPT
# If server has IPv6 address, also add ip6tables rules:
ip6tables -A INPUT -p tcp --dport 80 -j ACCEPT
ip6tables -A INPUT -p tcp --dport 443 -j ACCEPT

Note: Ports 9090 (Prometheus) and 3100 (Grafana) bind to 127.0.0.1 and are only accessible through nginx reverse proxy. No need to open them in firewall.

  1. Test the setup:

  2. Configure Alerting (optional): Update the email in /opt/grafana/prometheus-grafana-stack/grafana/provisioning/alerting/alert_resources.yaml with your email(s). Multiple emails can be separated by ;, \n, or ,.

Accessing Web Interfaces

After deployment, the monitoring stack provides two web interfaces accessible via nginx reverse proxy:

Service URL Username Password

Grafana

https://<domain>/

admin

.env → GRAFANA_ADMIN_PASSWORD

Prometheus

https://<domain>/prometheus

prometheus

.env → PROMETHEUS_PASSWORD

Note: Both services bind to 127.0.0.1 and are only accessible through nginx. Direct access via IP:3100 or IP:9090 is not available.

Note: The “Business metrics” dashboard requires connection to your FTACS application database (MySQL or Oracle) for tenant/domain lists. Configure FTACS_DB_* variables in .env (set FTACS_DB_TYPE to mysql or oracle) — this is the ACS database, not the monitoring database.

How it works:

All external traffic goes through nginx reverse proxy (ports 80/443). Nginx handles SSL, authentication, and forwards requests to internal services:

Browser → https://<domain>/          → nginx (port 443) → Grafana (127.0.0.1:3100)
Browser → https://<domain>/prometheus → nginx (port 443) → Prometheus (127.0.0.1:9090)

Grafana and Prometheus listen only on 127.0.0.1 (localhost) — they are not reachable directly from outside. This is why only ports 80 and 443 need to be opened in the firewall, not 3100 or 9090.

How to find your passwords:

cd /opt/grafana/prometheus-grafana-stack

# View Grafana password
grep GRAFANA_ADMIN_PASSWORD .env

# View Prometheus password
grep PROMETHEUS_PASSWORD .env

# Reset Grafana password if forgotten
docker exec grafana grafana cli admin reset-admin-password NEW_PASSWORD

Grafana UI pages:

Prometheus UI pages:

8.1 Configure Prometheus Targets

After the stack is running, configure which servers Prometheus should monitor.

Tip: When you deploy exporters using quick-deploy.sh or bootstrap.sh, the script outputs ready-to-use configuration lines for servers.env. Just copy and paste them.

The stack server registers itself. Re-running bootstrap.sh stack appends this host’s own exporters if they are not listed yet, so it is safe to run repeatedly. NODE, PROCESS and CADVISOR describe the machine, so there is exactly one of each per host: an entry already counts as present if either its address or its instance name matches. That matters because both can change under you — a new NIC or DHCP lease moves the address, and a different INSTANCE_PREFIX renames the instance. Matching on one key alone would let the other change through as a brand-new target, and the host would be scraped twice under two names, doubling every host-level panel. Nothing is ever rewritten in place: when the existing line disagrees with what was just detected, bootstrap prints both spellings and leaves the file alone, e.g.

⚠ servers.env: NODE already listed as '10.0.0.30:9100 DB-MySQL' for this host
  (detected: 10.0.0.30:9100 stack-DB-MySQL) — not duplicating; edit the line if it is out of date

Edit the line yourself if the old value is wrong, then re-run generate-targets.sh.

Step 1: Edit the server inventory

cd /opt/grafana/prometheus-grafana-stack/prometheus/targets
vi servers.env

First time setup:

  1. Run /opt/grafana/prometheus-grafana-stack/prometheus/scripts/generate-targets.sh — creates servers.env from template

  2. Edit servers.env — add your servers

  3. Run /opt/grafana/prometheus-grafana-stack/prometheus/scripts/generate-targets.sh again — generates JSON target files

Your servers.env is git-ignored and won’t be overwritten by repository updates.

Format: TYPE IP:PORT INSTANCE [EXTRA_LABELS]

Field Description Example

TYPE

Exporter type (see supported types below)

NODE, MYSQL, JMX_HIKARI

IP:PORT

Server IP and exporter port

65.109.58.165:9100

INSTANCE

Unique instance name for Grafana

DevOps-DB, Prod-ACS1

EXTRA_LABELS

Optional comma-separated Prometheus labels

service=mysql,role=database

EXTRA_LABELS are additional labels attached to all metrics from this target. Useful for:

  • Filtering in Grafana dashboards (service=mysql)

  • Grouping servers by role (role=database, role=application)

  • Environment tagging (env=prod, env=staging)

Instance Naming Convention (recommended):

Instance names are flexible - you can use any naming scheme that works for your organization. The recommended format is {Environment}-{ServerType}{Number}:

Prefix Server Role Examples

ACS

Application servers (WildFly/FTACS)

DevOps-ACS1, Prod-ACS2

DB

Database servers (MySQL, Oracle)

DevOps-DB, Prod-OracleDB

HC

Hazelcast cache servers (standalone)

DevOps-HC1, Prod-HC

API

Microservices (Provision, Northbound, Service, Subscription)

DevOps-APIs, Prod-API1

IIS

Windows/IIS web servers

DevOps-IIS, Prod-IIS1

💡 Tip: Use the role label for server grouping: role=database, role=application, role=cache, role=api, role=webserver, role=monitoring

Which exporters for which server type:

Server Type Required Exporters Optional Exporters

Any server

NODE, PROCESS, CADVISOR

-

MySQL DB

MYSQL

-

Oracle DB

ORACLE

-

ClickHouse

CLICKHOUSE

-

PostgreSQL

POSTGRES

-

WildFly/ACS

JMX_HIKARI, JMX_JVM, JMX_HC

ACS_METRICS, FT_SYSTEM

Hazelcast (standalone)

JMX_HC (native Prometheus)

-

Angular UI (HC1)

NGINX_PORTALS, FT_DEVICE_NETWORK, AI_AGENT

UI_BACKEND (requires actuator)

API servers

PROVISION_API, NORTHBOUND_API, SERVICE_API

-

FTACS servers

SUBSCRIPTION_API (native on port 8080)

-

Windows/IIS

NODE (Windows Exporter on port 9100)

-

Minimal example (one server of each type):

# MySQL database server
NODE 10.0.0.10:9100 Prod-DB
PROCESS 10.0.0.10:9256 Prod-DB
CADVISOR 10.0.0.10:9183 Prod-DB
MYSQL 10.0.0.10:9104 Prod-DB

# WildFly/ACS application server
NODE 10.0.0.20:9100 Prod-ACS1
PROCESS 10.0.0.20:9256 Prod-ACS1
CADVISOR 10.0.0.20:9183 Prod-ACS1
JMX_HIKARI 10.0.0.20:5556 Prod-ACS1
JMX_JVM 10.0.0.20:5557 Prod-ACS1

# Hazelcast cache server (standalone)
NODE 10.0.0.30:9100 Prod-HC1 role=cache
PROCESS 10.0.0.30:9256 Prod-HC1
CADVISOR 10.0.0.30:9183 Prod-HC1
JMX_HC 10.0.0.30:9101 Prod-HC1

# API server (microservices)
NODE 10.0.0.35:9100 Prod-API1 role=api
PROCESS 10.0.0.35:9256 Prod-API1
CADVISOR 10.0.0.35:9183 Prod-API1
PROVISION_API 10.0.0.35:8091 Prod-API1
NORTHBOUND_API 10.0.0.35:9880 Prod-API1
SERVICE_API 10.0.0.35:8085 Prod-API1
SUBSCRIPTION_API 10.0.0.35:8080 Prod-API1

# Oracle database server
NODE 10.0.0.40:9100 Prod-OracleDB
PROCESS 10.0.0.40:9256 Prod-OracleDB
CADVISOR 10.0.0.40:9183 Prod-OracleDB
ORACLE 10.0.0.40:9161 Prod-OracleDB

# Windows/IIS web server
NODE 10.0.0.50:9100 Prod-IIS service=IIS,role=webserver

Full example with labels and API endpoints

# === DATABASE SERVERS ===
NODE 65.109.58.165:9100 DevOps-DB service=mysql,role=database
PROCESS 65.109.58.165:9256 DevOps-DB
CADVISOR 65.109.58.165:9183 DevOps-DB
MYSQL 65.109.58.165:9104 DevOps-DB

# === ACS SERVERS ===
NODE 65.109.58.164:9100 DevOps-ACS1 service=ACS,role=application
PROCESS 65.109.58.164:9256 DevOps-ACS1
CADVISOR 65.109.58.164:9183 DevOps-ACS1
JMX_HIKARI 65.109.58.164:5556 DevOps-ACS1
JMX_JVM 65.109.58.164:5557 DevOps-ACS1
# Optional ACS endpoints:
FT_SYSTEM 65.109.58.164:8090 DevOps-ACS1
ACS_METRICS 65.109.58.164:8080 DevOps-ACS1
JMX_HC 65.109.58.164:9101 DevOps-ACS1

# === HAZELCAST SERVERS (standalone) ===
NODE 65.109.24.154:9100 DevOps-HC1 service=hazelcast,role=cache
PROCESS 65.109.24.154:9256 DevOps-HC1
CADVISOR 65.109.24.154:9183 DevOps-HC1
JMX_HC 65.109.24.154:9101 DevOps-HC1

# === API SERVERS (microservices) ===
NODE 65.109.24.155:9100 DevOps-API1 service=api,role=api
PROCESS 65.109.24.155:9256 DevOps-API1
CADVISOR 65.109.24.155:9183 DevOps-API1
PROVISION_API 65.109.24.155:8091 DevOps-API1
NORTHBOUND_API 65.109.24.155:9880 DevOps-API1
SERVICE_API 65.109.24.155:8085 DevOps-API1
SUBSCRIPTION_API 65.109.24.155:8080 DevOps-API1

# === ORACLE SERVERS ===
NODE 65.109.20.174:9100 DevOps-OracleDB service=oracle,role=database
PROCESS 65.109.20.174:9256 DevOps-OracleDB
CADVISOR 65.109.20.174:9183 DevOps-OracleDB
ORACLE 65.109.20.174:9161 DevOps-OracleDB

# === IIS SERVERS ===
NODE 65.109.49.150:9100 devops-ui.friendly-tech.com service=IIS,role=webserver,node_ip=65.109.49.150

Step 2: Generate Prometheus target files

cd /opt/grafana/prometheus-grafana-stack/prometheus
/opt/grafana/prometheus-grafana-stack/prometheus/scripts/generate-targets.sh

The script automatically:

  • ✅ Parses servers.env

  • ✅ Generates JSON files in targets/ folder (one per exporter type)

  • ✅ Adds exporter_type label to each target (e.g., node, hazelcast, hikari)

  • ✅ Validates JSON syntax

  • ✅ Checks target reachability

  • ✅ Shows summary of generated targets

Note: The script validates all entries before generating targets. If there are format errors (wrong TYPE, missing instance name, labels in instance field), it will show the exact line, what’s wrong, and how to fix it.

How it works: prometheus.yml uses file_sd_configs to read JSON files from targets/ folder. Prometheus auto-reloads within 30 seconds — no restart needed!

Adding new servers later:

  1. Edit servers.env — add new server lines

  2. Run /opt/grafana/prometheus-grafana-stack/prometheus/scripts/generate-targets.sh

  3. Done! Prometheus picks up changes automatically

Benefits:

  • ✅ Zero downtime - No Prometheus restart required

  • 🛡️ Error-proof - Cannot break YAML syntax

  • 📝 Simple format - Easy to read and edit

  • 🔄 Auto-reload - Prometheus picks up changes automatically

  • 🔒 Safe updates - Your servers.env is git-ignored and won’t be overwritten

Supported target types (TYPE field reference)

The TYPE field in servers.env determines which Prometheus job will scrape the target. Each type maps to a specific exporter and port:

TYPE Port Description Prometheus Job exporter_type

NODE

9100

System metrics (CPU, memory, disk)

node-exporter

node

PROCESS

9256

Host process metrics

process-exporter

process

CADVISOR

9183

Docker container metrics

cadvisor

cadvisor

MYSQL

9104

MySQL/MariaDB database metrics

mysql

mysql

ORACLE

9161

Oracle database metrics

oracledb

oracle

CLICKHOUSE

9363

ClickHouse database metrics (native endpoint)

clickhouse

clickhouse

POSTGRES

9187

PostgreSQL database metrics

postgres

postgres

NGINX_PORTALS

9113

Nginx web server metrics (Angular UI)

nginx

nginx

JMX_HIKARI

5556

HikariCP connection pool (WildFly)

JmxExporter

hikari

JMX_JVM

5557

JVM heap, GC, threads (WildFly)

JmxExporter

jvm

JMX_HC

9101

Hazelcast: native endpoint (embedded in ACS) or the jmx-hazelcast exporter (standalone/docker, JMX :9110 → :9101)

JmxExporter

hazelcast

FT_SYSTEM

8090

FT System Spring Boot metrics (WildFly)

ft-system-metrics

ft-system

ACS_METRICS

8080

ACS application metrics

acs-metrics

acs-metrics

FT_DEVICE_NETWORK

8383

FT Device Network Service (Actuator)

ft-device-network

ft-device-network

AI_AGENT

8084

AI Agent Python/FastAPI metrics

ai-agent

ai-agent

UI_BACKEND

8881

UI Backend Spring Boot Actuator

ui-backend

ui-backend

PROVISION_API

8091

Provision Portal API metrics

provision-api

provision-api

NORTHBOUND_API

9880

Northbound API metrics

northbound-api

northbound-api

SERVICE_API

8085

Service API metrics

service-api

service-api

SUBSCRIPTION_API

8080

Subscription API metrics

subscription-api

subscription-api

Parking a target that stopped answering. generate-targets.sh probes every row and reports the ones that did not respond. Two flags act on that list — neither ever deletes a line, because a row records an address somebody added for a reason and deleting it takes the reason with it:

# show what did not answer this run; changes nothing
bash /opt/grafana/prometheus-grafana-stack/prometheus/scripts/generate-targets.sh --park-dry-run

# comment those rows out, keeping the original line, with a backup
bash /opt/grafana/prometheus-grafana-stack/prometheus/scripts/generate-targets.sh --park

A parked row looks like this, and comes back by deleting the three #:

# PARKED 2026-08-05 - did not answer when targets were last generated
# JMX_HC 135.181.62.187:9101 ACS-qa12m
# -> uncomment when JMX_HC answers again at 135.181.62.187:9101

Parking never happens on its own. A target that is merely restarting reads as unreachable too, so --park has to be typed by someone who has looked at the dry-run list. A row that answered is never touched.

Metrics paths. Most targets are scraped on /metrics, but the Spring Boot services are not, and the path is not guessable from the service name. These are the paths the generator writes into metrics_path — use them when probing a target by hand:

  • NORTHBOUND_API and SERVICE_API — /iot-webservice/actuator/prometheus

  • PROVISION_API — /prov-portal/actuator/prometheus

  • FT_DEVICE_NETWORK — /api/actuator/prometheus

  • SUBSCRIPTION_API — /rest/subscription-metrics (ACS 6.4 only; 6.5+ serves these through ACS_METRICS instead, and the generator skips the target when the endpoint answers without metrics)

  • FT_SYSTEM, UI_BACKEND, AI_AGENT, ACS_METRICS — /actuator/prometheus

So a target that looks down is worth re-checking on its own path before anything else:

curl -s -o /dev/null -w '%{http_code}\n' \
  http://<ip>:8091/prov-portal/actuator/prometheus     # not /metrics

Note: The port in the table is the default. Always specify the actual port in your servers.env line: TYPE IP:PORT INSTANCE

How IP addresses appear in Grafana dashboards

No configuration needed — prometheus.yml already includes relabel_configs that extract IP addresses from target addresses and store them in the node_ip label.

Grafana dashboards use this label to display server IPs in tables and filters.

How it works (FYI):

# prometheus.yml already has this for each job:
relabel_configs:
  - source_labels: [__address__]
    target_label: node_ip
    regex: (.+):.*      # extracts IP from "IP:PORT"
    replacement: $1

8.2 Configure Nginx Proxy for Embedding

Enable a read-only reverse proxy for embedding Grafana dashboards in external applications (e.g., Angular UI portal).

The /grafana-ro/ endpoint requires two things configured in .env:

  • GRAFANA_SERVICE_TOKEN — Grafana service account token (auto-generated by bootstrap, or create manually — see below)

  • UI_PORTAL_ORIGINS — comma-separated list of portal URLs allowed to access /grafana-ro/ (IP whitelist)

Step 1: Configure allowed portal origins

Set UI_PORTAL_ORIGINS in .env to the URLs of applications that will embed Grafana:

cd /opt/grafana/prometheus-grafana-stack
vi .env

# Example: allow access from Angular UI portal
UI_PORTAL_ORIGINS=https://portal.example.com,https://10.0.0.50:8880

Without this setting, /grafana-ro/ returns 403 Forbidden for all external requests.

Step 2: Verify service token

Check that GRAFANA_SERVICE_TOKEN is set (not CHANGE_ME):

grep GRAFANA_SERVICE_TOKEN .env

If it shows CHANGE_ME, generate a token — see “Manual token generation” below.

Step 3: Restart containers

docker compose down && docker compose up -d

Test the Nginx reverse proxy:

Open https://grafana_domain/grafana-ro/ — you should see Grafana dashboards without needing to log in, with read-only permissions.

Manual token generation (if bootstrap failed or manual installation)

Run the automated script:

cd /opt/grafana
sudo bash /opt/grafana/misc/generate-grafana-token.sh --restart

The script creates a Grafana service account (ReadOnlySA) with Viewer role, generates a token, and updates .env automatically.

If the script fails, generate the token manually:

  1. Open Grafana: https:/// (Grafana binds to 127.0.0.1 only — it is reachable through nginx, not directly via the server IP)

  2. Go to: Administration → Users and access → Service accounts

  3. Click “Add service account” → Name: ReadOnlySA, Role: Viewer → Add

  4. Click “Add service account token” → Name: nginx_ro → Copy the token

  5. Update .env:

cd /opt/grafana/prometheus-grafana-stack
vi .env
# Set: GRAFANA_SERVICE_TOKEN=glsa_YourActualTokenHere_12345678

  1. Restart: docker compose down && docker compose up -d

How nginx proxy works

The nginx container uses a template file (nginx.conf.template) with ${GRAFANA_SERVICE_TOKEN} placeholder. On startup, the entrypoint script:

  1. Reads the template

  2. Replaces placeholder with value from .env

  3. Generates final nginx.conf

  4. Starts nginx

All services run in network_mode: host. Nginx connects to Grafana via http://127.0.0.1:3100 and serves HTTPS on ports 80/443.

8.3 Embedding Dashboards in iframe

The /grafana-ro/ path provides read-only access without authentication, making it perfect for embedding dashboards in external applications.

How to get dashboard URL for embedding:

  1. Open the dashboard in Grafana (via https://<your-domain>/, with authentication)

  2. Copy the dashboard path (e.g., /d/abc123/my-dashboard)

  3. Add parameters:

    • ?orgId=1 - Organization ID (required)

    • &kiosk - Kiosk mode (hides Grafana menus)

    • &refresh=30s - Auto-refresh interval (optional)

Example iframe code:

<iframe
  src="/d/business-metrics/business-metrics?orgId=1&kiosk&refresh=30s"
  width="100%"
  height="600"
  frameborder="0">
</iframe>

Available kiosk modes:

  • &kiosk - Full kiosk mode (no top nav, no side menu)

  • &kiosk=tv - TV mode (hides top nav, shows only dashboard)

Common parameters:

  • &from=now-6h&to=now - Time range

  • &var-instance=ServerName - Dashboard variable values

  • &theme=dark - Dark theme

  • &theme=light - Light theme

Important notes:

  • Do NOT use “Share externally” button in Grafana UI (requires additional permissions)

  • Service account with Viewer role cannot create shared dashboards

  • Always use direct dashboard URLs through /grafana-ro/ path

  • Test the URL in browser before embedding to ensure it works

Verify nginx proxy is working:

# Should return 200 OK without authentication
curl -kI https://YOUR_SERVER_DOMAIN/grafana-ro/

9. Managing Grafana Dashboards Provisioning

By default, Grafana automatically provisions all dashboards located under:

/opt/grafana/prometheus-grafana-stack/grafana/provisioning/dashboards/definitions/

Sometimes it is not necessary (or desirable) to install all dashboards at once. You have two options:

9.1 Skip Dashboards During Installation

  1. Before running the stack, remove or move unwanted dashboard JSON files from the definitions folder:

mkdir -p /opt/grafana/prometheus-grafana-stack/grafana/provisioning/dashboards/disabled
mv /opt/grafana/prometheus-grafana-stack/grafana/provisioning/dashboards/definitions/<dashboard-to-skip>.json \
/opt/grafana/prometheus-grafana-stack/grafana/provisioning/dashboards/disabled/

Grafana will only load dashboards present in the definitions folder during startup.

  1. Start the stack as usual:

cd /opt/grafana/prometheus-grafana-stack
docker compose up -d

9.2 Add Dashboards Later

If later you decide to enable some dashboards:

  1. Move the desired JSON file(s) back into the definitions folder:

mv /opt/grafana/prometheus-grafana-stack/grafana/provisioning/dashboards/disabled/<dashboard>.json \
/opt/grafana/prometheus-grafana-stack/grafana/provisioning/dashboards/definitions/

  1. Restart Grafana container to apply changes:

cd /opt/grafana/prometheus-grafana-stack
docker compose restart grafana

Tip: You don’t need to restart the whole stack, only the Grafana container.