Runbook: Verify the Installation

Version 1.6.9-option-3 | Updated: August 07, 2026

Runbook: Verify the Installation

Verifying the Installation

Everything below runs entirely offline — no internet access and no tools beyond what a standard Linux host and this repository already provide.

1. Containers are up:

docker ps

Nothing should be in a Restarting state. Give nginx_proxy a specific look: a crash-looping nginx makes both Grafana and Prometheus unreachable from outside, while every exporter container next to it still shows Up and looks perfectly healthy — the outage is easy to miss if you only skim the list.

2. The built-in diagnostic:

sudo bash /opt/grafana/misc/diagnose.sh

Checks Docker, containers, ports, firewall, and Prometheus targets in one pass.

3. Every Prometheus target is up:

curl -s http://localhost:9090/api/v1/targets | python3 -c '
import json, sys
data = json.load(sys.stdin)["data"]["activeTargets"]
down = [t for t in data if t["health"] != "up"]
print("{} targets, {} not up".format(len(data), len(down)))
for t in down:
    lb = t["labels"]
    print("  {}: {} {}".format(t["health"], lb.get("job"), lb.get("instance")))'

4. Grafana answers:

DOMAIN=$(grep -m1 '^GRAFANA_DOMAIN=' /opt/grafana/prometheus-grafana-stack/.env | cut -d= -f2)
curl -sk -o /dev/null -w '%{http_code}\n' -H "Host: $DOMAIN" https://127.0.0.1/login

Expect 200. Two details make this check fail for the wrong reason if you shorten it:

  • The Host header is required. nginx serves the stack from a server_name ${GRAFANA_DOMAIN} block and keeps a default_server that answers everything else with return 444 — a silent close. So curl https://localhost/login gets an empty reply and %{http_code} prints 000, which looks exactly like a dead Grafana while the stack is perfectly healthy.

  • Probe through nginx on 443, not Grafana’s own port (see Accessing Web Interfaces). Grafana binds to 127.0.0.1:3100 only, so the internal port answers on the host itself but never from another machine.

To separate “nginx is broken” from “Grafana is broken”, query Grafana directly on the host — 200 here with a failure above means the proxy is the problem:

curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:3100/login

5. The dashboards actually have data:

cd /opt/grafana && bash misc/testing/check-dashboard-panels.sh --quiet

A target being UP is not the same as a dashboard working: a panel can query a metric series that simply does not exist on this host, and it then shows “No data” with nothing anywhere reporting a problem. This script runs every panel’s own query against Prometheus and reports which ones come back empty. The exit code is the number of dashboards with at least one unexpected empty panel, and credentials are read from .env automatically — no flags needed on a standard deployment.

Some empty panels are expected and not a sign of a broken install:

  • windows-exporter-iis-node — empty when there are no Windows hosts in this deployment

  • oracle — empty on a MySQL deployment (and the mysql dashboards empty on an Oracle one)

  • api-metrics — api_method_requests_total only appears after real API traffic has occurred

  • business-metrics — needs devices in the FTACS database

  • a handful of panels in nodes-monitoring / system-dashboard — Pressure Stall Information needs a recent kernel, power-supply panels need hardware sensors that may not exist on this host/VM

On the reference monitoring host, a run reports 13 empty panels out of 475 — every one explained by something that host doesn’t have, not by a defect. That count is not a target to match; it moves with what each host actually runs.