Troubleshooting

Table of Contents

Troubleshooting
- Troubleshooting

Version 1.6.9-option-2 | Updated: August 07, 2026

Troubleshooting

Quick Diagnostics: Run sudo bash /opt/grafana/misc/diagnose.sh to automatically check Docker, containers, ports, firewall, and Prometheus targets.

Common Issues and Solutions

Prometheus cannot connect to exporter

Symptoms: Target is DOWN in Prometheus UI, curl hangs

Step 1: Is exporter listening?

ss -ltpn | grep <port>

Step 2: Is container running?

docker ps
docker logs -f <CONTAINER_ID>

Step 3: Is exporter accessible from Prometheus?

docker exec $(docker ps | grep prom/prometheus | awk '{print $1}') \
  wget http://<EXPORTER_IP>:<PORT>/metrics -O /tmp/test

If connection hangs — check firewall allows access from Prometheus host.

Cannot Login to Grafana

Symptom: Grafana login page loads but admin password is rejected (Invalid username or password)

Cause: Grafana stores the admin password in its SQLite database on first startup. Changing GRAFANA_ADMIN_PASSWORD in .env and restarting the container does not update the existing password — it only applies when creating a new admin user.

Solution: Reset password via CLI:

# On the Grafana server:
bash /opt/grafana/misc/bootstrap.sh --reset-password "new_password"

# Or directly:
docker exec grafana grafana cli admin reset-admin-password "new_password"

Spaces in .env Files

Wrong: MYSQL_USER = exporter (spaces around =)

Correct: MYSQL_USER=exporter (no spaces)

Forgetting to Create .env File

Symptom: Containers start but use default/empty values

Solution: Always run cp .env.example .env before docker compose up

Firewall Rules Not Persistent

Problem: Rules disappear after reboot

Solution: Our scripts auto-save rules. Manual save:

# Ubuntu/Debian
netfilter-persistent save

# CentOS/Rocky
service iptables save

Wrong Prometheus IP in Firewall

Symptom: Prometheus shows targets as “DOWN”

Solution: Verify Prometheus IP and update firewall rules.

Conflicting Container Names

Problem: “The container name is already in use”

Solution:

docker compose down
docker compose up -d

MySQL exporter shows mysql_up 0

Solution:

systemctl status mysql
mysql -u exporter -p -e "SHOW GRANTS;"

Nginx 502 Bad Gateway

Solution:

docker logs nginx_proxy
docker exec nginx_proxy env | grep GRAFANA_SERVICE_TOKEN
docker compose restart nginx

cAdvisor shows no container metrics (containerd image store)

Symptom: cAdvisor returns only root container (id="/"), no individual Docker containers visible. System Dashboard Basic shows no FT UI Services. cAdvisor logs show:

Failed to create existing container: failed to identify the read-write layer ID
open /rootfs/var/lib/docker/image/overlayfs/layerdb/mounts/.../mount-id: no such file or directory

Cause: Storage Driver: overlayfs means Docker uses the containerd image store — the default for fresh installs of Docker Engine ≥ 29 (opt-in on 28, and on Debian 13 / RHEL 9 static-tarball installs). There is no /var/lib/docker/image//layerdb in that mode, and *cAdvisor before v0.54.0 cannot resolve a container’s read-write layer: it skips every container and reports only the root cgroup, so all container_* series for that host are lost.

Upstream fixed this in cAdvisor v0.54.0 (PR #3709, released 2025-12-02) — it reads the container rootfs from the containerd spec instead of the graphdriver metadata. This repo pins a newer build, so the fix is the cAdvisor version, not the Docker storage driver.

Two conditions, not one. Because the new code talks to containerd, cAdvisor’s docker factory needs the socket dockerd actually uses. With a wrong path the container starts, the read-write layer ID spam stops and everything looks fixed, but the log says Registration of the docker container factory failed: … dial unix /run/containerd/containerd.sock and every container_* series arrives without name/container_label_* — only raw cgroup ids (verified on grafrd: 13 named containers with a reachable socket, 0 without). cAdvisor’s default is /run/containerd/containerd.sock, correct for package installs; the offline static-tarball install (dockerd spawns its own containerd) keeps it under /run/docker/containerd/. Hence CONTAINERD_SOCK in exporters/base/.env, detected by detect_containerd_sock() (misc/lib/container-helpers.sh) on every quick-deploy run.

Check:

docker info | grep -E 'Storage Driver|driver-type'   # overlayfs + io.containerd.snapshotter.v1
docker inspect --format='{{.Config.Image}}' cadvisor  # must be >= v0.54.0
docker logs --since 10m cadvisor 2>&1 | grep -c 'read-write layer ID'
docker logs --since 10m cadvisor 2>&1 | grep 'docker container factory'
source /opt/grafana/misc/lib/container-helpers.sh && detect_containerd_sock

Solution — upgrade cAdvisor and pin the socket. No Docker restart, no downtime for other containers:

cd /opt/grafana/exporters/base
sed -i 's|^CADVISOR_VERSION=.*|CADVISOR_VERSION=v0.55.1|' .env   # or add the line if missing
source /opt/grafana/misc/lib/container-helpers.sh
SOCK=$(detect_containerd_sock)
grep -q '^CONTAINERD_SOCK=' .env \
  && sed -i "s|^CONTAINERD_SOCK=.*|CONTAINERD_SOCK=$SOCK|" .env \
  || echo "CONTAINERD_SOCK=$SOCK" >> .env
docker compose up -d cadvisor

Air-gapped host — load the image first. A ready archive is published on the hub, together with a step-by-step page for customers: https://hub.friendly-tech.com/docs/pdf/grafana-cadvisor-fix.html

curl -fLO https://hub.friendly-tech.com/docs/pdf/cadvisor-v0.55.1.tar.gz
sha256sum cadvisor-v0.55.1.tar.gz
# expect 3439641369dea7121fc3ed1fb2722e979421b0bab3c574d8115fd12d977aa204
# transfer, then on the target host:
docker load -i cadvisor-v0.55.1.tar.gz

Or build it yourself on any machine with internet access:

docker pull gcr.io/cadvisor/cadvisor:v0.55.1
docker save gcr.io/cadvisor/cadvisor:v0.55.1 -o cadvisor-v0.55.1.tar
tar tf cadvisor-v0.55.1.tar | grep -m1 manifest.json   # classic format, loads anywhere

Verify:

docker logs --since 3m cadvisor 2>&1 | grep -c 'read-write layer ID'       # expect 0
curl -s 127.0.0.1:9183/metrics | grep -o 'name="[^"]*"' | sort -u | wc -l  # expect your container count

Last resort — switch Docker back to the classic store. Only for hosts that must keep an old cAdvisor build. Graph drivers are deprecated in Docker 29, this restarts Docker, and images already written to the containerd store disappear from docker images afterwards (they are not deleted — reverting daemon.json brings them back):

# add both keys to /etc/docker/daemon.json, keeping any existing settings:
#   {"features": {"containerd-snapshotter": false}, "storage-driver": "overlay2"}
systemctl restart docker
docker info | grep 'Storage Driver'   # should show overlay2
# air-gapped: re-load images before starting anything
docker load -i docker-images-bundle.tar.gz
cd /opt/grafana/exporters/base && docker compose up -d

On a fresh Docker 29 install this may not take effect at all — the daemon keeps the containerd store while its state exists, and a real switch means wiping /var/lib/docker. Upgrade cAdvisor instead.

Note: Bootstrap detects the containerd image store on every run and compares it with the deployed cAdvisor version: with v0.54.0+ it just logs that container metrics are supported; with an older build it prints an ACTION REQUIRED block in the final summary. diagnose.sh performs the same check. The --fix-storage-driver flag performs the last-resort switch above (works in non-interactive runs too), and is no longer offered interactively.

Grafana alerts not sending emails

Solution:

docker logs grafana | grep -i smtp
# Test in UI: Alerting > Contact points > email > Test

Alertmanager Config UI: “Failed to update .env file”

The UI runs as uid 1000 while bootstrap.sh runs as root, so the files it edits must be group-owned by the host docker group — the same group that lets it restart containers. Re-running bootstrap.sh stack sets this up (DOCKER_GID in .env, plus 0660 on .env and 0664 on the templates and prometheus_rules.yml, which other containers still have to read).

Check what the container actually sees:

cd /opt/grafana/prometheus-grafana-stack
docker compose exec alertmanager-config-ui sh -c 'id; ls -l /app/.env'
# root:root 0644 -> the grant is missing; re-run bootstrap, or:
chgrp "$(getent group docker | cut -d: -f3)" .env && chmod 0660 .env

A related symptom is “Configuration saved” with nothing changing: the write succeeded but the restart that applies it did not, because neither Alertmanager nor snmp-notifier re-reads .env while running. The UI then names the service that is still on the old settings. Applying it by hand:

docker compose up -d alertmanager snmp-notifier

If the UI keeps behaving like an older build after an upgrade, compare the image against the source — it is built from this repo, not pulled (see prometheus-grafana-stack/alertmanager-config-ui/README.md).