Operations

Table of Contents

Version 1.6.9-option-2 | Updated: August 07, 2026

Operations

Operations

Updating Monitoring Stack

curl -fsSL https://hub.friendly-tech.com/bootstrap/install.sh | sudo bash -s -- stack

Select “Keep current configuration” to preserve your .env settings.

For exporters:

cd /opt/grafana
sudo bash /opt/grafana/misc/quick-deploy.sh PROMETHEUS_IP

Just refresh the code, deploy nothing — --update pulls the requested branch into /opt/grafana and stops. It needs no PROMETHEUS_IP, installs nothing, and restarts no container, so it is safe on a server whose stack is running:

curl -fsSL https://hub.friendly-tech.com/bootstrap/install.sh | sudo bash -s -- --update --branch 1.6.9

It prints the commit it moved from and to. Use it to pick up a fix and apply it deliberately afterwards (bootstrap.sh stack, quick-deploy.sh, or a single docker compose up -d).

Why not plain git pull? The repository is private and servers are installed with an HTTPS remote and no stored credentials — git pull there fails with could not read Username for 'https://github.com'. The hub wrapper carries the token, hands it to bootstrap, and the token is stripped from the remote again right after the pull. An extracted offline copy has no history at all: --update says so instead of reporting success, and such a host is refreshed by delivering a new archive.

Updating Configuration

vi .env
docker compose up -d
docker compose restart nginx  # if nginx config changed

Updating Docker Images

vi .env  # Change version, e.g., PROMETHEUS_VERSION=v3.6.0
docker compose pull prometheus
docker compose up -d prometheus

Rollback Deployment

The deployment script creates a manifest tracking all changes. To rollback:

# Rollback to previous state (latest manifest)
curl -fsSL https://hub.friendly-tech.com/bootstrap/install.sh | sudo bash -s -- --rollback

# Or directly:
sudo /opt/grafana/misc/quick-deploy.sh --rollback

# List available manifests
sudo /opt/grafana/misc/quick-deploy.sh --list-manifests

# Rollback specific deployment
sudo /opt/grafana/misc/quick-deploy.sh --rollback /opt/grafana/.deploy-manifests/manifest-20260201-143022

What gets rolled back:

  • Stops deployed containers

  • Removes added firewall rules

  • Restores original configuration files (WildFly, Hazelcast)

Note: Services like WildFly/ACS may need manual restart after rollback.

Offline/Air-gapped Installation

Docker must already be installed on the air-gapped server (install-docker.sh needs internet).

Prepare (machine with internet) — use the 1.6.9 branch ZIP (not main), and build on a host with the classic Docker image store (Docker Desktop / Engine ≥ 28 use the containerd store → docker save writes OCI → old Docker on the target fails with unrecognized image format):

bash misc/download-docker-images.sh   # linux/amd64 by default; includes built stack images + Grafana plugins
tar tf docker-images-bundle.tar* | grep -m1 manifest.json && echo CLASSIC_OK   # NOT oci-layout
scp grafana.zip docker-images-bundle.tar.gz user@server:/tmp/
scp -r docker-images-bundle-grafana-plugins user@server:/tmp/

Install exporters (air-gapped server):

sudo docker load -i /tmp/docker-images-bundle.tar.gz
cd /opt && sudo unzip /tmp/grafana.zip && sudo mv grafana-* grafana
sudo bash /opt/grafana/misc/quick-deploy.sh PROMETHEUS_IP

Install monitoring stack (air-gapped server): load the bundle and the repository as above, then run bootstrap. Run it plain for the interactive configurator, or add -y to skip every prompt:

sudo docker load -i /tmp/docker-images-bundle.tar.gz
sudo bash /opt/grafana/misc/bootstrap.sh stack       # interactive: prompts for domain, passwords, SMTP
sudo bash /opt/grafana/misc/bootstrap.sh stack -y    # non-interactive: no questions; auto IP-as-domain + random passwords

With Docker and the repo already present, bootstrap runs fully offline — self-signed cert, htpasswd, alertmanager placeholder, containers from pre-loaded images, base exporters, targets — and auto-stages the bundled Grafana plugins if docker-images-bundle-grafana-plugins/ is next to /opt/grafana or in /tmp. No manual section-8 steps needed (see the Quick Start offline section for details and the Oracle datasource plugin).

Upgrading a stack: two images are built here, not pulled. snmp-catcher:local and alertmanager-config-ui:local are built from this repository. On an upgrade bootstrap rebuilds each one only if a file in its source directory is newer than the image, so a fix to their Python actually takes effect; if that build fails (no internet) it keeps the existing image and says so, since an air-gapped host already loaded a current image from the bundle. Everything else comes from a registry and is pinned in .env.

Security Best Practices

  • Never commit .env files to git

  • Use strong passwords (16+ characters)

  • Restrict permissions: chmod 600 .env

  • Rotate passwords every 90 days

  • Use dedicated users with minimal grants

Deployment Checklist

Before Starting:

  • Docker installed, repository cloned to /opt/grafana

  • Prometheus IP address known

  • Credentials gathered (MySQL/Oracle/SMTP)

For Each Server:

  • .env configured, containers deployed

  • Metrics endpoints working (curl localhost:PORT/metrics)

  • Firewall rules saved

On Prometheus/Grafana:

  • Targets show “UP” in Prometheus UI

  • Dashboards loading, alerts configured

  • Nginx proxy working (/grafana-ro/)