Reference: Operations

Table of Contents

Version 1.7.0 | Updated: October 06, 2026

Reference: Operations

Operations

Updating Monitoring Stack

curl -fsSL https://hub.friendly-tech.com/bootstrap/install.sh | sudo bash -s -- stack

Select “Keep current configuration” to preserve your .env settings.

With --branch, an existing /opt/grafana is first moved to the latest commit of that branch — from another branch, or from an older commit of the same one — and only then deployed; the log line Updated <old branch> @ <commit> → <branch> @ <commit> (VERSION …​) shows what was installed. Local edits to tracked files are stashed and archived to /root/stash-archive/ first (.env and servers.env are not touched). If the branch cannot be fetched, nothing is deployed. Without --branch, the checkout on disk is deployed exactly as it is.

For exporters:

cd /opt/grafana
sudo bash /opt/grafana/misc/quick-deploy.sh PROMETHEUS_IP

Just refresh the code, deploy nothing — --update pulls the requested branch into /opt/grafana and stops. It needs no PROMETHEUS_IP, installs nothing, and restarts no container, so it is safe on a server whose stack is running:

curl -fsSL https://hub.friendly-tech.com/bootstrap/install.sh | sudo bash -s -- --update

It prints the commit it moved from and to. Use it to pick up a fix and apply it deliberately afterwards (bootstrap.sh stack, quick-deploy.sh, or a single docker compose up -d).

Why not plain git pull? The repository is private and servers are installed with an HTTPS remote and no stored credentials — git pull there fails with could not read Username for 'https://github.com'. The hub wrapper carries the token, hands it to bootstrap, and the token is stripped from the remote again right after the pull. An extracted offline copy has no history at all: --update says so instead of reporting success, and such a host is refreshed by delivering a new archive.

Updating Configuration

vi .env
docker compose up -d
docker compose restart nginx  # if nginx config changed

Updating Docker Images

vi .env  # Set the version, e.g. PROMETHEUS_VERSION=v3.13.1
docker compose pull prometheus
docker compose up -d prometheus

The value in .env is what actually runs. The ${VAR:-default} in docker-compose.yml is a fallback that must not be relied on — if .env has no entry for an image, the compose default silently decides the version, which is how a host ended up running Prometheus v3.5.0 while every document described v3.13.1. Compare against .env.example after any upgrade.

Rollback Deployment

The deployment script creates a manifest tracking all changes. To rollback:

# Rollback to previous state (latest manifest)
curl -fsSL https://hub.friendly-tech.com/bootstrap/install.sh | sudo bash -s -- --rollback

# Or directly:
sudo /opt/grafana/misc/quick-deploy.sh --rollback

# List available manifests
sudo /opt/grafana/misc/quick-deploy.sh --list-manifests

# Rollback specific deployment
sudo /opt/grafana/misc/quick-deploy.sh --rollback /opt/grafana/.deploy-manifests/manifest-20260201-143022

What gets rolled back:

  • Stops deployed containers

  • Removes added firewall rules

  • Restores original configuration files (WildFly, Hazelcast)

Note: Services like WildFly/ACS may need manual restart after rollback.

Offline/Air-gapped Installation

Docker must already be installed on the air-gapped server (install-docker.sh needs internet).

Prepare (machine with internet) — use the 1.6.9 branch ZIP (not main), and build on a host with the classic Docker image store (Docker Desktop / Engine ≥ 28 use the containerd store → docker save writes OCI → old Docker on the target fails with unrecognized image format):

bash misc/download-docker-images.sh   # linux/amd64 by default; includes built stack images + Grafana plugins
tar tf docker-images-bundle.tar* | grep -m1 manifest.json && echo CLASSIC_OK   # NOT oci-layout
scp grafana-1.6.9.tar.gz docker-images-bundle.tar.gz user@server:/tmp/
scp -r docker-images-bundle-grafana-plugins user@server:/tmp/

Install exporters (air-gapped server):

sudo docker load -i /tmp/docker-images-bundle.tar.gz
cd /opt && sudo tar xzf /tmp/grafana-1.6.9.tar.gz && sudo mv grafana-1.6.9 grafana
sudo bash /opt/grafana/misc/quick-deploy.sh PROMETHEUS_IP

Install monitoring stack (air-gapped server): load the bundle and the repository as above, then run bootstrap. Run it plain for the interactive configurator, or add -y to skip every prompt:

sudo docker load -i /tmp/docker-images-bundle.tar.gz
sudo bash /opt/grafana/misc/bootstrap.sh stack       # interactive: prompts for domain, passwords, SMTP
sudo bash /opt/grafana/misc/bootstrap.sh stack -y    # non-interactive: no questions; auto IP-as-domain + random passwords

With Docker and the repo already present, bootstrap runs fully offline — self-signed cert, htpasswd, alertmanager placeholder, containers from pre-loaded images, base exporters, targets — and auto-stages the bundled Grafana plugins if docker-images-bundle-grafana-plugins/ is next to /opt/grafana or in /tmp. No manual section-8 steps needed (see the Quick Start offline section for details and the Oracle datasource plugin).

Every image is pulled; nothing is built on the target host. snmp-catcher and alertmanager-config-ui are built by CI and published to the private monitoring project, so an online install needs one docker login first (see Container images). Bootstrap checks both are present or pullable before starting the stack and stops with that instruction if not — an image already on the host, loaded from an offline bundle, is used as-is and never re-pulled. Versions are pinned in .env.

Security Best Practices

  • Never commit .env files to git

  • Use strong passwords (16+ characters)

  • Restrict permissions: chmod 600 .env

  • Rotate passwords every 90 days

  • Use dedicated users with minimal grants

Deployment Checklist

Before Starting:

  • Docker installed, repository cloned to /opt/grafana

  • Prometheus IP address known

  • Credentials gathered (MySQL/Oracle/SMTP)

For Each Server:

  • .env configured, containers deployed

  • Metrics endpoints working (curl localhost:PORT/metrics)

  • Firewall rules saved

On Prometheus/Grafana:

  • Targets show “UP” in Prometheus UI

  • Dashboards loading, alerts configured

  • Nginx proxy working (/grafana-ro/)

branch release/1.7.0 · commit 582e727b3700ec06c4c6e544e853886fb8d272b9 · page generated October 06, 2026