Runbook: Prepare and Start

Version 1.6.9-option-3 | Updated: August 07, 2026

Runbook: Prepare and Start

Test Environment (Local Stack)

To reproduce a complete environment for testing — the FTACS platform and the monitoring on one machine, from a host with nothing installed — follow the Local Stack guide. It covers the whole path end to end: fetching the code, starting the platform, deploying the exporters against it, the monitoring stack, what a healthy result looks like, and how to read the failures that actually occur.

See the Local Stack — Test Environment Guide.

Note that local-stack deploys the platform itself, so it does not belong on a server that already runs FTACS — there, deploy only the exporters as described below.

Quick Start

Setting up a test environment rather than monitoring real servers? Use the Local Stack — Test Environment Guide: it brings up the FTACS platform and the monitoring together on one machine, from scratch.

ONE-COMMAND INSTALL (Recommended):

For monitored servers (exporters):

# Auto-detect services and deploy appropriate exporters:
curl -fsSL https://hub.friendly-tech.com/bootstrap/install.sh | sudo bash -s -- PROMETHEUS_IP

# Example:
curl -fsSL https://hub.friendly-tech.com/bootstrap/install.sh | sudo bash -s -- 65.108.67.167

# With custom instance prefix (default: DevOps):
curl -fsSL https://hub.friendly-tech.com/bootstrap/install.sh | sudo bash -s -- 65.108.67.167 --prefix Prod

Install specific version (dev/testing):

# Install a specific version (required for testing dev branches):
curl -fsSL https://hub.friendly-tech.com/bootstrap/install.sh | sudo bash -s -- --branch 1.6.9 PROMETHEUS_IP

# Example:
curl -fsSL https://hub.friendly-tech.com/bootstrap/install.sh | sudo bash -s -- --branch 1.6.9 65.108.67.167

For Prometheus/Grafana server (monitoring stack):

# Deploy Prometheus + Grafana + Nginx proxy with interactive configuration:
curl -fsSL https://hub.friendly-tech.com/bootstrap/install.sh | sudo bash -s -- stack

# Stack with specific version:
curl -fsSL https://hub.friendly-tech.com/bootstrap/install.sh | sudo bash -s -- stack --branch 1.6.9

Bootstrap options:

Option Description Default Example

--prefix NAME

Instance name prefix for servers.env output

DevOps

--prefix Prod → Prod-acs1

--token TOKEN

GitHub token for private repository access

-

--token ghp_xxxx

--branch VERSION

Use specific version branch or tag

main

--branch 1.6.4

--version-list, -l

Show all available versions (tags + branches)

-

--version-list

--rollback [MANIFEST]

Rollback deployment changes

latest

--rollback

--list-manifests

List available rollback manifests

-

--list-manifests

--diagnose

Collect system diagnostics into a report file

-

--diagnose

--reset-password [PW]

Reset Grafana admin password

-

--reset-password "newpass"

--reset-prometheus-password [PW]

Reset Prometheus basic auth password

-

--reset-prometheus-password "newpass"

--version, -v

Show bootstrap version

-

--version

--docs, --readme

Show documentation links

-

--docs

--local-stack

Configure exporters from a running local-stack without asking

-

bootstrap.sh <IP> auto --local-stack -y

--no-local-stack

Ignore a running local-stack entirely

-

--no-local-stack

--update-only

Alias of --update

-

--update-only

--debug

Verbose tracing for troubleshooting the installer itself

-

--debug

Examples:

# List all available versions (releases and dev branches)
curl -fsSL https://hub.friendly-tech.com/bootstrap/install.sh | bash -s -- --version-list

# Show documentation links
curl -fsSL https://hub.friendly-tech.com/bootstrap/install.sh | bash -s -- --docs

# Reset Grafana admin password (if forgotten or DB persisted from previous install)
curl -fsSL https://hub.friendly-tech.com/bootstrap/install.sh | sudo bash -s -- --branch 1.6.9 --reset-password "`new_password`"

# Reset Prometheus basic auth password
curl -fsSL https://hub.friendly-tech.com/bootstrap/install.sh | sudo bash -s -- --branch 1.6.9 --reset-prometheus-password "`new_password`"

# Install specific version
curl -fsSL https://hub.friendly-tech.com/bootstrap/install.sh | sudo bash -s -- stack --branch 1.6.4

On the interactive path (no -y), bootstrap.sh stack opens a full-screen arrow-key configuration menu — a hub with sections for Server (IP
access domain), Prometheus (port, retention, basic-auth), Grafana (port, admin login, shared SMTP), FTACS Database (MySQL/Oracle for the Business dashboard), SSL / HTTPS (self-signed / Let’s Encrypt / existing), Alerting (drill into Email/Telegram/Slack/Teams/Webhook/SMS/SNMP), and UI Portal integration, ending in Review & apply.

  • ↑↓ + Enter to navigate; Esc goes back; each section/field shows a help panel and passwords are masked. Nothing is written until you choose Apply.

  • Add -y for a non-interactive run (no menu — auto-derives the domain from the server IP and generates random passwords).

What bootstrap does (exporters):

  1. Installs Docker and Docker Compose (if not present)

  2. Clones this repository to /opt/grafana (or updates existing)

  3. Stashes local changes before update (with restore instructions)

  4. Auto-detects running services (Hazelcast, WildFly, MySQL, Oracle, etc.)

  5. Shows deployment summary with 20-second countdown

  6. Prompts for database credentials interactively (MySQL/Oracle)

  7. Deploys appropriate exporters based on detected services

  8. Configures firewall rules for Prometheus access

Auto-detected services:

Service Exporter Directory Ports

WildFly/FTACS HikariCP

jmx-wildfly

5556

WildFly/FTACS JVM (bare-metal only)

jmx-wildfly (profile jvm)

5557

Hazelcast

(native prometheus)

9101

MySQL/MariaDB

mysql

9104

Oracle DB

oracle

9161

ClickHouse

clickhouse

9363

PostgreSQL

postgres

9187

Nginx (Angular UI)

nginx

9113

FT UI Services

(native endpoints)

8383, 8084, 8881

FT APIs

(native endpoints)

8091, 9880, 8085

(always)

base

9100, 9256, 9183

Base exporters include:

  • node-exporter (9100) - system metrics

  • process-exporter (9256) - host process metrics

  • cadvisor (9183) - Docker container metrics

FT UI Services (native Spring Boot Actuator / FastAPI metrics):

  • ft-device-network (8383) - WiFi mesh management service

  • ai-agent (8084) - AI assistant (Python/FastAPI)

  • ui-backend (8881) - Angular UI backend (when actuator enabled)

FT API Services (native Spring Boot Actuator metrics):

  • provision-api (8091) - Provision Portal API

  • northbound-api (9880) - Northbound integration API

  • service-api (8085) - Service management API

Note: Bootstrap does not support Windows. For Windows servers, install Windows Exporter manually (see section 7).

Manual/Offline Installation (air-gapped servers)

For servers without internet access. Requires pre-downloaded files.

Prerequisite: Docker and Docker Compose must already be installed on the air-gapped server — /opt/grafana/misc/install-docker.sh needs internet access (OS package repositories). Use your OS vendor’s offline packages if Docker is missing.

  1. On a machine with internet: download the 1.6.9 version branch as ZIP — https://github.com/Friendly-Technologies/grafana/archive/refs/heads/1.6.9.zip (extracts as grafana-1.6.9). ⚠️ Do NOT use the default main branch — main only receives release merges and lacks the current fixes.

  2. Download all Docker images into a bundle: bash misc/download-docker-images.sh

    • ⚠️ Build the bundle on a host whose Docker uses the classic image store. Docker Desktop and Docker Engine ≥ 28 default to the containerd image store, whose docker save writes an OCI-format archive that older Docker on the target rejects with unrecognized image format on docker load. Use an x86_64 Linux host with the classic (overlay2) store, or disable containerd: /etc/docker/daemon.json → {"features":{"containerd-snapshotter":false}} then systemctl restart docker and rebuild.

    • Verify the archive is classic before shipping (must contain manifest.json, not oci-layout): tar tf docker-images-bundle.tar* | grep -m1 manifest.json && echo CLASSIC_OK

    • Images are pulled for linux/amd64 by default (override: PLATFORM=linux/arm64 bash misc/download-docker-images.sh)

    • The bundle also includes the locally-built stack images (snmp-catcher, alertmanager-config-ui)

    • Grafana plugin packages are downloaded into docker-images-bundle-grafana-plugins/ (Grafana installs plugins from grafana.com at container start, which fails air-gapped)

    • Preview without downloading: bash misc/download-docker-images.sh --list-only

  3. Transfer files to the server: scp grafana.zip docker-images-bundle.tar.gz user@server:/tmp/ && scp -r docker-images-bundle-grafana-plugins user@server:/tmp/

  4. On the target server:

# Load Docker images
sudo docker load -i /tmp/docker-images-bundle.tar.gz

# Extract repository and deploy exporters
cd /opt && sudo unzip /tmp/grafana.zip && sudo mv grafana-* grafana
cd grafana && sudo bash /opt/grafana/misc/quick-deploy.sh PROMETHEUS_IP  # Auto-detects services

What this does:

  1. Loads pre-downloaded Docker images into the local Docker storage

  2. Deploys exporters from local images (no internet required)

  3. Configures firewall rules for Prometheus access

Monitoring stack node (Prometheus/Grafana) offline: once the image bundle is loaded (docker load), install the stack — no manual section-8 steps needed. Pick the mode:

# Interactive (recommended for a manual install) — runs the configurator and
# PROMPTS for GRAFANA_DOMAIN, admin/prometheus passwords, SMTP, etc.:
sudo bash /opt/grafana/misc/bootstrap.sh stack

# Non-interactive (for curl|bash / headless) — asks NOTHING. It auto-derives
# GRAFANA_DOMAIN from the server IP (self-signed cert) and generates random
# admin/prometheus passwords, printed in the final summary:
sudo bash /opt/grafana/misc/bootstrap.sh stack -y

Both use localhost for Prometheus. -y never ships the .env CHANGE_ME placeholders — the must-have values (IP/domain, passwords, service token) are filled automatically; only optional bits (SMTP, Business-dashboard DB) stay unset.

With Docker already present and the repo in place, bootstrap skips the internet-only steps (Docker install, git clone, Let’s Encrypt) and does the rest offline: creates .env from .env.example, generates a self-signed cert + htpasswd + the alertmanager placeholder, starts all containers from the pre-loaded images, deploys the base exporters, and generates targets. It also auto-stages the bundled Grafana plugins: it looks for a docker-images-bundle-grafana-plugins/ folder — with the plugin .zip files directly inside it, no subfolder — in /opt, /tmp and /opt/grafana, checked in that order, and copies its zips into grafana/data/offline-plugins/; the Grafana entrypoint then installs them from disk and skips grafana.com. Auto-staging only runs when offline-plugins/ is empty — if it already holds any .zip, bootstrap leaves it alone.

offline-plugins/ vs plugins/: offline-plugins/ is the input — the zips bootstrap (or you) stage — while plugins/ is the output the entrypoint unzips them into; copying zips into plugins/ does nothing. If you keep the plugin zips outside the three auto-staged locations, stage them manually first:

sudo mkdir -p /opt/grafana/prometheus-grafana-stack/grafana/data/offline-plugins
sudo cp /tmp/docker-images-bundle-grafana-plugins/*.zip \
     /opt/grafana/prometheus-grafana-stack/grafana/data/offline-plugins/
sudo chown -R 472:472 /opt/grafana/prometheus-grafana-stack/grafana/data

The chown matters — Grafana runs as uid 472 and must be able to write under grafana/data.

For Oracle deployments, also extract the Oracle datasource plugin (normally downloaded by bootstrap from GitHub):

sudo mkdir -p /opt/grafana/prometheus-grafana-stack/grafana/data/plugins/albertowd-oraclegrafana-datasource
sudo tar -xzf /tmp/docker-images-bundle-grafana-plugins/albertowd-oraclegrafana-datasource.tar.gz \
     -C /opt/grafana/prometheus-grafana-stack/grafana/data/plugins/albertowd-oraclegrafana-datasource

Tips:

  • TROUBLESHOOTING: Run sudo bash /opt/grafana/misc/diagnose.sh to automatically check Docker, containers, ports, firewall, and Prometheus targets

  • Environment Configuration: All deployments use environment variables (.env files) for secure and flexible configuration

Pre-Deployment Checklist

Before starting, gather the following information:

  • Prometheus Server IP - Will be used in firewall rules on all nodes

  • SMTP Credentials (optional) - For Grafana alerting (host, port, user, password)

  • MySQL Credentials - If deploying MySQL exporter (user, password)

  • Oracle Credentials - If deploying Oracle exporter (user, password)

  • Server IPs - All servers that will be monitored

For clarity in these instructions, we use the following servers:

  • acs1, acs2, hc1, hc2, db, iis

In this example, Prometheus is installed on the same server as the DB. You need to know its IP address during setup to whitelist it on the servers where the exporters are installed:

ACS1 - 65.109.58.164
ACS2 - 65.109.58.100
HC1  - 65.109.24.154
HC2  - 65.108.67.167
DB   - 65.109.58.165 <- PROMETHEUS_IP
IIS  - 65.109.49.150

1. Install Docker and Docker Compose on All Linux Servers

Target servers: acs1, acs2, hc1, hc2, db, *-api servers

Before installing exporters, ensure that Docker and Docker Compose are installed on all target servers (except Windows-based servers).

Automated Installation (Recommended)

Use the automated installation script:

# If repository is already cloned:
cd /opt/grafana
sudo bash /opt/grafana/misc/install-docker.sh

# Or via one-command bootstrap (installs Docker automatically):
curl -fsSL https://hub.friendly-tech.com/bootstrap/install.sh | sudo bash -s -- PROMETHEUS_IP

The script automatically:

  • ✅ Detects your OS and package manager

  • ✅ Fixes EOL repos (CentOS 7/8)

  • ✅ Installs Docker CE from official repository

  • ✅ Installs Docker Compose plugin (falls back to static binary if needed)

  • ✅ Enables and starts Docker service

  • ✅ Verifies installation

Verify Installation

docker -v
docker compose version

Expected output (versions may vary):

Docker version 28.0.1, build 068a01e
Docker Compose version v2.33.1

Manual Installation (if script fails)

Ubuntu/Debian:
sudo apt-get update
sudo apt-get install -y ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update && sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-compose-plugin
sudo systemctl enable --now docker
CentOS/Rocky/AlmaLinux:
sudo yum install -y yum-utils
sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
sudo yum install -y docker-ce docker-ce-cli containerd.io docker-compose-plugin
sudo systemctl enable --now docker