Runbook: Prepare and Start

Version 1.7.0 | Updated: October 06, 2026

Runbook: Prepare and Start

Test Environment (Local Stack)

To reproduce a complete environment for testing — the FTACS platform and the monitoring on one machine, from a host with nothing installed — follow the Local Stack guide. It covers the whole path end to end: fetching the code, starting the platform, deploying the exporters against it, the monitoring stack, what a healthy result looks like, and how to read the failures that actually occur.

Note that local-stack deploys the platform itself, so it does not belong on a server that already runs FTACS — there, deploy only the exporters as described below.

Quick Start

Setting up a test environment rather than monitoring real servers? Use the Local Stack — Test Environment Guide: it brings up the FTACS platform and the monitoring together on one machine, from scratch.

ONE-COMMAND INSTALL (Recommended):

No --branch installs main — the latest production release. That is the installer’s own default (misc/hub/install.sh, misc/bootstrap.sh), not just this page’s convention: every command below is safe to copy as-is. Pin to a specific supported line instead with --branch release/<major>.<minor>.x (e.g. release/1.7.x) — see docs/RELEASE-WORKFLOW.md for which lines are currently maintained. A bare --branch main also works and says the same thing out loud, if you prefer to be explicit.

For monitored servers (exporters):

# Auto-detect services and deploy appropriate exporters:
curl -fsSL https://hub.friendly-tech.com/bootstrap/install.sh | sudo bash -s -- PROMETHEUS_IP

# Example:
curl -fsSL https://hub.friendly-tech.com/bootstrap/install.sh | sudo bash -s -- 65.108.67.167

# With custom instance prefix (default: DevOps):
curl -fsSL https://hub.friendly-tech.com/bootstrap/install.sh | sudo bash -s -- 65.108.67.167 --prefix Prod

For Prometheus/Grafana server (monitoring stack):

One-time on each stack host: two of the images are ours and live in a private project on the hub, so log in before installing. Everything else is public.

docker login hub.friendly-tech.com -u ast   # read-only account, ask for the password

Skip it and the install stops before starting anything, telling you to run exactly this. An air-gapped install needs no login at all — the images come in the bundle.

# Deploy Prometheus + Grafana + Nginx proxy with interactive configuration:
curl -fsSL https://hub.friendly-tech.com/bootstrap/install.sh | sudo bash -s -- stack

# Same, explicit about the branch (installs the same thing as above):
curl -fsSL https://hub.friendly-tech.com/bootstrap/install.sh | sudo bash -s -- stack --branch main

Bootstrap options:

Option Description Default Example

--prefix NAME

Instance name prefix for servers.env output

DevOps

--prefix Prod → Prod-acs1

--token TOKEN

GitHub token for private repository access

-

--token ghp_xxxx

--branch REF

Branch or tag to install: main (latest release), release/X.Y.x (a supported LTS line), or a vX.Y.Z tag

main

--branch release/1.7.x

--version-list, -l

Show all available versions (tags + branches)

-

--version-list

--rollback [MANIFEST]

Rollback deployment changes

latest

--rollback

--list-manifests

List available rollback manifests

-

--list-manifests

--diagnose

Collect system diagnostics into a report file

-

--diagnose

--reset-password [PW]

Reset Grafana admin password

-

--reset-password "newpass"

--reset-prometheus-password [PW]

Reset Prometheus basic auth password

-

--reset-prometheus-password "newpass"

--version, -v

Show bootstrap version

-

--version

--docs, --readme

Show documentation links

-

--docs

--local-stack

Configure exporters from a running local-stack without asking

-

bootstrap.sh <IP> auto --local-stack -y

--no-local-stack

Ignore a running local-stack entirely

-

--no-local-stack

--update-only

Alias of --update

-

--update-only

--debug

Verbose tracing for troubleshooting the installer itself

-

--debug

Examples:

# List all available versions (releases and dev branches)
curl -fsSL https://hub.friendly-tech.com/bootstrap/install.sh | bash -s -- --version-list

# Show documentation links
curl -fsSL https://hub.friendly-tech.com/bootstrap/install.sh | bash -s -- --docs

# Reset Grafana admin password (if forgotten or DB persisted from previous install)
curl -fsSL https://hub.friendly-tech.com/bootstrap/install.sh | sudo bash -s -- --reset-password "`new_password`"

# Reset Prometheus basic auth password
curl -fsSL https://hub.friendly-tech.com/bootstrap/install.sh | sudo bash -s -- --reset-prometheus-password "`new_password`"

# Install specific version
curl -fsSL https://hub.friendly-tech.com/bootstrap/install.sh | sudo bash -s -- stack

On the interactive path (no -y), bootstrap.sh stack opens a full-screen arrow-key configuration menu — a hub with sections for Server (IP
access domain), Prometheus (port, retention, basic-auth), Grafana (port, admin login, shared SMTP), FTACS Database (MySQL/Oracle for the Business dashboard), SSL / HTTPS (self-signed / Let’s Encrypt / existing), Alerting (drill into Email/Telegram/Slack/Teams/Webhook/SMS/SNMP), and UI Portal integration, ending in Review & apply.

  • ↑↓ + Enter to navigate; Esc goes back; each section/field shows a help panel and passwords are masked. Nothing is written until you choose Apply.

  • Add -y for a non-interactive run (no menu — auto-derives the domain from the server IP and generates random passwords).

What bootstrap does (exporters):

  1. Installs Docker and Docker Compose (if not present)

  2. Clones this repository to /opt/grafana (or updates existing)

  3. Stashes local changes before update (with restore instructions)

  4. Auto-detects running services (Hazelcast, WildFly, MySQL, Oracle, etc.)

  5. Shows deployment summary with 20-second countdown

  6. Prompts for database credentials interactively (MySQL/Oracle)

  7. Deploys appropriate exporters based on detected services

  8. Configures firewall rules for Prometheus access

Auto-detected services:

Service Exporter Directory Ports

WildFly/FTACS HikariCP

jmx-wildfly

5556

WildFly/FTACS JVM (bare-metal only)

jmx-wildfly (profile jvm)

5557

Hazelcast

(native prometheus)

9101

MySQL/MariaDB

mysql

9104

Oracle DB

oracle

9161

ClickHouse

clickhouse

9363

PostgreSQL

postgres

9187

Nginx (Angular UI)

nginx

9113

FT UI Services

(native endpoints)

8383, 8084, 8881

FT APIs

(native endpoints)

8091, 9880, 8085

(always)

base

9100, 9256, 9183

Base exporters include:

  • node-exporter (9100) - system metrics

  • process-exporter (9256) - host process metrics

  • cadvisor (9183) - Docker container metrics

FT UI Services (native Spring Boot Actuator / FastAPI metrics):

  • ft-device-network (8383) - WiFi mesh management service

  • ai-agent (8084) - AI assistant (Python/FastAPI)

  • ui-backend (8881) - Angular UI backend (when actuator enabled)

FT API Services (native Spring Boot Actuator metrics):

  • provision-api (8091) - Provision Portal API

  • northbound-api (9880) - Northbound integration API

  • service-api (8085) - Service management API

Note: Bootstrap does not support Windows. For Windows servers, install Windows Exporter manually (see section 7).

Manual/Offline Installation (air-gapped servers)

For servers without internet access. Requires pre-downloaded files.

No registry login is needed here. Everything, including the two images we build ourselves, comes from the bundle. If Docker asks for authorization during an offline install, the images are not on the host — see Troubleshooting → Docker asks for authorization, which lists the three reasons in order of likelihood.

Prerequisite: Docker and Docker Compose must already be installed on the air-gapped server — /opt/grafana/misc/install-docker.sh needs internet access (OS package repositories). Use your OS vendor’s offline packages if Docker is missing.

  1. On a machine with internet: download the 1.6.9 version branch as a tarball from the hub — https://hub.friendly-tech.com/repo/1.6.9.tar.gz (extracts as grafana-1.6.9). It asks for the ast account, the same one used to pull our images. The hub rebuilds it on every push to the branch, so it is that branch’s head and not a snapshot. ⚠️ Do NOT take the default main branch — main only receives release merges and lacks the current fixes.

    A tarball rather than the ZIP on purpose: extracting it needs only tar, which the target host must already have to load the image bundle. unzip is absent from a minimal Debian or RHEL install, and on an air-gapped host there is no way to add it — an install that gets that far and then cannot unpack itself is a poor way to find out. The ZIP is still served at …/1.6.9.zip if you prefer it and have unzip.

  2. Enter the extracted directory, then download all Docker images into a bundle:

    cd grafana-1.6.9
    bash misc/download-docker-images.sh
    • Both artifacts are written to the directory you run it from, not into the repository: docker-images-bundle.tar.gz and docker-images-bundle-grafana-plugins/. The script finds the repository by its own location, so it runs from anywhere — but running it from inside misc/ leaves the bundle in misc/, and step 3 below then copies nothing. Run it from the extracted directory as shown.

    • ⚠️ Build the bundle on a host whose Docker uses the classic image store. Docker Desktop and Docker Engine ≥ 28 default to the containerd image store, whose docker save writes an OCI-format archive that older Docker on the target rejects with unrecognized image format on docker load. Use an x86_64 Linux host with the classic (overlay2) store, or disable containerd: /etc/docker/daemon.json → {"features":{"containerd-snapshotter":false}} then systemctl restart docker and rebuild.

    • Verify the archive is classic before shipping (must contain manifest.json, not oci-layout): tar tf docker-images-bundle.tar* | grep -m1 manifest.json && echo CLASSIC_OK

    • Images are pulled for linux/amd64 by default (override: PLATFORM=linux/arm64 bash misc/download-docker-images.sh)

    • The bytes come from our own registry, not from six third-party ones. Every image the stack runs is mirrored to hub.friendly-tech.com/mirror/… at the exact version this branch ships, and the bundle saves them under those mirror names — which is what the compose files ask the daemon for, because .env.example pins each image there. So docker load produces exactly what a deployment references. If a version is missing from the mirror the script says so, names the workflow to re-run, and falls back to pulling it from upstream — re-tagged to the mirror name, so the bundle still matches compose — rather than failing quietly. --no-mirror builds the list from the upstream names instead (for checking that an upstream still has what we mirrored; a bundle built that way is not what we ship and will not match a deployment’s .env)

    • The bundle also includes the locally-built stack images (snmp-catcher, alertmanager-config-ui)

    • Grafana plugin packages are downloaded into docker-images-bundle-grafana-plugins/ (Grafana installs plugins from grafana.com at container start, which fails air-gapped)

    • Preview without downloading: bash misc/download-docker-images.sh --list-only

  3. Transfer files to the server. Three things go across, and after step 2 you are inside grafana-1.6.9 while the source tarball is beside it:

    scp ../grafana-1.6.9.tar.gz docker-images-bundle.tar.gz user@server:/tmp/
    scp -r docker-images-bundle-grafana-plugins user@server:/tmp/
  4. On the target server:

    # Load Docker images
    sudo docker load -i /tmp/docker-images-bundle.tar.gz
    
    # Extract repository and deploy exporters.
    # Name the directory explicitly rather than `mv grafana-* grafana`: our own
    # tooling leaves backups like /opt/grafana-backup-20260428-230021, the glob
    # then matches two directories and mv refuses with "target is not a
    # directory" — on exactly the hosts that have been installed before.
    cd /opt && sudo tar xzf /tmp/grafana-1.6.9.tar.gz && sudo mv grafana-1.6.9 grafana
    cd grafana && sudo bash /opt/grafana/misc/quick-deploy.sh PROMETHEUS_IP  # Auto-detects services

What this does:

  1. Loads pre-downloaded Docker images into the local Docker storage

  2. Deploys exporters from local images (no internet required)

  3. Configures firewall rules for Prometheus access

Monitoring stack node (Prometheus/Grafana) offline: once the image bundle is loaded (docker load), install the stack — no manual section-8 steps needed. Pick the mode:

# Interactive (recommended for a manual install) — runs the configurator and
# PROMPTS for GRAFANA_DOMAIN, admin/prometheus passwords, SMTP, etc.:
sudo bash /opt/grafana/misc/bootstrap.sh stack

# Non-interactive (for curl|bash / headless) — asks NOTHING. It auto-derives
# GRAFANA_DOMAIN from the server IP (self-signed cert) and generates random
# admin/prometheus passwords, printed in the final summary:
sudo bash /opt/grafana/misc/bootstrap.sh stack -y

Both use localhost for Prometheus. -y never ships the .env CHANGE_ME placeholders — the must-have values (IP/domain, passwords, service token) are filled automatically; only optional bits (SMTP, Business-dashboard DB) stay unset.

With Docker already present and the repo in place, bootstrap skips the internet-only steps (Docker install, git clone, Let’s Encrypt) and does the rest offline: creates .env from .env.example, generates a self-signed cert + htpasswd + the alertmanager placeholder, starts all containers from the pre-loaded images, deploys the base exporters, and generates targets. It also auto-stages the bundled Grafana plugins: it looks for a docker-images-bundle-grafana-plugins/ folder — with the plugin .zip files directly inside it, no subfolder — in /opt, /tmp and /opt/grafana, checked in that order, and copies its zips into grafana/data/offline-plugins/; the Grafana entrypoint then installs them from disk and skips grafana.com. Auto-staging only runs when offline-plugins/ is empty — if it already holds any .zip, bootstrap leaves it alone.

offline-plugins/ vs plugins/: offline-plugins/ is the input — the zips bootstrap (or you) stage — while plugins/ is the output the entrypoint unzips them into; copying zips into plugins/ does nothing. If you keep the plugin zips outside the three auto-staged locations, stage them manually first:

sudo mkdir -p /opt/grafana/prometheus-grafana-stack/grafana/data/offline-plugins
sudo cp /tmp/docker-images-bundle-grafana-plugins/*.zip \
     /opt/grafana/prometheus-grafana-stack/grafana/data/offline-plugins/
sudo chown -R 472:472 /opt/grafana/prometheus-grafana-stack/grafana/data

The chown matters — Grafana runs as uid 472 and must be able to write under grafana/data.

For Oracle deployments, also extract the Oracle datasource plugin (normally downloaded by bootstrap from GitHub):

sudo mkdir -p /opt/grafana/prometheus-grafana-stack/grafana/data/plugins/albertowd-oraclegrafana-datasource
sudo tar -xzf /tmp/docker-images-bundle-grafana-plugins/albertowd-oraclegrafana-datasource.tar.gz \
     -C /opt/grafana/prometheus-grafana-stack/grafana/data/plugins/albertowd-oraclegrafana-datasource

Tips:

  • TROUBLESHOOTING: Run sudo bash /opt/grafana/misc/diagnose.sh to automatically check Docker, containers, ports, firewall, and Prometheus targets

  • A container that looks like it is leaking memory: run sudo bash /opt/grafana/misc/diagnose-memory.sh. It samples one suspect container and every other one as a control, reads the cgroup and the process’s own counters, collects nginx’s configuration and logs, pulls 30 days of history from Prometheus, and prints a verdict before it packages anything. Read-only, needs no network, five minutes.

  • Environment Configuration: All deployments use environment variables (.env files) for secure and flexible configuration

Pre-Deployment Checklist

Before starting, gather the following information:

  • Prometheus Server IP - Will be used in firewall rules on all nodes

  • SMTP Credentials (optional) - For Grafana alerting (host, port, user, password)

  • MySQL Credentials - If deploying MySQL exporter (user, password)

  • Oracle Credentials - If deploying Oracle exporter (user, password)

  • Server IPs - All servers that will be monitored

For clarity in these instructions, we use the following servers:

  • acs1, acs2, hc1, hc2, db, iis

In this example, Prometheus is installed on the same server as the DB. You need to know its IP address during setup to whitelist it on the servers where the exporters are installed:

ACS1 - 65.109.58.164
ACS2 - 65.109.58.100
HC1  - 65.109.24.154
HC2  - 65.108.67.167
DB   - 65.109.58.165 <- PROMETHEUS_IP
IIS  - 65.109.49.150

1. Install Docker and Docker Compose on All Linux Servers

Target servers: acs1, acs2, hc1, hc2, db, *-api servers

Before installing exporters, ensure that Docker and Docker Compose are installed on all target servers (except Windows-based servers).

Use the automated installation script:

# If repository is already cloned:
cd /opt/grafana
sudo bash /opt/grafana/misc/install-docker.sh

# Or via one-command bootstrap (installs Docker automatically):
curl -fsSL https://hub.friendly-tech.com/bootstrap/install.sh | sudo bash -s -- PROMETHEUS_IP

The script automatically:

  • ✅ Detects your OS and package manager

  • ✅ Fixes EOL repos (CentOS 7/8)

  • ✅ Installs Docker CE from official repository

  • ✅ Installs Docker Compose plugin (falls back to static binary if needed)

  • ✅ Enables and starts Docker service

  • ✅ Verifies installation

Verify Installation

docker -v
docker compose version

Expected output (versions may vary):

Docker version 28.0.1, build 068a01e
Docker Compose version v2.33.1

Manual Installation (if script fails)

Ubuntu/Debian:
sudo apt-get update
sudo apt-get install -y ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update && sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-compose-plugin
sudo systemctl enable --now docker
CentOS/Rocky/AlmaLinux:
sudo yum install -y yum-utils
sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
sudo yum install -y docker-ce docker-ce-cli containerd.io docker-compose-plugin
sudo systemctl enable --now docker


branch release/1.7.0 · commit 582e727b3700ec06c4c6e544e853886fb8d272b9 · page generated October 06, 2026