Runbook: Prepare and Start
Version 1.7.0 | Updated: October 06, 2026
Runbook: Prepare and Start
Test Environment (Local Stack)
To reproduce a complete environment for testing — the FTACS platform and the monitoring on one machine, from a host with nothing installed — follow the Local Stack guide. It covers the whole path end to end: fetching the code, starting the platform, deploying the exporters against it, the monitoring stack, what a healthy result looks like, and how to read the failures that actually occur.
See the Local Stack — Test Environment Guide.
Note that local-stack deploys the platform itself, so it does not belong on a server that already runs FTACS — there, deploy only the exporters as described below.
Quick Start
Setting up a test environment rather than monitoring real servers? Use the Local Stack — Test Environment Guide: it brings up the FTACS platform and the monitoring together on one machine, from scratch.
ONE-COMMAND INSTALL (Recommended):
No
--branchinstallsmain— the latest production release. That is the installer’s own default (misc/hub/install.sh,misc/bootstrap.sh), not just this page’s convention: every command below is safe to copy as-is. Pin to a specific supported line instead with--branch release/<major>.<minor>.x(e.g.release/1.7.x) — seedocs/RELEASE-WORKFLOW.mdfor which lines are currently maintained. A bare--branch mainalso works and says the same thing out loud, if you prefer to be explicit.
For monitored servers (exporters):
# Auto-detect services and deploy appropriate exporters:
curl -fsSL https://hub.friendly-tech.com/bootstrap/install.sh | sudo bash -s -- PROMETHEUS_IP
# Example:
curl -fsSL https://hub.friendly-tech.com/bootstrap/install.sh | sudo bash -s -- 65.108.67.167
# With custom instance prefix (default: DevOps):
curl -fsSL https://hub.friendly-tech.com/bootstrap/install.sh | sudo bash -s -- 65.108.67.167 --prefix Prod
For Prometheus/Grafana server (monitoring stack):
One-time on each stack host: two of the images are ours and live in a private project on the hub, so log in before installing. Everything else is public.
docker login hub.friendly-tech.com -u ast # read-only account, ask for the passwordSkip it and the install stops before starting anything, telling you to run exactly this. An air-gapped install needs no login at all — the images come in the bundle.
# Deploy Prometheus + Grafana + Nginx proxy with interactive configuration:
curl -fsSL https://hub.friendly-tech.com/bootstrap/install.sh | sudo bash -s -- stack
# Same, explicit about the branch (installs the same thing as above):
curl -fsSL https://hub.friendly-tech.com/bootstrap/install.sh | sudo bash -s -- stack --branch main
Bootstrap options:
| Option | Description | Default | Example |
|---|---|---|---|
|
Instance name prefix for servers.env output |
|
|
|
GitHub token for private repository access |
- |
|
|
Branch or tag to install: |
|
|
|
Show all available versions (tags + branches) |
- |
|
|
Rollback deployment changes |
latest |
|
|
List available rollback manifests |
- |
|
|
Collect system diagnostics into a report file |
- |
|
|
Reset Grafana admin password |
- |
|
|
Reset Prometheus basic auth password |
- |
|
|
Show bootstrap version |
- |
|
|
Show documentation links |
- |
|
|
Configure exporters from a running local-stack without asking |
- |
|
|
Ignore a running local-stack entirely |
- |
|
|
Alias of |
- |
|
|
Verbose tracing for troubleshooting the installer itself |
- |
|
Examples:
# List all available versions (releases and dev branches)
curl -fsSL https://hub.friendly-tech.com/bootstrap/install.sh | bash -s -- --version-list
# Show documentation links
curl -fsSL https://hub.friendly-tech.com/bootstrap/install.sh | bash -s -- --docs
# Reset Grafana admin password (if forgotten or DB persisted from previous install)
curl -fsSL https://hub.friendly-tech.com/bootstrap/install.sh | sudo bash -s -- --reset-password "`new_password`"
# Reset Prometheus basic auth password
curl -fsSL https://hub.friendly-tech.com/bootstrap/install.sh | sudo bash -s -- --reset-prometheus-password "`new_password`"
# Install specific version
curl -fsSL https://hub.friendly-tech.com/bootstrap/install.sh | sudo bash -s -- stack
On the interactive path (no -y), bootstrap.sh stack opens a full-screen
arrow-key configuration menu — a hub with sections for Server (IP
access domain), Prometheus (port, retention, basic-auth), Grafana
(port, admin login, shared SMTP), FTACS Database (MySQL/Oracle for the
Business dashboard), SSL / HTTPS (self-signed / Let’s Encrypt / existing),
Alerting (drill into Email/Telegram/Slack/Teams/Webhook/SMS/SNMP), and
UI Portal integration, ending in Review & apply.
-
↑↓ + Enter to navigate; Esc goes back; each section/field shows a help panel and passwords are masked. Nothing is written until you choose Apply.
-
Add
-yfor a non-interactive run (no menu — auto-derives the domain from the server IP and generates random passwords).
What bootstrap does (exporters):
-
Installs Docker and Docker Compose (if not present)
-
Clones this repository to
/opt/grafana(or updates existing) -
Stashes local changes before update (with restore instructions)
-
Auto-detects running services (Hazelcast, WildFly, MySQL, Oracle, etc.)
-
Shows deployment summary with 20-second countdown
-
Prompts for database credentials interactively (MySQL/Oracle)
-
Deploys appropriate exporters based on detected services
-
Configures firewall rules for Prometheus access
Auto-detected services:
| Service | Exporter Directory | Ports |
|---|---|---|
WildFly/FTACS HikariCP |
|
5556 |
WildFly/FTACS JVM (bare-metal only) |
|
5557 |
Hazelcast |
(native prometheus) |
9101 |
MySQL/MariaDB |
|
9104 |
Oracle DB |
|
9161 |
ClickHouse |
|
9363 |
PostgreSQL |
|
9187 |
Nginx (Angular UI) |
|
9113 |
FT UI Services |
(native endpoints) |
8383, 8084, 8881 |
FT APIs |
(native endpoints) |
8091, 9880, 8085 |
(always) |
|
9100, 9256, 9183 |
Base exporters include:
-
node-exporter(9100) - system metrics -
process-exporter(9256) - host process metrics -
cadvisor(9183) - Docker container metrics
FT UI Services (native Spring Boot Actuator / FastAPI metrics):
-
ft-device-network(8383) - WiFi mesh management service -
ai-agent(8084) - AI assistant (Python/FastAPI) -
ui-backend(8881) - Angular UI backend (when actuator enabled)
FT API Services (native Spring Boot Actuator metrics):
-
provision-api(8091) - Provision Portal API -
northbound-api(9880) - Northbound integration API -
service-api(8085) - Service management API
Note: Bootstrap does not support Windows. For Windows servers, install Windows Exporter manually (see section 7).
Manual/Offline Installation (air-gapped servers)
For servers without internet access. Requires pre-downloaded files.
No registry login is needed here. Everything, including the two images we build ourselves, comes from the bundle. If Docker asks for authorization during an offline install, the images are not on the host — see Troubleshooting → Docker asks for authorization, which lists the three reasons in order of likelihood.
Prerequisite: Docker and Docker Compose must already be installed on the air-gapped server —
/opt/grafana/misc/install-docker.shneeds internet access (OS package repositories). Use your OS vendor’s offline packages if Docker is missing.
-
On a machine with internet: download the
1.6.9version branch as a tarball from the hub —https://hub.friendly-tech.com/repo/1.6.9.tar.gz(extracts asgrafana-1.6.9). It asks for theastaccount, the same one used to pull our images. The hub rebuilds it on every push to the branch, so it is that branch’s head and not a snapshot. ⚠️ Do NOT take the defaultmainbranch —mainonly receives release merges and lacks the current fixes.A tarball rather than the ZIP on purpose: extracting it needs only
tar, which the target host must already have to load the image bundle.unzipis absent from a minimal Debian or RHEL install, and on an air-gapped host there is no way to add it — an install that gets that far and then cannot unpack itself is a poor way to find out. The ZIP is still served at…/1.6.9.zipif you prefer it and haveunzip. -
Enter the extracted directory, then download all Docker images into a bundle:
cd grafana-1.6.9 bash misc/download-docker-images.sh-
Both artifacts are written to the directory you run it from, not into the repository:
docker-images-bundle.tar.gzanddocker-images-bundle-grafana-plugins/. The script finds the repository by its own location, so it runs from anywhere — but running it from insidemisc/leaves the bundle inmisc/, and step 3 below then copies nothing. Run it from the extracted directory as shown. -
⚠️ Build the bundle on a host whose Docker uses the classic image store. Docker Desktop and Docker Engine ≥ 28 default to the containerd image store, whose
docker savewrites an OCI-format archive that older Docker on the target rejects withunrecognized image formatondocker load. Use an x86_64 Linux host with the classic (overlay2) store, or disable containerd:/etc/docker/daemon.json→{"features":{"containerd-snapshotter":false}}thensystemctl restart dockerand rebuild. -
Verify the archive is classic before shipping (must contain
manifest.json, notoci-layout):tar tf docker-images-bundle.tar* | grep -m1 manifest.json && echo CLASSIC_OK -
Images are pulled for
linux/amd64by default (override:PLATFORM=linux/arm64 bash misc/download-docker-images.sh) -
The bytes come from our own registry, not from six third-party ones. Every image the stack runs is mirrored to
hub.friendly-tech.com/mirror/…at the exact version this branch ships, and the bundle saves them under those mirror names — which is what the compose files ask the daemon for, because.env.examplepins each image there. Sodocker loadproduces exactly what a deployment references. If a version is missing from the mirror the script says so, names the workflow to re-run, and falls back to pulling it from upstream — re-tagged to the mirror name, so the bundle still matches compose — rather than failing quietly.--no-mirrorbuilds the list from the upstream names instead (for checking that an upstream still has what we mirrored; a bundle built that way is not what we ship and will not match a deployment’s.env) -
The bundle also includes the locally-built stack images (
snmp-catcher,alertmanager-config-ui) -
Grafana plugin packages are downloaded into
docker-images-bundle-grafana-plugins/(Grafana installs plugins from grafana.com at container start, which fails air-gapped) -
Preview without downloading:
bash misc/download-docker-images.sh --list-only
-
-
Transfer files to the server. Three things go across, and after step 2 you are inside
grafana-1.6.9while the source tarball is beside it:scp ../grafana-1.6.9.tar.gz docker-images-bundle.tar.gz user@server:/tmp/ scp -r docker-images-bundle-grafana-plugins user@server:/tmp/ -
On the target server:
# Load Docker images sudo docker load -i /tmp/docker-images-bundle.tar.gz # Extract repository and deploy exporters. # Name the directory explicitly rather than `mv grafana-* grafana`: our own # tooling leaves backups like /opt/grafana-backup-20260428-230021, the glob # then matches two directories and mv refuses with "target is not a # directory" — on exactly the hosts that have been installed before. cd /opt && sudo tar xzf /tmp/grafana-1.6.9.tar.gz && sudo mv grafana-1.6.9 grafana cd grafana && sudo bash /opt/grafana/misc/quick-deploy.sh PROMETHEUS_IP # Auto-detects services
What this does:
-
Loads pre-downloaded Docker images into the local Docker storage
-
Deploys exporters from local images (no internet required)
-
Configures firewall rules for Prometheus access
Monitoring stack node (Prometheus/Grafana) offline: once the image bundle is loaded (
docker load), install the stack — no manual section-8 steps needed. Pick the mode:# Interactive (recommended for a manual install) — runs the configurator and # PROMPTS for GRAFANA_DOMAIN, admin/prometheus passwords, SMTP, etc.: sudo bash /opt/grafana/misc/bootstrap.sh stack # Non-interactive (for curl|bash / headless) — asks NOTHING. It auto-derives # GRAFANA_DOMAIN from the server IP (self-signed cert) and generates random # admin/prometheus passwords, printed in the final summary: sudo bash /opt/grafana/misc/bootstrap.sh stack -yBoth use localhost for Prometheus.
-ynever ships the.envCHANGE_MEplaceholders — the must-have values (IP/domain, passwords, service token) are filled automatically; only optional bits (SMTP, Business-dashboard DB) stay unset.With Docker already present and the repo in place, bootstrap skips the internet-only steps (Docker install, git clone, Let’s Encrypt) and does the rest offline: creates
.envfrom.env.example, generates a self-signed cert + htpasswd + the alertmanager placeholder, starts all containers from the pre-loaded images, deploys the base exporters, and generates targets. It also auto-stages the bundled Grafana plugins: it looks for adocker-images-bundle-grafana-plugins/folder — with the plugin.zipfiles directly inside it, no subfolder — in/opt,/tmpand/opt/grafana, checked in that order, and copies its zips intografana/data/offline-plugins/; the Grafana entrypoint then installs them from disk and skips grafana.com. Auto-staging only runs whenoffline-plugins/is empty — if it already holds any.zip, bootstrap leaves it alone.
offline-plugins/vsplugins/:offline-plugins/is the input — the zips bootstrap (or you) stage — whileplugins/is the output the entrypoint unzips them into; copying zips intoplugins/does nothing. If you keep the plugin zips outside the three auto-staged locations, stage them manually first:sudo mkdir -p /opt/grafana/prometheus-grafana-stack/grafana/data/offline-plugins sudo cp /tmp/docker-images-bundle-grafana-plugins/*.zip \ /opt/grafana/prometheus-grafana-stack/grafana/data/offline-plugins/ sudo chown -R 472:472 /opt/grafana/prometheus-grafana-stack/grafana/dataThe
chownmatters — Grafana runs as uid472and must be able to write undergrafana/data.For Oracle deployments, also extract the Oracle datasource plugin (normally downloaded by bootstrap from GitHub):
sudo mkdir -p /opt/grafana/prometheus-grafana-stack/grafana/data/plugins/albertowd-oraclegrafana-datasource sudo tar -xzf /tmp/docker-images-bundle-grafana-plugins/albertowd-oraclegrafana-datasource.tar.gz \ -C /opt/grafana/prometheus-grafana-stack/grafana/data/plugins/albertowd-oraclegrafana-datasource
Tips:
-
TROUBLESHOOTING: Run
sudo bash /opt/grafana/misc/diagnose.shto automatically check Docker, containers, ports, firewall, and Prometheus targets -
A container that looks like it is leaking memory: run
sudo bash /opt/grafana/misc/diagnose-memory.sh. It samples one suspect container and every other one as a control, reads the cgroup and the process’s own counters, collects nginx’s configuration and logs, pulls 30 days of history from Prometheus, and prints a verdict before it packages anything. Read-only, needs no network, five minutes. -
Environment Configuration: All deployments use environment variables (
.envfiles) for secure and flexible configuration
Pre-Deployment Checklist
Before starting, gather the following information:
-
Prometheus Server IP - Will be used in firewall rules on all nodes
-
SMTP Credentials (optional) - For Grafana alerting (host, port, user, password)
-
MySQL Credentials - If deploying MySQL exporter (user, password)
-
Oracle Credentials - If deploying Oracle exporter (user, password)
-
Server IPs - All servers that will be monitored
For clarity in these instructions, we use the following servers:
-
acs1,acs2,hc1,hc2,db,iis
In this example, Prometheus is installed on the same server as the DB. You need to know its IP address during setup to whitelist it on the servers where the exporters are installed:
ACS1 - 65.109.58.164 ACS2 - 65.109.58.100 HC1 - 65.109.24.154 HC2 - 65.108.67.167 DB - 65.109.58.165 <- PROMETHEUS_IP IIS - 65.109.49.150
1. Install Docker and Docker Compose on All Linux Servers
Target servers: acs1, acs2, hc1, hc2, db, *-api servers
Before installing exporters, ensure that Docker and Docker Compose are installed on all target servers (except Windows-based servers).
Automated Installation (Recommended)
Use the automated installation script:
# If repository is already cloned:
cd /opt/grafana
sudo bash /opt/grafana/misc/install-docker.sh
# Or via one-command bootstrap (installs Docker automatically):
curl -fsSL https://hub.friendly-tech.com/bootstrap/install.sh | sudo bash -s -- PROMETHEUS_IP
The script automatically:
-
✅ Detects your OS and package manager
-
✅ Fixes EOL repos (CentOS 7/8)
-
✅ Installs Docker CE from official repository
-
✅ Installs Docker Compose plugin (falls back to static binary if needed)
-
✅ Enables and starts Docker service
-
✅ Verifies installation
Verify Installation
docker -v
docker compose version
Expected output (versions may vary):
Docker version 28.0.1, build 068a01e Docker Compose version v2.33.1
Manual Installation (if script fails)
Ubuntu/Debian:
sudo apt-get update
sudo apt-get install -y ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update && sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-compose-plugin
sudo systemctl enable --now docker
CentOS/Rocky/AlmaLinux:
sudo yum install -y yum-utils
sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
sudo yum install -y docker-ce docker-ce-cli containerd.io docker-compose-plugin
sudo systemctl enable --now docker
branch release/1.7.0 · commit 582e727b3700ec06c4c6e544e853886fb8d272b9 · page generated October 06, 2026