Local Stack — Guide

Version 1.6.9 | Updated: August 08, 2026

Local Stack — Guide

Local development environment for the FTACS platform. Spins up the platform with a single command — up to 19 containers, depending on the service group, then deploys monitoring (Prometheus, Grafana, Alertmanager) with exporters and 14 pre-configured dashboards.

Full Setup (Start to Finish)

This is the whole path on a machine with nothing installed, exactly as it was last verified end to end. Support can follow it as-is to reproduce a test environment without asking anyone.

What this is, and when NOT to use it

local-stack deploys the FTACS platform itself — MySQL, FTACS, Hazelcast, the APIs and the UI — in containers, and then the monitoring is pointed at it. That makes it a self-contained test bed, and it also means it does not belong on a server that already runs FTACS: the ports it needs are exactly the ones the real platform occupies. On such a server deploy only the exporters (bootstrap.sh <prometheus-ip> auto).

Prerequisites

  • Linux server with Docker and Docker Compose v2.20+

  • 16 GB RAM minimum (32 GB recommended for with-ai), 25 GB free disk

  • The metrics ports free — 9101, 9110, 8080, 8843, 9100, 5556. They are not auto-shifted: monitoring discovers them by number, so a moved port would mean a stand that looks healthy and is never scraped.

  • Nothing else to prepare: both repositories are fetched by the commands below, including qa-shared with its .env files.

Step 1 — Fetch the code

curl -fsSL https://hub.friendly-tech.com/bootstrap/install.sh | sudo bash -s -- --branch 1.6.9 --update

--update means get the code and deploy nothing. On a clean host it clones, on an existing checkout it pulls. The repository is private, so this wrapper is the only way to fetch it — it carries the access token.

Step 2 — Start the platform

Two ways to run it. Interactively, if you want to see and choose everything:

cd /opt/grafana/local-stack
bash local-stack.sh up

The wizard asks four things, each with a description of what it means:

  1. Database — MySQL (faster, lighter) or Oracle (heavier, for Oracle-specific testing).

  2. Service group — how much of the platform to start: minimal, api, full, with-ai, or custom to pick services one by one.

  3. Image tags — the pinned release build, or set the tag per component.

  4. Confirmation — the summary before anything starts.

Or non-interactively, with the same choices as flags:

cd /opt/grafana/local-stack
bash local-stack.sh up --db=mysql --group=with-ai -y

qa-shared is cloned automatically. On a clean host the first run takes roughly 10 minutes, nearly all of it pulling about 18 GB of images; once they are cached a start takes 2—​4 minutes on MySQL.

docker ps --format '{{.Names}}' | grep -c '^ls_'      # 19 for with-ai

What each group actually starts. Measured on the test stand, 06.08.2026, all eight combinations with cached images, every container healthy:

  • minimal — 9 containers. MySQL 3m37s, Oracle 6m46s

  • api — 13 containers. MySQL 2m27s

  • full — 17 containers. MySQL 3m30s, Oracle 2m22s

  • with-ai — 19 containers. MySQL 2m14s, Oracle 3m19s

The database does not change what starts: on the same group Oracle and MySQL give the same containers. It changes how long the first start takes — Oracle spends several minutes initialising before anything else can come up, which is most of the difference on minimal.

custom is not “only the services you name”. --services="ftacs northbound-api" started 12 containers: the database, Hazelcast, ClickHouse, the JDBC bridge and the rest of the infrastructure come along, because the named services do not run without them.

Customising what starts
Want to Do

Pick services one by one

--group=custom --services="ftacs northbound-api portals"

Use Oracle

--db=oracle

Re-run the wizard from scratch

--clean

See the full command reference

bash local-stack.sh help

Core services (database, Hazelcast, FTACS) are always included — the picker marks them (required) and they cannot be switched off, because nothing else starts without them.

Changing the version of an application

Every FT image is pinned to a known-good build in local-stack/image-tags.conf. To run a different build of one component:

bash local-stack.sh up --tag=ftacs=v7.1.2-b0.0.1

The flag is repeatable, so several components can be moved at once:

bash local-stack.sh up --tag=ftacs=v7.1.2-b0.0.1 --tag=ui-backend=master_0.0.37

The wizard offers the same under Image tags › customize: it lists the components this run will start, each prefilled with the tag it would get, and you edit only what you need.

Either way the choice is saved with the rest of the configuration and reused on the next up, so a stand keeps the versions you gave it. --clean forgets them and goes back to the pinned defaults.

To move the whole stand to a newer release for good, edit image-tags.conf — that is the file the defaults come from. See Image Tags for how the pinning works and why qa-shared keeps :latest.

Step 3 — Deploy the exporters

cd /opt/grafana
bash misc/bootstrap.sh 65.109.58.165 auto --local-stack -y

Replace the IP with the address of the Prometheus server (the same machine here).

--local-stack tells bootstrap that the platform it just detected is a stand: it reads the ports and credentials from the stand’s own configuration, provisions a dedicated exporter database user inside the container, and never touches the host’s native database. Without the flag it asks; under -y it takes the stand by default; --no-local-stack opts out.

There is no restart step. This guide used to tell you to restart the stand and deploy again, on the grounds that the Hazelcast exporter and the stand compete for port 9101 and that only the exporter serves isClusterSafe. Measured on the stand, 06.08.2026, none of that holds: no Hazelcast exporter is deployed here at all — quick-deploy skips it and says so (“Hazelcast already serves the full metric set on :9101”) — ls_hazelcast owns 9101 before and after a restart alike, and isClusterSafe is served from the moment the stand comes up, with no exporter involved. The restart cost a stand cycle per run and changed nothing.

Step 4 — Deploy the monitoring stack

The stack itself — Prometheus, Grafana, Alertmanager and the alert channels — is documented in the main installation guide; only what the stand needs is repeated here.

cd /opt/grafana
bash misc/bootstrap.sh 65.109.58.165 stack

Asks for the Grafana and Prometheus passwords, SMTP and the alert channels. Add -y to take defaults: passwords are then generated and printed at the end.

Step 5 — Verify

cd /opt/grafana && bash misc/bootstrap.sh 65.109.58.165 auto --local-stack -y 2>&1 | tail -25

A healthy environment reports 0 failed and looks like this:

  • JMX_HC — about 9900 metrics. Six means the exporter cannot reach JMX.

  • JMX_HIKARI — about 20. Six means the same.

  • MYSQL — 30000+. Around 23000 means it is scraping the host’s native database instead of the stand: the --local-stack step did not apply.

  • UI_BACKEND — about 400, on port 9881. Its actuator is on a management port of its own; the service port answers 500 and the debug port has no metrics.

Dashboards need a separate check — a target can be UP while the series a panel asks for does not exist, and the panel then reads “No data” with nothing reporting a problem:

cd /opt/grafana && bash misc/testing/check-dashboard-panels.sh --quiet

It runs every panel’s own query and lists the empty ones. Some emptiness is correct and should not be chased:

  • windows-exporter-iis-node — no Windows hosts on the machine

  • oracle — empty on a MySQL stand, and also on an Oracle stand: the Oracle exporter is not deployed for a stand at all (--local-stack covers MySQL and PostgreSQL only), so oracledb_up has no series either way

  • business-metrics — the platform starts with an empty database, and a gauge over zero rows produces no series. These panels fill once the stand has devices; confirm with select count(*) from cpe before suspecting anything

  • parts of api-metrics — the counters appear after the APIs are called

A surprising result is more likely a bad query rewrite than a broken panel: open the dashboard in Grafana before acting on it. The variables are substituted textually here, and a matcher written with single quotes (tag0='$map') once turned into an exact match on the literal .*, reporting a dozen healthy panels as empty.

Comparing against a reference installation is what separates “this stand lacks the source” from “this panel is broken for everyone” — run the same script on the monitoring host that watches real servers and diff the two.

Then check Prometheus and Grafana:

https://<IP>/prometheus/targets     # every target UP
https://<IP>/                       # Grafana; credentials printed by step 5

Credentials are also in /opt/grafana/prometheus-grafana-stack/.env (GRAFANA_ADMIN_PASSWORD, PROMETHEUS_USER, PROMETHEUS_PASSWORD).

Running the stand on Oracle

Same path, one flag:

bash local-stack.sh up --db=oracle --group=with-ai -y

Oracle is heavier — the database alone takes several minutes before anything else can start. Expect the same containers as on MySQL (19 for with-ai) and, with the monitoring deployed, 16 targets UP. Measured with cached images: minimal 6m46s against MySQL’s 3m37s, the gap being Oracle’s initialisation; once it is up the larger groups are no slower than MySQL.

Two things are specific to it:

  • The Oracle exporter is not deployed. Nothing configures it from the stand, so oracledb_up has no series and the oracle dashboard stays empty. The platform itself is fully monitored; only the database’s own metrics are missing.

  • Timezone. A containerised Oracle XE does not carry the full timezone region set, so a JDBC driver that sends a region name is refused. The stand passes -Doracle.jdbc.timezoneAsRegion=false to every FT service to avoid it — see the troubleshooting entry below for what it looks like when it bites.

Step 6 — Tearing it down

cd /opt/grafana/local-stack && bash local-stack.sh clean   # stand only, keeps the monitoring

To reset the machine completely, including images:

bash misc/free-stand.sh --host <this-stand>          # show what would go
bash misc/free-stand.sh --host <this-stand> --yes    # carry it out

--host must name the machine you are on — its short name, its FQDN or one of its addresses. That is the point of it: the commands below are the same ones that reset a test stand and destroy a monitoring host, and nothing in them says which is which. Without --yes the script only prints the plan.

A native (non-Docker) database on the host is untouched. The script records which database ports are served by a non-container process before it starts and fails if one of them stops answering, so “the wipe took the host’s MySQL with it” cannot pass unnoticed.

What it runs, for reference:

docker rm -f $(docker ps -aq)
docker system prune -a --volumes -f
rm -rf /opt/grafana /opt/qa-shared /root/.local-stack

plus the leftovers experiments accumulate (/root/hz-test, /root/podman-poc, /root/grafana-backup-*). Note that the first line fails on an already-clean machine — with no containers the substitution is empty and docker reports “requires at least 1 argument”; the script handles that.

Commands

  • bash local-stack.sh — interactive menu (or first-run wizard)

  • bash local-stack.sh up — start (if already configured: offers start/modify/clean)

  • bash local-stack.sh down — stop all QA services

  • bash local-stack.sh restart — full restart (down + up)

  • bash local-stack.sh reload — restart only stopped/failed containers

  • bash local-stack.sh status — container status table

  • bash local-stack.sh dashboard — regenerate dashboard and show URL

  • bash local-stack.sh logs — FTACS logs (tail -f)

  • bash local-stack.sh logs <service> — specific service logs (e.g. logs northbound-api)

  • bash local-stack.sh monitoring — monitoring setup instructions

  • bash local-stack.sh clean — remove all data, configs, volumes

  • bash local-stack.sh help — command reference

Flags

  • --debug or DEBUG=1 — verbose output, full command logging

  • --clean — reset saved config, re-run wizard

  • --qa-dir=/path — manually specify qa-shared repo path

  • --db=mysql|oracle — skip the database question

  • --group=minimal|api|full|with-ai|custom — skip the service-group question

  • --services="svc1 svc2" — services for --group=custom

  • --tag=<service>=<tag> — pin one component’s image tag (repeatable)

  • -y, --yes — answer every prompt with its default (CI/CD mode)

Image Tags

qa-shared pins every FT image to :latest and keeps doing so — Jenkins both publishes and consumes that moving tag. The stand needs the opposite: a fixed, known-good build that still starts months from now. It therefore rewrites tags in the compose override it generates and never edits qa-shared.

The pinned set lives in local-stack/image-tags.conf, one <repository> <tag> pair per line. Only images from hub.friendly-tech.com are rewritten — MySQL, PostgreSQL, Keycloak and ClickHouse keep whatever qa-shared says.

Four places a tag can come from

They are consulted in this order — the first one that has an answer wins:

  1. --tag=<service>=<tag> on the command line — for a single run, repeatable:

    bash local-stack.sh up --tag=ftacs=v7.1.2-b0.0.1 --tag=ui-backend=master_0.0.37
  2. The wizard, under Image tags › customize — a form listing the components this run will start, each prefilled with the tag it would get. Edit only the ones you care about and pick [Save].

  3. ~/.local-stack/image-tags — where the two above are remembered, one service=tag per line. It is a plain file; editing it by hand works and is the easiest way to see what a stand is currently pinned to:

    cat ~/.local-stack/image-tags
    ftacs=v7.1.2-b0.0.1

    bash local-stack.sh up --clean deletes it and returns to the defaults.

  4. local-stack/image-tags.conf — the defaults themselves, per repository. Edit this to move every future stand to a newer release; edit the file above to change just this machine.

A component with no entry anywhere keeps whatever qa-shared specifies.

Not through the qa-shared .env. That file has no tag variables — the image lines in the compose files are literal — so setting something like FTACS_TAG= there has no effect. Tags are the stand’s own layer, applied in the override it generates, which is also why qa-shared can keep :latest untouched for Jenkins.

To check what a running stand actually pulled:

docker ps --format '{{.Names}}\t{{.Image}}' | grep '^ls_'

Service Groups

Container names below are the ones docker ps shows, without the ls_ prefix. The composition was measured on the test stand, 06.08.2026, and is the same on MySQL and Oracle — the database changes the startup time, not the service list.

minimal — Minimal FTACS Environment — 9 containers

mysql (or oracle), hazelcast, clickhouse, jdbc-bridge, jstun, ftacs, ft-configs-service, ft-configs-ui, ft-system-metrics RAM usage: ~8 GB

api — FTACS + Public APIs (no UI) — 13 containers

Everything from minimal + northbound-api, service-api, provision-api, provision-portal RAM usage: ~9 GB

full — Full Stack with UI — 17 containers

Everything from api + portals, ui-backend, ft-device-network, keycloak RAM usage: ~12 GB

with-ai — Full Stack + AI Agent — 19 containers

Everything from full + ui-ai-agent, postgres RAM usage: ~14 GB

custom — Manual Selection

Interactive checklist: space to toggle, Enter to confirm. Core services (MySQL, Hazelcast, ClickHouse, FTACS) are locked — always included. Dependencies are resolved automatically (e.g., selecting Portals adds UI Backend).

Services and Ports

Databases

  • MySQL — port 3307 (user: ftacs / ftacs, DB: ftacs)

  • Oracle — port 1522 (user: ftacs / ftacs, DB: ftacs)

  • PostgreSQL — port 5432 (user: postgres / postgres, DB: ftl_ai) — AI Agent only

  • ClickHouse — port 8123 (user: ftacs / ftacs, DB: ftacs)

Infrastructure

  • Hazelcast — port 5701 (in-memory cache/clustering)

  • JDBC Bridge — port 9000 (ClickHouse JDBC bridge)

  • Keycloak — port 8500 (SSO/authentication, user: admin / admin)

Core

  • FTACS — port 8080, path /FTACS/ (user: friendly / linux4israel)

    • HTTPS: 8443

    • MQTT: 1883, MQTT TLS: 8883

    • USP WebSocket: 8025, USP MQTT: 18881, USP STOMP: 61617

    • JMX: 9999 (for HikariCP monitoring)

    • Prometheus javaagent: 5557 (JVM metrics)

APIs

  • Northbound API — port 8082, health: /iot-webservice/actuator/health

  • Service API — port 8083, health: /iot-webservice/actuator/health

  • Provision API — port 8084, health: /prov-portal/actuator/health

UI

  • Portals — port 8880 (management web UI)

  • UI Backend — port 8881 (Angular API Backend), actuator on 9881

  • Provision Portal — port 8890, path /provision-portal/

  • Device Network — port 8383 (Network Discovery Service)

  • Configs Service — port 8086, health: /configs-service/actuator/health

  • Configs UI — port 3001

AI and Monitoring

  • AI Agent — port 8184 (FastAPI, requires PostgreSQL)

  • System Metrics — port 8680 (HikariCP + business metrics)

  • QoE Web — port 8085

Emulators

  • TR-069 Emulator — CPE device emulator

  • IoT Emulator — port 8556, IoT device emulator

Dashboard

  • Dashboard — port 8888 (HTTP Basic Auth, password auto-generated)

Service Dependencies

Dependencies are resolved automatically. Selecting a service adds all its dependencies.

  • portals → ui-backend → ftacs → (mysql/oracle, hazelcast, clickhouse)

  • service-api → northbound-api → ftacs

  • provision-portal → provision-api → northbound-api → ftacs

  • ui-ai-agent → postgres + ui-backend + ft-device-network-service

  • ft-configs-ui → ft-configs-service

Port Conflicts

On startup, local-stack checks all ports automatically. If a port is occupied (e.g., monitoring-stack Prometheus on 9090), the script finds a free port:

⚠ Port conflicts detected (auto-resolved):
·   Prometheus: 9090→9091
·   Grafana: 3100→3101

Resolved conflicts are saved in the state file and reapplied on each startup.

Monitoring

Local-stack does NOT manage the monitoring stack (Prometheus, Grafana, Alertmanager). They are managed separately via bootstrap:

# Step 1 — Exporters (node-exporter, mysqld-exporter, etc.)
cd /opt/grafana && bash misc/bootstrap.sh <IP> auto --local-stack

# Step 2 — Monitoring stack (Prometheus + Grafana)
cd /opt/grafana && bash misc/bootstrap.sh <IP> stack

JMX Metrics

FTACS and Hazelcast ship with built-in jmx_prometheus_javaagent:

  • FTACS JVM — port 5557 (jvm_info, jvm_memory, java_lang)

  • Hazelcast — port 9101 (com_hazelcast_*, jvm_info)

  • HikariCP — port 5556 (standalone jmx-exporter-hikari via JMX 9999)

Container Memory Limits

Limits are set in the auto-generated docker-compose.qa-network.yml:

  • FTACS: 4 GB

  • MySQL: 2 GB

  • Hazelcast: 2 GB

  • ClickHouse: 1.5 GB

  • Keycloak: 1.5 GB

  • UI Backend: 1.5 GB

  • Provision API: 1 GB

  • Northbound/Service API: 768 MB

  • Device Network: 768 MB

  • PostgreSQL, AI Agent, FT Configs Service: 1 GB

  • Portals, Provision Portal, Configs UI, System Metrics, JDBC Bridge: 512 MB

Java heap is auto-scaled based on available Docker memory.

Files and Directories

Generated files (not committed)

  • docker-compose.qa-network.yml — compose override (network, ports, limits, javaagent)

  • dashboard.html — generated HTML dashboard page

  • data/logs/ — session logs (local-stack-YYYYMMDD-HHMMSS.log)

  • ~/.local-stack/state — saved configuration (DB_TYPE, SERVICE_GROUP, etc.)

  • ~/.local-stack/dashboard-pass — dashboard password

Source files

  • local-stack.sh — entry point, command routing

  • lib/config.sh — service groups, dependencies, config generation

  • lib/compose.sh — Docker Compose orchestration, health-check waiting

  • lib/checks.sh — prerequisites, port management, firewall checks

  • lib/dashboard.sh — dashboard HTML generation and terminal summary

  • lib/ui.sh — colored output, spinners, interactive menus

  • service-registry.yaml — service metadata (ports, credentials, descriptions)

  • jmx-agent-config.yml — JMX Prometheus config (MBeans whitelist)

  • jmx_prometheus_javaagent.jar — JAR for JVM metrics

  • templates/dashboard.html — dashboard HTML template

Testing

bash tests/test-local-stack.sh

31 sections: service groups, profiles, dependencies, config generation, dashboard, state persistence, cleanup — and error paths. The suite prints its own total; what must hold is 0 failed.

Section 31 covers the degraded environments rather than the happy path: qa-shared missing, Docker not running, yq absent, a dependencies.yaml that exists but does not parse, and occupied ports. The suite itself hard-requires Docker and yq (see the top of the file), so each broken state is injected into a subshell instead of by breaking the host.

Two behaviours it pins down, because both used to fail silently:

  • A present but unparseable dependencies.yaml passes the “file exists and yq is installed” guard, so yq errors out into /dev/null and the reader returns nothing. get_service_group, list_groups, get_core_services and get_hidden_services all fall back to their built-in lists instead — otherwise the picker shows no groups and the stack starts with no core infrastructure.

  • When dependency resolution cannot run at all (no yq, no schema), local-stack says so on screen, once per run. It used to just return an empty dependency list, so services quietly never started and the failure surfaced later as unrelated container errors.

Troubleshooting

Every case below was hit for real while bringing this environment up; the symptom is what you actually see, not what the code says.

Everything looks green in Prometheus, but panels say “No data”

The dashboards filter by instance. If one host is registered in servers.env under several names — say NODE as DB-MySQL while JMX_HC is still poc-hazelcast from an old experiment — the series under the odd name match nothing and those panels stay empty while every target is UP.

awk '$2 ~ /^<IP>:/ {print $3}' /opt/grafana/prometheus-grafana-stack/prometheus/targets/servers.env | sort -u

More than one name (ignoring the documented -grafana-proxy suffix) is the problem. Make the lines agree, then:

bash /opt/grafana/prometheus-grafana-stack/prometheus/scripts/generate-targets.sh

The deploy warns about this, and never rewrites the file itself — the inventory is yours to edit.

The stand refuses to start, naming a busy port

Deliberate. Monitoring finds these ports by number, so shifting one would give a stand that comes up, looks healthy and is never scraped. The message names the process holding the port. Free it — usually a real FTACS platform on the same host, another stand, or a leftover test container.

The one exception is 9101: if the monitoring stack’s own Hazelcast exporter holds it, the stand steps aside instead of refusing (see step 4 above).

Containers all sit in “Created” and nothing runs

compose aborts the entire up when a single service cannot be created — a missing image, a port already bound. Everything else is left in Created, which reads like a hang rather than a failure. Look above the summary for the actual compose error; one service explains all of them.

docker ps -a --format '{{.Names}}\t{{.Status}}' | grep ls_

Hazelcast or HikariCP panels are empty, exporters return ~6 metrics

The exporters reach JMX over RMI, and RMI hands the client an address for the follow-up connection. The exporters run with host networking, so that address must be the host’s — a container name resolves only inside the compose network.

docker logs jmx-exporter-hazelcast 2>&1 | tail -5     # "Unknown host: ..." confirms it

Grafana, Prometheus and Alertmanager are all unreachable, exporters fine

The reverse proxy is crash-looping. Its self-monitoring listener collides with something else on the host and nginx exits on the failed bind, taking the whole web surface with it.

docker logs nginx_proxy 2>&1 | tail -3                # "Address already in use"

The port is configurable (NGINX_STATUS_PORT in the stack’s .env) and the deploy moves it automatically when it is taken.

docker compose cannot find the env file

The qa-shared clone is incomplete — usually interrupted. Its .env.mysql and .env.oracle are tracked in the repository, so a healthy clone always has them. Re-fetch:

rm -rf /opt/grafana/qa-shared
cd /opt/grafana/local-stack && bash local-stack.sh up --db=mysql --group=with-ai -y

A service answers HTTP 500 on Oracle, with ORA-00604 in its log

ORA-00604: error occurred at recursive SQL level 1

That is the wrapper, and it reads like a session-limit problem. Read the nested error — it is almost always:

ORA-01882: timezone region not found

A containerised Oracle XE lacks the full timezone region set, so a JDBC driver sending a region name (Europe/Kyiv) is refused. The stand passes -Doracle.jdbc.timezoneAsRegion=false to every FT service; if you see this again, check that the flag reached the container:

docker inspect ls_ft-system-metrics --format '{{range .Config.Env}}{{println .}}{{end}}' | grep -i timezone

A dashboard is empty although its exporter returns 200

Check whether there is anything to report before suspecting the build. Business panels read from the platform’s own tables, and a fresh stand has none:

# oracle
docker exec ls_oracle bash -lc "echo 'select count(*) from cpe;' | sqlplus -s ftacs/ftacs@//localhost:1521/XEPDB1"
# mysql
docker exec ls_mysql mysql -uftacs -pftacs ftacs -e 'select count(*) from cpe;'

Zero rows means the panels are correct and simply have nothing to draw. The exporter’s own counters confirm it is querying: look for spring_data_repository_invocations_seconds_count with repository="CpeRepository".

Docker not running

✗ Docker daemon is not running

systemctl start docker

Image pull keeps failing

⚠ Failed (attempt 1/3). Retrying in 5s...

Three automatic retries. If it persists, check connectivity to hub.friendly-tech.com. Note that the tags the stand uses are pinned in local-stack/image-tags.conf — a tag that no longer exists in the registry fails here, and the fix is to update that file, not to retry.

A container keeps restarting

bash local-stack.sh logs <service>   # its own log first
bash local-stack.sh restart          # full restart

branch 1.6.9 · commit a862b45d92590285f229f80a6b5d2efabcb3b1b0 · page generated August 08, 2026