Local Stack — Guide
- Local Stack — Guide
- Table of Contents
- Full Setup (Start to Finish)
- Commands
- Image Tags
- Service Groups
- Services and Ports
- Service Dependencies
- Port Conflicts
- Monitoring
- Container Memory Limits
- Files and Directories
- Testing
- Troubleshooting
- Everything looks green in Prometheus, but panels say “No data”
- The stand refuses to start, naming a busy port
- Containers all sit in “Created” and nothing runs
- Hazelcast or HikariCP panels are empty, exporters return ~6 metrics
- Grafana, Prometheus and Alertmanager are all unreachable, exporters fine
docker composecannot find the env file- A service answers HTTP 500 on Oracle, with ORA-00604 in its log
- A dashboard is empty although its exporter returns 200
- Docker not running
- Image pull keeps failing
- A container keeps restarting
Version 1.6.9 | Updated: August 08, 2026
Local Stack — Guide
Local development environment for the FTACS platform. Spins up the platform with a single command — up to 19 containers, depending on the service group, then deploys monitoring (Prometheus, Grafana, Alertmanager) with exporters and 14 pre-configured dashboards.
Table of Contents
For monitoring real servers rather than a test environment, follow the main installation guide instead — there you deploy only the exporters, without the platform. The architecture overview explains how the pieces fit together.
Full Setup (Start to Finish)
This is the whole path on a machine with nothing installed, exactly as it was last verified end to end. Support can follow it as-is to reproduce a test environment without asking anyone.
What this is, and when NOT to use it
local-stack deploys the FTACS platform itself — MySQL, FTACS, Hazelcast, the
APIs and the UI — in containers, and then the monitoring is pointed at it. That
makes it a self-contained test bed, and it also means it does not belong on a
server that already runs FTACS: the ports it needs are exactly the ones the real
platform occupies. On such a server deploy only the exporters
(bootstrap.sh <prometheus-ip> auto).
Prerequisites
-
Linux server with Docker and Docker Compose v2.20+
-
16 GB RAM minimum (32 GB recommended for
with-ai), 25 GB free disk -
The metrics ports free —
9101,9110,8080,8843,9100,5556. They are not auto-shifted: monitoring discovers them by number, so a moved port would mean a stand that looks healthy and is never scraped. -
Nothing else to prepare: both repositories are fetched by the commands below, including
qa-sharedwith its.envfiles.
Step 1 — Fetch the code
curl -fsSL https://hub.friendly-tech.com/bootstrap/install.sh | sudo bash -s -- --branch 1.6.9 --update
--update means get the code and deploy nothing. On a clean host it clones,
on an existing checkout it pulls. The repository is private, so this wrapper is
the only way to fetch it — it carries the access token.
Step 2 — Start the platform
Two ways to run it. Interactively, if you want to see and choose everything:
cd /opt/grafana/local-stack
bash local-stack.sh up
The wizard asks four things, each with a description of what it means:
-
Database — MySQL (faster, lighter) or Oracle (heavier, for Oracle-specific testing).
-
Service group — how much of the platform to start:
minimal,api,full,with-ai, orcustomto pick services one by one. -
Image tags — the pinned release build, or set the tag per component.
-
Confirmation — the summary before anything starts.
Or non-interactively, with the same choices as flags:
cd /opt/grafana/local-stack
bash local-stack.sh up --db=mysql --group=with-ai -y
qa-shared is cloned automatically. On a clean host the first run takes roughly
10 minutes, nearly all of it pulling about 18 GB of images; once they are cached
a start takes 2—4 minutes on MySQL.
docker ps --format '{{.Names}}' | grep -c '^ls_' # 19 for with-ai
What each group actually starts. Measured on the test stand, 06.08.2026, all eight combinations with cached images, every container healthy:
-
minimal— 9 containers. MySQL 3m37s, Oracle 6m46s -
api— 13 containers. MySQL 2m27s -
full— 17 containers. MySQL 3m30s, Oracle 2m22s -
with-ai— 19 containers. MySQL 2m14s, Oracle 3m19s
The database does not change what starts: on the same group Oracle and MySQL give
the same containers. It changes how long the first start takes — Oracle spends
several minutes initialising before anything else can come up, which is most of
the difference on minimal.
custom is not “only the services you name”. --services="ftacs northbound-api"
started 12 containers: the database, Hazelcast, ClickHouse, the JDBC bridge
and the rest of the infrastructure come along, because the named services do not
run without them.
Customising what starts
| Want to | Do |
|---|---|
Pick services one by one |
|
Use Oracle |
|
Re-run the wizard from scratch |
|
See the full command reference |
|
Core services (database, Hazelcast, FTACS) are always included — the picker marks
them (required) and they cannot be switched off, because nothing else starts
without them.
Changing the version of an application
Every FT image is pinned to a known-good build in
local-stack/image-tags.conf. To run a different build of one component:
bash local-stack.sh up --tag=ftacs=v7.1.2-b0.0.1
The flag is repeatable, so several components can be moved at once:
bash local-stack.sh up --tag=ftacs=v7.1.2-b0.0.1 --tag=ui-backend=master_0.0.37
The wizard offers the same under Image tags › customize: it lists the components this run will start, each prefilled with the tag it would get, and you edit only what you need.
Either way the choice is saved with the rest of the configuration and reused on
the next up, so a stand keeps the versions you gave it. --clean forgets them
and goes back to the pinned defaults.
To move the whole stand to a newer release for good, edit image-tags.conf —
that is the file the defaults come from. See Image Tags for how
the pinning works and why qa-shared keeps :latest.
Step 3 — Deploy the exporters
cd /opt/grafana
bash misc/bootstrap.sh 65.109.58.165 auto --local-stack -y
Replace the IP with the address of the Prometheus server (the same machine here).
--local-stack tells bootstrap that the platform it just detected is a stand:
it reads the ports and credentials from the stand’s own configuration, provisions
a dedicated exporter database user inside the container, and never touches the
host’s native database. Without the flag it asks; under -y it takes the stand
by default; --no-local-stack opts out.
There is no restart step. This guide used to tell you to restart the stand and deploy again, on the grounds that the Hazelcast exporter and the stand compete for port
9101and that only the exporter servesisClusterSafe. Measured on the stand, 06.08.2026, none of that holds: no Hazelcast exporter is deployed here at all —quick-deployskips it and says so (“Hazelcast already serves the full metric set on :9101”) —ls_hazelcastowns9101before and after a restart alike, andisClusterSafeis served from the moment the stand comes up, with no exporter involved. The restart cost a stand cycle per run and changed nothing.
Step 4 — Deploy the monitoring stack
The stack itself — Prometheus, Grafana, Alertmanager and the alert channels — is documented in the main installation guide; only what the stand needs is repeated here.
cd /opt/grafana
bash misc/bootstrap.sh 65.109.58.165 stack
Asks for the Grafana and Prometheus passwords, SMTP and the alert channels. Add
-y to take defaults: passwords are then generated and printed at the end.
Step 5 — Verify
cd /opt/grafana && bash misc/bootstrap.sh 65.109.58.165 auto --local-stack -y 2>&1 | tail -25
A healthy environment reports 0 failed and looks like this:
-
JMX_HC— about 9900 metrics. Six means the exporter cannot reach JMX. -
JMX_HIKARI— about 20. Six means the same. -
MYSQL— 30000+. Around 23000 means it is scraping the host’s native database instead of the stand: the--local-stackstep did not apply. -
UI_BACKEND— about 400, on port9881. Its actuator is on a management port of its own; the service port answers 500 and the debug port has no metrics.
Dashboards need a separate check — a target can be UP while the series a panel asks for does not exist, and the panel then reads “No data” with nothing reporting a problem:
cd /opt/grafana && bash misc/testing/check-dashboard-panels.sh --quiet
It runs every panel’s own query and lists the empty ones. Some emptiness is correct and should not be chased:
-
windows-exporter-iis-node— no Windows hosts on the machine -
oracle— empty on a MySQL stand, and also on an Oracle stand: the Oracle exporter is not deployed for a stand at all (--local-stackcovers MySQL and PostgreSQL only), sooracledb_uphas no series either way -
business-metrics— the platform starts with an empty database, and a gauge over zero rows produces no series. These panels fill once the stand has devices; confirm withselect count(*) from cpebefore suspecting anything -
parts of
api-metrics— the counters appear after the APIs are called
A surprising result is more likely a bad query rewrite than a broken panel: open
the dashboard in Grafana before acting on it. The variables are substituted
textually here, and a matcher written with single quotes (tag0='$map') once
turned into an exact match on the literal .*, reporting a dozen healthy panels
as empty.
Comparing against a reference installation is what separates “this stand lacks the source” from “this panel is broken for everyone” — run the same script on the monitoring host that watches real servers and diff the two.
Then check Prometheus and Grafana:
https://<IP>/prometheus/targets # every target UP https://<IP>/ # Grafana; credentials printed by step 5
Credentials are also in /opt/grafana/prometheus-grafana-stack/.env
(GRAFANA_ADMIN_PASSWORD, PROMETHEUS_USER, PROMETHEUS_PASSWORD).
Running the stand on Oracle
Same path, one flag:
bash local-stack.sh up --db=oracle --group=with-ai -y
Oracle is heavier — the database alone takes several minutes before anything else
can start. Expect the same containers as on MySQL (19 for with-ai) and,
with the monitoring deployed, 16 targets UP. Measured with cached images:
minimal 6m46s against MySQL’s 3m37s, the gap being Oracle’s initialisation;
once it is up the larger groups are no slower than MySQL.
Two things are specific to it:
-
The Oracle exporter is not deployed. Nothing configures it from the stand, so
oracledb_uphas no series and theoracledashboard stays empty. The platform itself is fully monitored; only the database’s own metrics are missing. -
Timezone. A containerised Oracle XE does not carry the full timezone region set, so a JDBC driver that sends a region name is refused. The stand passes
-Doracle.jdbc.timezoneAsRegion=falseto every FT service to avoid it — see the troubleshooting entry below for what it looks like when it bites.
Step 6 — Tearing it down
cd /opt/grafana/local-stack && bash local-stack.sh clean # stand only, keeps the monitoring
To reset the machine completely, including images:
bash misc/free-stand.sh --host <this-stand> # show what would go
bash misc/free-stand.sh --host <this-stand> --yes # carry it out
--host must name the machine you are on — its short name, its FQDN or one of
its addresses. That is the point of it: the commands below are the same ones
that reset a test stand and destroy a monitoring host, and nothing in them says
which is which. Without --yes the script only prints the plan.
A native (non-Docker) database on the host is untouched. The script records which database ports are served by a non-container process before it starts and fails if one of them stops answering, so “the wipe took the host’s MySQL with it” cannot pass unnoticed.
What it runs, for reference:
docker rm -f $(docker ps -aq)
docker system prune -a --volumes -f
rm -rf /opt/grafana /opt/qa-shared /root/.local-stack
plus the leftovers experiments accumulate (/root/hz-test, /root/podman-poc,
/root/grafana-backup-*). Note that the first line fails on an already-clean
machine — with no containers the substitution is empty and docker reports
“requires at least 1 argument”; the script handles that.
Commands
-
bash local-stack.sh— interactive menu (or first-run wizard) -
bash local-stack.sh up— start (if already configured: offers start/modify/clean) -
bash local-stack.sh down— stop all QA services -
bash local-stack.sh restart— full restart (down + up) -
bash local-stack.sh reload— restart only stopped/failed containers -
bash local-stack.sh status— container status table -
bash local-stack.sh dashboard— regenerate dashboard and show URL -
bash local-stack.sh logs— FTACS logs (tail -f) -
bash local-stack.sh logs <service>— specific service logs (e.g.logs northbound-api) -
bash local-stack.sh monitoring— monitoring setup instructions -
bash local-stack.sh clean— remove all data, configs, volumes -
bash local-stack.sh help— command reference
Flags
-
--debugorDEBUG=1— verbose output, full command logging -
--clean— reset saved config, re-run wizard -
--qa-dir=/path— manually specify qa-shared repo path -
--db=mysql|oracle— skip the database question -
--group=minimal|api|full|with-ai|custom— skip the service-group question -
--services="svc1 svc2"— services for--group=custom -
--tag=<service>=<tag>— pin one component’s image tag (repeatable) -
-y,--yes— answer every prompt with its default (CI/CD mode)
Image Tags
qa-shared pins every FT image to :latest and keeps doing so — Jenkins both
publishes and consumes that moving tag. The stand needs the opposite: a fixed,
known-good build that still starts months from now. It therefore rewrites tags
in the compose override it generates and never edits qa-shared.
The pinned set lives in local-stack/image-tags.conf, one
<repository> <tag> pair per line. Only images from hub.friendly-tech.com
are rewritten — MySQL, PostgreSQL, Keycloak and ClickHouse keep whatever
qa-shared says.
Four places a tag can come from
They are consulted in this order — the first one that has an answer wins:
-
--tag=<service>=<tag>on the command line — for a single run, repeatable:bash local-stack.sh up --tag=ftacs=v7.1.2-b0.0.1 --tag=ui-backend=master_0.0.37 -
The wizard, under Image tags › customize — a form listing the components this run will start, each prefilled with the tag it would get. Edit only the ones you care about and pick
[Save]. -
~/.local-stack/image-tags— where the two above are remembered, oneservice=tagper line. It is a plain file; editing it by hand works and is the easiest way to see what a stand is currently pinned to:cat ~/.local-stack/image-tags ftacs=v7.1.2-b0.0.1bash local-stack.sh up --cleandeletes it and returns to the defaults. -
local-stack/image-tags.conf— the defaults themselves, per repository. Edit this to move every future stand to a newer release; edit the file above to change just this machine.
A component with no entry anywhere keeps whatever qa-shared specifies.
Not through the qa-shared
.env. That file has no tag variables — the image lines in the compose files are literal — so setting something likeFTACS_TAG=there has no effect. Tags are the stand’s own layer, applied in the override it generates, which is also why qa-shared can keep:latestuntouched for Jenkins.
To check what a running stand actually pulled:
docker ps --format '{{.Names}}\t{{.Image}}' | grep '^ls_'
Service Groups
Container names below are the ones docker ps shows, without the ls_ prefix.
The composition was measured on the test stand, 06.08.2026, and is the same on
MySQL and Oracle — the database changes the startup time, not the service list.
minimal — Minimal FTACS Environment — 9 containers
mysql (or oracle), hazelcast, clickhouse, jdbc-bridge, jstun,
ftacs, ft-configs-service, ft-configs-ui, ft-system-metrics
RAM usage: ~8 GB
api — FTACS + Public APIs (no UI) — 13 containers
Everything from minimal + northbound-api, service-api, provision-api,
provision-portal
RAM usage: ~9 GB
full — Full Stack with UI — 17 containers
Everything from api + portals, ui-backend, ft-device-network, keycloak
RAM usage: ~12 GB
Services and Ports
Databases
-
MySQL — port 3307 (user: ftacs / ftacs, DB: ftacs)
-
Oracle — port 1522 (user: ftacs / ftacs, DB: ftacs)
-
PostgreSQL — port 5432 (user: postgres / postgres, DB: ftl_ai) — AI Agent only
-
ClickHouse — port 8123 (user: ftacs / ftacs, DB: ftacs)
Infrastructure
-
Hazelcast — port 5701 (in-memory cache/clustering)
-
JDBC Bridge — port 9000 (ClickHouse JDBC bridge)
-
Keycloak — port 8500 (SSO/authentication, user: admin / admin)
Core
-
FTACS — port 8080, path
/FTACS/(user: friendly / linux4israel)-
HTTPS: 8443
-
MQTT: 1883, MQTT TLS: 8883
-
USP WebSocket: 8025, USP MQTT: 18881, USP STOMP: 61617
-
JMX: 9999 (for HikariCP monitoring)
-
Prometheus javaagent: 5557 (JVM metrics)
-
APIs
-
Northbound API — port 8082, health:
/iot-webservice/actuator/health -
Service API — port 8083, health:
/iot-webservice/actuator/health -
Provision API — port 8084, health:
/prov-portal/actuator/health
UI
-
Portals — port 8880 (management web UI)
-
UI Backend — port 8881 (Angular API Backend), actuator on 9881
-
Provision Portal — port 8890, path
/provision-portal/ -
Device Network — port 8383 (Network Discovery Service)
-
Configs Service — port 8086, health:
/configs-service/actuator/health -
Configs UI — port 3001
Service Dependencies
Dependencies are resolved automatically. Selecting a service adds all its dependencies.
-
portals→ ui-backend → ftacs → (mysql/oracle, hazelcast, clickhouse) -
service-api→ northbound-api → ftacs -
provision-portal→ provision-api → northbound-api → ftacs -
ui-ai-agent→ postgres + ui-backend + ft-device-network-service -
ft-configs-ui→ ft-configs-service
Port Conflicts
On startup, local-stack checks all ports automatically. If a port is occupied (e.g., monitoring-stack Prometheus on 9090), the script finds a free port:
⚠ Port conflicts detected (auto-resolved): · Prometheus: 9090→9091 · Grafana: 3100→3101
Resolved conflicts are saved in the state file and reapplied on each startup.
Monitoring
Local-stack does NOT manage the monitoring stack (Prometheus, Grafana, Alertmanager). They are managed separately via bootstrap:
# Step 1 — Exporters (node-exporter, mysqld-exporter, etc.)
cd /opt/grafana && bash misc/bootstrap.sh <IP> auto --local-stack
# Step 2 — Monitoring stack (Prometheus + Grafana)
cd /opt/grafana && bash misc/bootstrap.sh <IP> stack
Container Memory Limits
Limits are set in the auto-generated docker-compose.qa-network.yml:
-
FTACS: 4 GB
-
MySQL: 2 GB
-
Hazelcast: 2 GB
-
ClickHouse: 1.5 GB
-
Keycloak: 1.5 GB
-
UI Backend: 1.5 GB
-
Provision API: 1 GB
-
Northbound/Service API: 768 MB
-
Device Network: 768 MB
-
PostgreSQL, AI Agent, FT Configs Service: 1 GB
-
Portals, Provision Portal, Configs UI, System Metrics, JDBC Bridge: 512 MB
Java heap is auto-scaled based on available Docker memory.
Files and Directories
Generated files (not committed)
-
docker-compose.qa-network.yml— compose override (network, ports, limits, javaagent) -
dashboard.html— generated HTML dashboard page -
data/logs/— session logs (local-stack-YYYYMMDD-HHMMSS.log) -
~/.local-stack/state— saved configuration (DB_TYPE, SERVICE_GROUP, etc.) -
~/.local-stack/dashboard-pass— dashboard password
Source files
-
local-stack.sh— entry point, command routing -
lib/config.sh— service groups, dependencies, config generation -
lib/compose.sh— Docker Compose orchestration, health-check waiting -
lib/checks.sh— prerequisites, port management, firewall checks -
lib/dashboard.sh— dashboard HTML generation and terminal summary -
lib/ui.sh— colored output, spinners, interactive menus -
service-registry.yaml— service metadata (ports, credentials, descriptions) -
jmx-agent-config.yml— JMX Prometheus config (MBeans whitelist) -
jmx_prometheus_javaagent.jar— JAR for JVM metrics -
templates/dashboard.html— dashboard HTML template
Testing
bash tests/test-local-stack.sh
31 sections: service groups, profiles, dependencies, config generation, dashboard, state persistence, cleanup — and error paths. The suite prints its own total; what must hold is 0 failed.
Section 31 covers the degraded environments rather than the happy path: qa-shared
missing, Docker not running, yq absent, a dependencies.yaml that exists but does
not parse, and occupied ports. The suite itself hard-requires Docker and yq (see
the top of the file), so each broken state is injected into a subshell instead of
by breaking the host.
Two behaviours it pins down, because both used to fail silently:
-
A present but unparseable
dependencies.yamlpasses the “file exists and yq is installed” guard, soyqerrors out into/dev/nulland the reader returns nothing.get_service_group,list_groups,get_core_servicesandget_hidden_servicesall fall back to their built-in lists instead — otherwise the picker shows no groups and the stack starts with no core infrastructure. -
When dependency resolution cannot run at all (no
yq, no schema),local-stacksays so on screen, once per run. It used to just return an empty dependency list, so services quietly never started and the failure surfaced later as unrelated container errors.
Troubleshooting
Every case below was hit for real while bringing this environment up; the symptom is what you actually see, not what the code says.
Everything looks green in Prometheus, but panels say “No data”
The dashboards filter by instance. If one host is registered in servers.env
under several names — say NODE as DB-MySQL while JMX_HC is still
poc-hazelcast from an old experiment — the series under the odd name match
nothing and those panels stay empty while every target is UP.
awk '$2 ~ /^<IP>:/ {print $3}' /opt/grafana/prometheus-grafana-stack/prometheus/targets/servers.env | sort -u
More than one name (ignoring the documented -grafana-proxy suffix) is the
problem. Make the lines agree, then:
bash /opt/grafana/prometheus-grafana-stack/prometheus/scripts/generate-targets.sh
The deploy warns about this, and never rewrites the file itself — the inventory is yours to edit.
The stand refuses to start, naming a busy port
Deliberate. Monitoring finds these ports by number, so shifting one would give a stand that comes up, looks healthy and is never scraped. The message names the process holding the port. Free it — usually a real FTACS platform on the same host, another stand, or a leftover test container.
The one exception is 9101: if the monitoring stack’s own Hazelcast exporter
holds it, the stand steps aside instead of refusing (see step 4 above).
Containers all sit in “Created” and nothing runs
compose aborts the entire up when a single service cannot be created — a
missing image, a port already bound. Everything else is left in Created, which
reads like a hang rather than a failure. Look above the summary for the actual
compose error; one service explains all of them.
docker ps -a --format '{{.Names}}\t{{.Status}}' | grep ls_
Hazelcast or HikariCP panels are empty, exporters return ~6 metrics
The exporters reach JMX over RMI, and RMI hands the client an address for the follow-up connection. The exporters run with host networking, so that address must be the host’s — a container name resolves only inside the compose network.
docker logs jmx-exporter-hazelcast 2>&1 | tail -5 # "Unknown host: ..." confirms it
Grafana, Prometheus and Alertmanager are all unreachable, exporters fine
The reverse proxy is crash-looping. Its self-monitoring listener collides with something else on the host and nginx exits on the failed bind, taking the whole web surface with it.
docker logs nginx_proxy 2>&1 | tail -3 # "Address already in use"
The port is configurable (NGINX_STATUS_PORT in the stack’s .env) and the
deploy moves it automatically when it is taken.
docker compose cannot find the env file
The qa-shared clone is incomplete — usually interrupted. Its .env.mysql and
.env.oracle are tracked in the repository, so a healthy clone always has
them. Re-fetch:
rm -rf /opt/grafana/qa-shared
cd /opt/grafana/local-stack && bash local-stack.sh up --db=mysql --group=with-ai -y
A service answers HTTP 500 on Oracle, with ORA-00604 in its log
ORA-00604: error occurred at recursive SQL level 1
That is the wrapper, and it reads like a session-limit problem. Read the nested error — it is almost always:
ORA-01882: timezone region not found
A containerised Oracle XE lacks the full timezone region set, so a JDBC driver
sending a region name (Europe/Kyiv) is refused. The stand passes
-Doracle.jdbc.timezoneAsRegion=false to every FT service; if you see this
again, check that the flag reached the container:
docker inspect ls_ft-system-metrics --format '{{range .Config.Env}}{{println .}}{{end}}' | grep -i timezone
A dashboard is empty although its exporter returns 200
Check whether there is anything to report before suspecting the build. Business panels read from the platform’s own tables, and a fresh stand has none:
# oracle
docker exec ls_oracle bash -lc "echo 'select count(*) from cpe;' | sqlplus -s ftacs/ftacs@//localhost:1521/XEPDB1"
# mysql
docker exec ls_mysql mysql -uftacs -pftacs ftacs -e 'select count(*) from cpe;'
Zero rows means the panels are correct and simply have nothing to draw. The
exporter’s own counters confirm it is querying: look for
spring_data_repository_invocations_seconds_count with repository="CpeRepository".
Image pull keeps failing
⚠ Failed (attempt 1/3). Retrying in 5s...
Three automatic retries. If it persists, check connectivity to
hub.friendly-tech.com. Note that the tags the stand uses are pinned in
local-stack/image-tags.conf — a tag that no longer exists in the registry fails
here, and the fix is to update that file, not to retry.