Installation & Deployment

This guide covers deploying FTACS — the TR-069/TR-369 Auto Configuration Server — as a single service.

Two deployment methods are supported: Docker Compose (recommended) and manual installation from the distribution package on bare metal or a VM.

The infrastructure FTACS depends on — a relational database, ClickHouse, Hazelcast, and FT Configs Service — is assumed to be already deployed and reachable; this guide only points FTACS at it. To deploy that infrastructure, see All in one server deployment or Separate server deployment.
For the environment variable reference, see Configuration Guide — Environment Variables.

1. Overview

FTACS is a single container (ftacs) — or a single JVM process in a manual deployment — exposing the TR-069, USP, MQTT, LWM2M, SOAP, and REST endpoints. It is a plain Compose service on the ft-network bridge, deployed from /usr/local/ft-system/.

Deployment method What it installs Section

Docker Compose

The ftacs container, its configuration directory, and its logs

Docker Compose

Manual

The distribution package on a JDK 25 host, driven by bin/app.env

Manual (dist package)

Everything else in the diagram below is a prerequisite, not part of this installation:

CPE devices  ->  FTACS  ->  MySQL / Oracle  +  ClickHouse  +  Hazelcast  <-  FT Configs Service

FTACS uses the Compose project name ft-system and the directory root /usr/local/ft-system/ — the same layout as every deployment guide in the Friendly Tech portal. Keeping the layout identical means the ftacs service can be added to an existing ft-system stack without moving any directory. Use the project name to manage it from anywhere on the host, e.g. docker compose -p ft-system ps.

1.1. Choosing a Deployment Method

  • Docker Compose — recommended. Use this unless a policy forbids containers.

  • Manual — the distribution package runs directly on a JDK 25 host. Use it when Docker is unavailable, or when FTACS must run alongside existing bare-metal services.

Since version 6.5.2, FT Configs Service is a hard dependency — it owns the ACS runtime configuration that used to live in ftacs/config/ and publishes it through Hazelcast. FTACS is not usable without it. See Prerequisites.

Deploying it is out of scope here: use FT Configs deployment, or Separate server deployment — Server C, which brings up FTACS and FT Configs Service together on one host.

2. Prerequisites

2.1. Host Requirements

Component Minimum Recommended Notes

CPU

4 cores

 — 

Baseline for a single FTACS instance

RAM

8 GB

 — 

Must cover the JVM heap set through FT_ACS_JAVA_RAM — see the sizing table below

Disk

20 GB

 — 

Application, configuration, and logs. The database and ClickHouse keep their data on their own hosts

Network

1 Gbps interface

 — 

Device traffic plus the outbound connections listed in Network Requirements

Docker Engine

20.10

 — 

Docker deployment only; Docker Compose 2.0+ as well

JDK

25

 — 

Manual deployment only — the application is built and run on Java 25

JVM sizing by deployment scale:

Deployment size Devices JVM settings

Small

< 1,000

-Xms512m -Xmx2g

Medium

1,000—​5,000

-Xms1g -Xmx4g

Large

5,000—​20,000

-Xms2g -Xmx8g

Very large

20,000+

-Xms4g -Xmx16g

2.2. Required External Dependencies

These services must be installed, running, and reachable from this host before FTACS starts. None of them are deployed by this guide.

Component Minimum Version Why It Is Needed Port Required

MySQL or Oracle

MySQL 8.4 / Oracle 21c (the reference compose in this repo builds MySQL from mysql:8.4.6 and runs gvenzl/oracle-xe:21)

Holds the ftacs and ftacs_quartz schemas; Liquibase applies the FTACS migrations into them on first start. Without it FTACS does not start.

3306 (TCP) for MySQL, 1521 (TCP) for Oracle

Yes

ClickHouse

26.2

Stores the QoE and statistics data (schema ftacs_qoe_ui_data). FTACS connects with the clickhouse-jdbc client from clickhouse-java over HTTP, so the native TCP port is not sufficient. Without it QoE and statistics queries fail.

8123 (HTTP)

Yes

Hazelcast

5.5 (FTACS embeds the com.hazelcast:hazelcast 5.5.0 client; the reference compose image is built from hazelcast/hazelcast:5.5.0-slim)

Distributed cache and the only channel through which FT Configs Service delivers the ACS runtime configuration. Without it FTACS cannot initialise its Hazelcast client and receives no configuration.

5701, 5711 (TCP)

Yes

FT Configs Service

verify with the FT Configs Service owner

Owns the ACS runtime configuration since FTACS 6.5.2 and publishes it into the shared Hazelcast cluster. Without it FTACS starts with no ACS configuration, no web service users, and no parameter-name cache rules.

No direct port — delivered through Hazelcast; its own health endpoint is 8087 (HTTP)

Yes

Deploying that infrastructure is out of scope here — see All in one server deployment for a single-host stack, or Separate server deployment for a split topology, and FT Configs deployment for FT Configs Service.

The database schemas (ftacs, ftacs_quartz) and the ClickHouse schema (ftacs_qoe_ui_data) must exist and be accessible to the FTACS user before the first start.

26.2 is the supported ClickHouse version. The reference compose in this repo (FT_IoT/FTAPP/ftapp-build/docker/docker-compose.yml) still pins clickhouse/clickhouse-server:24.8; that pin is a development-stack artefact and is tracked in docs/decisions/LOG.md. Deploy 26.2 unless FT DevOps tells you otherwise.

FTACS and FT Configs Service must be clients of the same Hazelcast cluster — that cluster is the only channel between them. If HZ_MEMBERS on the two services points at different clusters, FTACS starts cleanly and then behaves as if no configuration had ever been set: this failure is silent, with no connection error in the log.

2.3. Supported Operating Systems

Deployment Operating system

Docker

Linux with Docker Engine 20.10+ (recommended), macOS, or Windows with WSL2

Manual

Linux (recommended), Windows Server — JDK 25 required

2.4. Registry Access

Network access to the hub.friendly-tech.com Docker registry (or offline image archives — see Offline Servers).

Access to Friendly Tech SharePoint (FT_DISK) for the deployment files (compose.yml, .env, ftacs/.env) and the pre-configured ftacs-conf package.

3. Network Requirements

Outbound connections FTACS opens to its infrastructure, and inbound connections it accepts. Each row is a connection that must be open through firewalls when the peer is on another host; traffic to peers on the same Docker bridge needs no rule.

Destination Port Protocol Purpose

Database

3306 (Oracle: 1521)

TCP

Outbound: relational database connection

ClickHouse

8123

HTTP

Outbound: ClickHouse HTTP API — FTACS uses the clickhouse-jdbc HTTP client

Hazelcast

5701, 5711

TCP

Outbound: Hazelcast client and member discovery

FTACS

8080, 8181, 8443

TCP

Inbound from CPE devices: TR-069 (CWMP) sessions, SOAP, REST

FTACS

1883, 8883

TCP

Inbound from CPE devices: MQTT and MQTT over TLS

FTACS

8025, 4430, 18881, 18882, 18891, 61617, 61618

TCP

Inbound from CPE devices: USP (TR-369) — WebSocket, MQTT, Bulk Data, STOMP

FTACS

5783, 5784

UDP

Inbound from CPE devices: USP CoAP and CoAP over DTLS

FTACS

5680—​5685

UDP

Inbound from CPE devices: LWM2M Bootstrap and LWM2M Server

FTACS

8080

TCP

Inbound from monitoring: Prometheus scrapes /actuator/prometheus

For a quick connectivity check from any host:

# Replace with target host IP and port
nc -zv <target-ip> <port>

3.1. Docker Networking

The compose file in Docker Compose puts FTACS on the ft-network bridge with container_name: ftacs, so peers on that same bridge reach it as ftacs and need no firewall rule. The infrastructure services are addressed the same way when they share the bridge — that is what the comment on DB_HOST in the environment file means by "use the container name if it shares this bridge".

Peers on another host must be given a routable address instead: DB_HOST, HZ_MEMBERS and CLICKHOUSE_HOST take an IP or DNS name, and every port in the table above has to be open through the firewall. Mixing the two is the usual cause of a container that starts and then cannot reach its database: a container name only resolves inside the bridge that defines it.

If a peer runs on the same host but outside Docker, use host.docker.internal rather than localhost — inside a container localhost is the container itself:

DB_HOST=host.docker.internal
HZ_MEMBERS=host.docker.internal:5701
CLICKHOUSE_HOST=host.docker.internal

host.docker.internal resolves to the host’s gateway address. On Docker Desktop (macOS, Windows) it works out of the box; on Linux it exists only if the container maps it, which the compose file in Docker Compose does:

extra_hosts:
  - "host.docker.internal:host-gateway"

The alternative is the host’s real LAN IP address.

4. Registry Authentication

All FTACS images are pulled from hub.friendly-tech.com. Authenticate once per host before the first docker compose up.

docker login hub.friendly-tech.com

Enter the read-only pull credentials when prompted:

Field Value

Username

readonly

Password

fokxuw-fymte1-taSxyc

The readonly account provides pull-only access to the published images. It cannot push.

Alternatively, log in non-interactively:

echo "fokxuw-fymte1-taSxyc" | docker login hub.friendly-tech.com -u readonly --password-stdin

Verify authentication:

docker info | grep -A 5 Registry

4.1. Offline Servers

When the FTACS host cannot reach hub.friendly-tech.com, pull the image on a machine that does have registry access, export it to an archive, transfer it, and load it on the offline host.

The connected machine can run Linux, macOS, or Windows — commands are given for both shells below. It does not need to be the same platform as the offline host.

An explicit --platform matching the offline host’s architecture is required. The images in Harbor are multi-arch (linux/amd64, linux/arm64); without --platform, docker pull selects the host architecture, which may not match the target. On an Apple Silicon (arm64) Mac or an arm64 Windows machine without --platform, the resulting archive will be arm64 and will fail with a platform does not match warning on amd64 servers. The examples below use linux/amd64; step 1 shows how to read the correct value off the offline host.

  1. On the offline host, find out which architecture it runs — this is the value you will pass as PLATFORM below. Ask Docker itself, since it reports what the daemon will actually accept:

    docker version --format '{{.Server.Arch}}'

    If Docker is not installed there yet, use the operating system instead — uname -m on Linux, or echo $env:PROCESSOR_ARCHITECTURE in PowerShell on Windows. Map the result:

    docker version reports uname -m / Windows reports Use as PLATFORM

    amd64

    x86_64 / AMD64

    linux/amd64

    arm64

    aarch64 / ARM64

    linux/arm64

    On Windows with Docker Desktop, {{.Server.Arch}} reports the architecture of the Linux VM that actually runs the containers — which is the value you want, not the Windows host’s own architecture.

  2. On a machine with registry access, log in. Run this on its own — it prompts for a password, so anything pasted after it on the same go would be swallowed as input:

    docker login hub.friendly-tech.com
  3. Pull and export the image. Paste the whole block as-is; the only lines to change are PLATFORM and TAG on top.

    Linux / macOS (bash):

    PLATFORM=linux/amd64
    TAG=latest
    
    docker pull --platform "$PLATFORM" "hub.friendly-tech.com/ftacs/ftacs:$TAG"
    docker save "hub.friendly-tech.com/ftacs/ftacs:$TAG" | gzip > "ftacs-$TAG.tar.gz"

    Windows (PowerShell):

    $PLATFORM = "linux/amd64"
    $TAG = "latest"
    
    docker pull --platform $PLATFORM "hub.friendly-tech.com/ftacs/ftacs:$TAG"
    docker save -o "ftacs-$TAG.tar" "hub.friendly-tech.com/ftacs/ftacs:$TAG"

    On Windows, always write the archive with docker save -o <file>. Piping or redirecting docker save from PowerShell (docker save …​ > file.tar) corrupts the archive, because the PowerShell pipeline re-encodes the stream as text instead of passing raw bytes. docker load then fails with unexpected EOF or invalid tar header. To compress for transfer, use the bundled tar.exe (Windows 10 1803+ / Server 2019+): tar.exe -czf ftacs.tar.gz ftacs-$TAG.tar.

  4. Transfer the archive to the offline host, together with compose.yml, .env, ftacs/.env, and the three files in ftacs/config/.

  5. On the offline host, load the archive and start the service. Set TAG to the same value you used above:

    Linux (bash):

    TAG=latest
    
    gzip -dc "ftacs-$TAG.tar.gz" | docker load
    docker compose up -d ftacs

    Windows (PowerShell):

    $TAG = "latest"
    
    docker load -i "ftacs-$TAG.tar"
    docker compose up -d ftacs

Confirm the image is present before starting, so a missing or mis-architected image fails here rather than mid-startup:

docker images hub.friendly-tech.com/ftacs/ftacs

Upgrades use the same flow: pull the new tag on the connected machine, transfer and load the archive, then docker compose up -d ftacs.

This covers the FTACS image only. The database, ClickHouse, Hazelcast, and FT Configs Service are separate images on their own hosts — for the offline procedure covering a whole stack, see All in one server deployment — Offline Servers, which derives the image list from compose.yml itself.

5. Preparation

Create the directories FTACS needs on the host and point it at the infrastructure listed in Prerequisites.

5.1. First: Check the Existing Database (7.1.2 and later)

Applies whenever 7.1.2 or later is brought up against a database that already contains devices — an upgrade, a re-install on an existing schema, or a migration of the database to a new host. A fresh, empty database needs none of this.

Release 7.1.2 carries a data-backfill migration: unlike a schema change, it rewrites existing rows, so its cost scales with the size of the cpe table. Before anything else, find out how big that is:

SELECT COUNT(*) FROM cpe;
Devices in cpe What this means for the installation

Up to ~1 million

Nothing to prepare. The migration completes in seconds during startup. Continue with Directory Structure.

More than ~1 million

A manual step is required before starting the new version, and the cluster must be stopped for the upgrade rather than updated node by node. Read Upgrading a Large Installation now — it explains the backfill script, the verification queries and the failure modes of skipping it.

On a large installation, starting 7.1.2 without that preparation is not merely slow: the whole cpe table is closed for writes while the migration runs, and a migration killed mid-transaction leaves the Liquibase lock held, so every later start fails until it is cleared by hand. See Upgrading a Large Installation.

5.2. Directory Structure

mkdir -p /usr/local/ft-system/ftacs/{config,logs}
cd /usr/local/ft-system

5.2.1. Directory Layout

FTACS is one slice of the platform layout described in Prepare Working Directory — the stack root holds compose.yml and the shared .env, and each service owns a directory below it:

/usr/local/ft-system/
├── compose.yml                             # from FT_DISK
├── .env                                    # shared stack environment
└── ftacs/
    ├── .env                                # per-service environment
    ├── sensitive_params.xml                # optional, mounted separately
    ├── config/                             # -> /opt/FTACS/conf (read-only)
    │   ├── ftacs.keystore
    │   ├── hazelcast-client.xml
    │   └── logback-spring.xml
    └── logs/                               # -> /opt/FTACS/logs
Path Content Backup

compose.yml

Stack definition; the FTACS service block is shown in Deployment

Yes

.env

Shared stack environment — database connection, ClickHouse, Hazelcast, timezone

Yes

ftacs/.env

FTACS-only environment — published host ports, HikariCP pool sizes, JVM settings

Yes

ftacs/config/

The three files FTACS reads from disk, mounted read-only at /opt/FTACS/conf

Yes

ftacs/sensitive_params.xml

Optional; mounted individually into /opt/FTACS/app/WEB-INF/lib/, not part of config/ — see Sensitive Parameters

Yes

ftacs/logs/

Application file logs, mounted at /opt/FTACS/logs; written by the container at runtime

No

Download the following from FT_DISK:

Component FT_DISK Target directory Files

FTACS environment

FT_DISK — ftacs

ftacs/.env

Only one file: the .env in the folder root

FTACS configuration

FT_DISK — ftacs-conf

ftacs/config/

Only three files: ftacs.keystore, hazelcast-client.xml, logback-spring.xml

The ftacs/config/ directory is mounted read-only at /opt/FTACS/conf inside the container and holds only the three files FTACS must read from disk before it can reach anything else:

File Purpose

ftacs.keystore

SSL keystore for the HTTPS ports

hazelcast-client.xml

Hazelcast cluster client configuration — also the transport FTACS receives its runtime configuration over

logback-spring.xml

Logging configuration

Since version 6.5.2 the remaining runtime configuration lives in FT Configs Service, not on disk.

acs_configuration.xml, bulk_data.xml, ctn_info.properties, external_trace.properties, fcc.properties, force_qoe_stop.properties, ftacs_task.properties, ftacsws-roles.properties, ftacsws-users.properties, hardcoded_event_soap_template.xml, and parameterNamesCacheRules.yaml were moved there. Do not place them in ftacs/config/ — edit them through FT Configs UI instead.

FT Configs Service publishes each configuration snapshot into the shared Hazelcast cluster, and FTACS consumes it from there; there is no direct URL from FTACS to the service. This is why both must be clients of the same Hazelcast cluster — see Prerequisites.

See Configuration Guide for the reference on each configuration item, and FT Configs deployment for deploying the service that now owns them.

5.3. Environment Configuration

The stack uses two environment layers, and every service — FTACS included — reads both:

File Holds

.env

Shared by the whole stack: database connection (DB_HOST, DB_PORT, DB_USER, DB_PASSWORD), database vendor selection (DB_PROFILE, SPRING_PROFILES_ACTIVE, DB_SERVICE), ClickHouse, Hazelcast, timezone

ftacs/.env

FTACS only: published host ports, HikariCP pool sizes (FTACS_MAX_POOL, QUARTZ_MAX_POOL, CLICKHOUSE_MAX_POOL), JVM settings, SSL_KEYSTORE_PASSWORD

There is no per-vendor environment file. MySQL is the default; switching to Oracle means changing DB_PROFILE, SPRING_PROFILES_ACTIVE, DB_HOST, DB_PORT and DB_SERVICE inside the single root .env — see Switching to Oracle. compose.yml maps these generic DB_* names onto the vendor-specific names FTACS reads, through the x-db-env anchor shown in Deployment.

# /usr/local/ft-system/.env
TZ=Europe/Kyiv

# Database (existing)
DB_HOST=<db-host-ip>                       # <-- replace; use the container name if it shares this bridge
DB_PORT=3306
DB_USER=ftacs
DB_PASSWORD=<your-db-password>             # <-- replace
DB_SERVICE=XEPDB1
DB_PROFILE=mysql
SPRING_PROFILES_ACTIVE=mysql

# Hazelcast (existing)
HZ_MEMBERS=<hazelcast-host-ip>:5701        # <-- replace

# ClickHouse (existing)
CLICKHOUSE_HOST=<clickhouse-host-ip>       # <-- replace
CLICKHOUSE_PORT=9000
CLICKHOUSE_HTTP_PORT=8123
CLICKHOUSE_SCHEMA=ftacs_qoe_ui_data
CLICKHOUSE_USERNAME=ftacs
CLICKHOUSE_PASWORD=<your-ch-password>      # <-- replace; single "S" is intentional, see the note below

# FTACS
FT_ACS_JAVA_RAM=-Xms1g -Xmx4g

The port changes together with the host, not only the hostname. Across hosts you connect to the published port (the left side of ports:); inside the Docker bridge you connect to the container port (the right side). ClickHouse is 8123 → 8123 and 9000 → 9000, so both forms match here — but FTACS must always target the HTTP port 8123, never the native TCP port 9000, because it uses the clickhouse-java HTTP client.

The ClickHouse password variable is misspelled in the FTACS code: CLICKHOUSE_PASWORD, with a single "S". FT_IoT/FTAPP/ftapp-build/src/main/resources/application.yaml line 130 reads ${CLICKHOUSE_PASWORD:ftacs}, and FT_IoT/FTAPP/ftapp-build/bin/app.env line 31 sets CLICKHOUSE_PASWORD. Setting the correctly spelled CLICKHOUSE_PASSWORD has no effect on FTACS — it silently falls back to the built-in default ftacs and the ClickHouse connection fails with an authentication error. The same applies to the schema: FTACS reads CLICKHOUSE_SCHEMA, not CLICKHOUSE_DATABASE. CLICKHOUSE_PASSWORD and CLICKHOUSE_DATABASE are still meaningful for other containers in the stack (the ClickHouse server image itself and ft-device-network-service), so both spellings can legitimately appear in the same .env. Tracked in docs/decisions/LOG.md for a code fix.

The variables FTACS itself reads are defined in this repository by FT_IoT/FTAPP/ftapp-build/docker/ftacs.env (Docker), FT_IoT/FTAPP/ftapp-build/bin/app.env (manual) and FT_IoT/FTAPP/ftapp-build/docker/docker-entrypoint.sh:

Variable Description Default Required

APP_HOME

Installation root; conf/, logs/, ext/ and patches/ derive from it

/opt/ftacs in the entrypoint; the reference compose sets /opt/FTACS

No

DB_VENDOR

Database vendor; selects the active Spring profile (mysql or oracle)

mysql

No

DB_HOST

Hostname or IP of the database server

none

Yes

DB_PASSWORD

Password of the FTACS database user

none

Yes

MYSQL_PORT

MySQL listener port

3306

Yes (MySQL)

MYSQL_USER

MySQL user owning the ftacs schema

ftacs

Yes (MySQL)

ORACLE_PORT

Oracle listener port

1521

Yes (Oracle)

ORACLE_SERVICE

Oracle service name

orclpdb1 (application.yaml line 123)

Yes (Oracle)

ORACLE_USER_FTACS

Oracle user owning the ftacs schema

ftacs

Yes (Oracle)

ORACLE_USER_QUARTZ

Oracle user owning the ftacs_quartz schema

ftacs_quartz

Yes (Oracle)

HZ_MEMBERS

Hazelcast member list as host:port; the only channel to FT Configs Service

none

Yes

CLICKHOUSE_HOST

Hostname or IP of the ClickHouse server

none

Yes

CLICKHOUSE_PORT

ClickHouse port FTACS connects to. FTACS uses the HTTP client, so this must be the HTTP port 8123, not the native TCP port 9000

8123 in bin/app.env; docker/ftacs.env still ships 9000 — tracked in docs/decisions/LOG.md

Yes

CLICKHOUSE_USERNAME

ClickHouse user

ftacs

Yes

CLICKHOUSE_PASWORD

ClickHouse password. Misspelled in the code (single "S") — application.yaml line 130 reads ${CLICKHOUSE_PASWORD:ftacs}. CLICKHOUSE_PASSWORD is ignored by FTACS

ftacs

Yes

CLICKHOUSE_SCHEMA

ClickHouse database FTACS writes QoE data to. FTACS does not read CLICKHOUSE_DATABASE

ftacs_qoe_ui_data

Yes

HTTP_PORTS

HTTP listener ports; comma or space separated

8080,8181 in application.yaml and bin/app.env; the Docker entrypoint falls back to 8080 when the variable is unset

No

HTTPS_PORTS

HTTPS listener port passed to the application as app.web.https-ports

8443

No

SSL_KEYSTORE

Path to the Java keystore serving the TLS ports

empty

No

SSL_KEYSTORE_PASSWORD

Password of that keystore

empty

No

WEB_TR069_NO_AUTH

Servlet path of the unauthenticated TR-069 endpoint

/ftacs/ACS

No

WEB_TR069_BASIC

Servlet path of the HTTP Basic TR-069 endpoint

/ftacs-basic/ACS

No

WEB_TR069_DIGEST

Servlet path of the HTTP Digest TR-069 endpoint

/ftacs-digest/ACS

No

JAVA_OPTS

JVM options passed to the java command

-Xms1g -Xmx4g -Duser.timezone=Europe/Kiev

No

The names used in the shared .env above (DB_PORT, DB_USER, DB_SERVICE, DB_PROFILE, CLICKHOUSE_HTTP_PORT, FT_ACS_JAVA_RAM) come from the portal-wide deployment bundle and do not appear in this repository’s env files; verify with FT DevOps which set applies to your build.

Create ftacs/.env with the per-service variables documented in Configuration Guide — Environment Variables; the full platform-wide reference is FTACS Environment Variables.

Click to expand a commented ftacs/.env reference
# =============================================================================
# HikariCP Connection Pools
# =============================================================================

# Shared pool defaults (apply to all datasources unless overridden below)
DB_MAX_POOL_SIZE=10
DB_MIN_IDLE=5
DB_CONNECTION_TIMEOUT_MS=30000

# Main FTACS datasource pool. Handles CPE management, parameters,
# provisioning operations. Needs a larger pool due to high concurrency.
FTACS_MAX_POOL=30
FTACS_MIN_POOL=5

# Quartz job scheduler datasource. Used for scheduled task persistence
# (UpdateGroup, QoE, Events). Lower load than the main datasource.
QUARTZ_MAX_POOL=10
QUARTZ_MIN_POOL=2

# ClickHouse analytics datasource. Used for QoE data and statistics.
CLICKHOUSE_MAX_POOL=10
CLICKHOUSE_MIN_POOL=2

# =============================================================================
# ACS Deployment Mode
# Controls which functionality is active. Used for distributed deployments
# where provisioning and management run on separate nodes.
# =============================================================================

# ALL    = full functionality (provisioning + management) -- single-node default
# PROV   = provisioning only (device bootstrap, session handling)
# MNGMNT = management only (device monitoring, configuration)
# In MNGMNT mode, FTACS connects to a separate Hazelcast "provision" cluster.
# FTACS_MODE=ALL

# =============================================================================
# Server IP Address
# Advertised to CPE devices for callbacks and connection requests.
# Auto-detected from the hostname if not set. Set explicitly in Docker/NAT
# environments where auto-detection returns an unreachable address.
# =============================================================================

# IPv4 address advertised to CPE devices (e.g. 192.168.1.100)
# FTACS_IP_ADDRESS=

# IPv6 address for dual-stack deployments. Brackets are added automatically.
# FTACS_IPV6_ADDRESS=

# =============================================================================
# Event Subscription Parameters
# Control when parameter change notifications are sent to subscriber URLs.
# =============================================================================

# 0 = notify on every parameter update (EACH_VALUE trigger)
# 1 = notify only when the value actually changes (VALUE_CHANGE trigger)
# FTACS_SUBSCRIPTION_PARAMETER_ONLY_VALUE_CHANGE=0

# 0 = include all parameter values (including null/empty)
# 1 = skip notifications for null or empty values
# FTACS_SUBSCRIPTION_PARAMETER_NOT_NULL_ONLY=0

# =============================================================================
# TR-069 Endpoint URL Mappings
# Each defines the servlet URL path for a different authentication mode.
# Multiple paths can be comma-separated.
# =============================================================================

# No authentication -- CPE connects without credentials
WEB_TR069_NO_AUTH=/ftacs/ACS

# HTTP Basic Authentication (RFC 7617) -- CPE sends base64(user:password)
WEB_TR069_BASIC=/ftacs-basic/ACS

# HTTP Digest Authentication (RFC 2617) -- challenge-response, no plaintext password
WEB_TR069_DIGEST=/ftacs-digest/ACS

# =============================================================================
# MQTT Protocol (optional)
# Activates the embedded MQTT broker for TR-369/USP MQTT-MTP transport.
# =============================================================================

MQTT_ENABLE=false

# Additional MQTT transport connector URI (ActiveMQ format).
# Format: mqtt+nio://<host>:<port> or mqtt+nio+ssl://<host>:<port>
# FT_MQTT_ADDITIONAL_CONNECTOR_URI=

# Active MQTT API versions. Incoming requests are validated against this list.
# MQTT_ACTIVE_API=api-v1,api-v2

# =============================================================================
# USP Protocol (optional)
# =============================================================================

# Path to the USP data model XML specification (BBF TR-181-2).
# Loaded from the filesystem first; falls back to a classpath resource.
# USP_SERVER_DM_PATH=tr-181-2-16-0-usp-full.xml

# =============================================================================
# X509 Certificates -- USP TLS/DTLS (optional)
# Used for secure USP MTP connections (WebSocket, STOMP, CoAP over DTLS).
# Format: JKS or PKCS12 keystore files.
# =============================================================================
# FT_USP_X509_KEYSTORE_PATH=        # Server keystore with private key and certificate
# FT_USP_X509_KEYSTORE_PASSWORD=    # Keystore file password
# FT_USP_X509_KEYSTORE_PK_PASSWORD= # Private key password (if it differs)
# FT_USP_X509_TRUSTSTORE_PATH=      # Truststore with trusted CA certificates
# FT_USP_X509_TRUSTSTORE_PASSWORD=  # Truststore file password

# =============================================================================
# X509 Certificates -- MQTT TLS (optional)
# Used for MQTTS on port 8883. Enables mutual TLS when a truststore is set.
# =============================================================================
# FT_MQTT_X509_KEYSTORE_PATH=
# FT_MQTT_X509_KEYSTORE_PASSWORD=
# FT_MQTT_X509_PK_PASSWORD=
# FT_MQTT_X509_TRUSTSTORE_PATH=
# FT_MQTT_X509_TRUSTSTORE_PASSWORD=

# =============================================================================
# General Async Executor (used by @Async methods without an explicit executor)
# =============================================================================

# Minimum threads always running. Idle core threads are reclaimed after
# ASYNC_KEEPALIVE seconds (allowCoreThreadTimeout=true).
ASYNC_CORE=10

# Maximum threads under load. When ASYNC_QUEUE=0, threads spawn immediately
# up to this limit before tasks go to an external 10,000-task overflow queue.
ASYNC_MAX=200

# Internal queue capacity before spawning new threads.
# 0 = no internal queue (SynchronousQueue); tasks spawn threads immediately.
# An external OverflowingExecutor with a 10,000-task queue wraps the pool as a
# safety net, blocking on overflow to provide backpressure.
ASYNC_QUEUE=0

# Seconds an idle thread stays alive before termination.
ASYNC_KEEPALIVE=60

# Thread pool size for @Scheduled methods (virtual threads enabled).
# Handles periodic tasks: statistics flush (every 60s), counter reset (daily).
# Quartz jobs use a separate JDBC-based scheduler, not this pool.
SCHED_POOL=300

# =============================================================================
# HikariCP Tuning (optional)
# Applies to ALL three datasources (ftacs, quartz, clickhouse).
# =============================================================================

# Max time (ms) a connection can sit idle before removal. Default: 60000
# HIKARI_IDLE_TIMEOUT_MS=60000

# Max time (ms) to wait for a connection from the pool. Default: 45000
# HIKARI_CONN_TIMEOUT_MS=45000

# Max age (ms) of a connection before forced rotation. Default: 180000
# Must be greater than HIKARI_IDLE_TIMEOUT_MS.
# HIKARI_MAX_LIFETIME_MS=180000

# Max time (ms) for the connection validation query (SELECT 1). Default: 3000
# HIKARI_VALIDATION_TIMEOUT_MS=3000

# Register HikariCP pool metrics as JMX MBeans (true/false).
# HIKARI_REGISTER_MBEANS=false

5.4. TLS Keystore (optional)

FTACS serves its HTTPS port (8443) and the other TLS ports from a single Java keystore, ftacs.keystore. The bundle on FT_DISK ships a self-signed keystore — suitable for testing only. This step applies to both deployment methods; only the target directory and the file holding the password differ.

To use a CA-issued or otherwise custom certificate:

  1. Generate the keystore (or import an existing certificate into one):

    keytool -genkey -alias ftacs -keyalg RSA -keysize 2048 -keystore ftacs.keystore
  2. Place ftacs.keystore in the FTACS configuration directory:

    • Docker: ftacs/config/ftacs.keystore (mounted read-only at /opt/FTACS/conf).

    • Manual: conf/ftacs.keystore in the distribution package.

  3. Set the keystore password. SSL_KEYSTORE_PASSWORD must match the password used when the keystore was created:

    • Docker: in ftacs/.env.

    • Manual: in bin/app.env, alongside SSL_KEYSTORE=${CONF_DIR}/ftacs.keystore.

Both paths resolve to ${CONF_DIR}/ftacs.keystore inside the application — /opt/FTACS/conf in the container, ${APP_HOME}/conf in a manual deployment. Replacing the keystore requires a restart; it is not hot-reloaded.

6. Deployment

Choose one method: Docker Compose (recommended) or the distribution package on a JDK 25 host.

6.1. Startup Dependencies

FTACS has no depends_on on its infrastructure — the database, ClickHouse, Hazelcast, and FT Configs Service live outside this stack, so Compose cannot gate on their health. All of them must be up before FTACS launches: Liquibase applies the schema migrations on first start, the Hazelcast client connects during context initialisation, and FT Configs Service must already have published its configuration snapshots into Hazelcast.

Wait for each of them explicitly before starting the service:

# Wait for the database
until nc -z <db-host-ip> 3306; do sleep 2; done

# Wait for ClickHouse (HTTP port)
until curl -sf http://<clickhouse-host-ip>:8123/ping; do sleep 2; done

# Wait for Hazelcast
until nc -z <hazelcast-host-ip> 5701; do sleep 2; done

# Wait for FT Configs Service to be healthy (it warms up its Hazelcast snapshots on start)
until curl -sf http://<configs-service-host-ip>:8087/configs-service/actuator/health; do sleep 2; done

# Then start FTACS
docker compose up -d ftacs

FTACS itself restarts cleanly at any time once the infrastructure is up — restart: unless-stopped reconnects it after a host reboot.

6.2. Docker Compose

Click to expand compose.yml (FTACS)
x-db-env: &db-env
  DB_VENDOR: ${DB_PROFILE}
  DB_HOST: ${DB_HOST}
  MYSQL_PORT: ${DB_PORT}
  MYSQL_USER: ${DB_USER}
  MYSQL_SCHEMA_FTACS: ftacs
  MYSQL_SCHEMA_QUARTZ: ftacs_quartz
  FTACS_PASSWORD: ${DB_PASSWORD}
  QUARTZ_PASSWORD: ${DB_PASSWORD}

services:

  ftacs:
    image: hub.friendly-tech.com/ftacs/ftacs:latest
    container_name: ftacs
    # The database, ClickHouse and Hazelcast are external to this stack, so there
    # is no depends_on. They must be healthy before ftacs starts -- DB_HOST,
    # CLICKHOUSE_HOST and HZ_MEMBERS in .env point at them.
    env_file:
      - .env
      - ./ftacs/.env
    environment:
      <<: *db-env
      USE_NOHUP: 0
      # ftacs uses the clickhouse-java HTTP client, so it must target the HTTP
      # port (8123), not the native TCP port (9000) that CLICKHOUSE_PORT defaults
      # to in the shared .env. Without this remap, ClickHouse connections fail.
      CLICKHOUSE_PORT: "${CLICKHOUSE_HTTP_PORT:-8123}"
      JAVA_OPTS: "${FT_ACS_JAVA_RAM:--Xms1g -Xmx4g} -Duser.timezone=${TZ:-Europe/Kyiv}"
    ports:
      # TR-069 / REST / SOAP
      - "${FTACS_HTTP_PORT:-8080}:8080"
      - "${FTACS_HTTPS_PORT:-8443}:8443"
      - "${ACS_HTTP_ADDITIONAL:-8181}:8181"
      # MQTT
      - "${ACS_MQTT:-1883}:1883"
      - "${ACS_MQTT_TLS:-8883}:8883"
      # USP (TR-369)
      - "${ACS_USP_WS:-8025}:8025"
      - "${ACS_USP_WSS:-4430}:4430"
      - "${ACS_USP_MQTT:-18881}:18881"
      - "${ACS_USP_MQTT_TLS:-18882}:18882"
      - "${ACS_USP_BULK:-18891}:18891"
      - "${ACS_USP_STOMP:-61617}:61617"
      - "${ACS_USP_STOMP_TLS:-61618}:61618"
      - "${ACS_USP_CoAP:-5783}:5783/udp"
      - "5784:5784/udp"
      # LWM2M Bootstrap
      - "${BS_PORT_NOSEC:-5680}:5680/udp"
      - "${BS_PORT_PSK:-5681}:5681/udp"
      - "${BS_PORT_X509:-5682}:5682/udp"
      - "${BS_TCP_PORT_NOSEC:-5880}:5880"
      # LWM2M Server
      - "${LWM2M_PORT_NOSEC:-5683}:5683/udp"
      - "${LWM2M_PORT_PSK:-5684}:5684/udp"
      - "${LWM2M_PORT_X509:-5685}:5685/udp"
      - "${LWM2M_TCP_PORT_NOSEC:-5883}:5883"
    volumes:
      - ./ftacs/config:/opt/FTACS/conf:ro
      - ./ftacs/logs:/opt/FTACS/logs
    healthcheck:
      test: ["CMD", "pgrep", "-f", "com.friendly.acs.FtAcsApp"]
      interval: 20s
      timeout: 5s
      retries: 10
      start_period: 60s
    restart: unless-stopped
    stop_grace_period: 45s
    extra_hosts:
      - "host.docker.internal:host-gateway"   # peers running on this host outside Docker
    networks:
      - ft-network

For Oracle, keep the same two env_file entries — there is no per-vendor environment file — set the vendor variables in the root .env as described in Switching to Oracle, and change the x-db-env anchor to Oracle-specific mappings:

x-db-env: &db-env
  DB_VENDOR: ${DB_PROFILE}
  DB_HOST: ${DB_HOST}
  ORACLE_PORT: ${DB_PORT}
  ORACLE_SERVICE: ${DB_SERVICE}
  ORACLE_USER_FTACS: ${DB_USER}
  ORACLE_USER_QUARTZ: ftacs_quartz
  FTACS_PASSWORD: ${DB_PASSWORD}
  QUARTZ_PASSWORD: ${DB_PASSWORD}

The anchors above map the shared .env names onto the names FTACS actually reads. The database host is DB_HOST for both vendors (application.yaml line 94, vars.host) — there is no MYSQL_HOST or ORACLE_HOST. Schemas and users are per role: MYSQL_SCHEMA_FTACS / MYSQL_SCHEMA_QUARTZ and ORACLE_USER_FTACS / ORACLE_USER_QUARTZ (application.yaml lines 115-116 and 122-123) — there is no MYSQL_SCHEMA or ORACLE_USER. The database password reaches the datasources as FTACS_PASSWORD and QUARTZ_PASSWORD (application.yaml lines 95-96); bin/app.env lines 101-102 derive both from DB_PASSWORD, but that derivation lives in the env file, not in the application, so a compose file must set them explicitly.

6.2.1. Sensitive Parameters

To run FTACS with a sensitive_params.xml file, mount it from the host into the container as an additional volume:

    volumes:
      - ./ftacs/config:/opt/FTACS/conf:ro
      - ./ftacs/logs:/opt/FTACS/logs
      - ./ftacs/sensitive_params.xml:/opt/FTACS/app/WEB-INF/lib/sensitive_params.xml:ro

6.2.2. Start the Stack

Start FTACS only after the database, ClickHouse, Hazelcast, and FT Configs Service accept connections — see Startup Dependencies.

cd /usr/local/ft-system
docker login hub.friendly-tech.com
docker compose up -d ftacs
docker compose ps
# FTACS takes 1-2 minutes to start
docker compose logs -f ftacs | grep -i "started"
curl -s -u <ws-user>:<ws-password> http://localhost:8080/acsstats

To run FTACS as a single container without Compose:

cd /usr/local/ft-system

docker run -d \
  --name ftacs \
  --restart unless-stopped \
  --env-file .env --env-file ./ftacs/.env \
  -p 8080:8080 \
  -p 8443:8443 \
  -v ./ftacs/config:/opt/FTACS/conf:ro \
  -v ./ftacs/logs:/opt/FTACS/logs \
  hub.friendly-tech.com/ftacs/ftacs:latest

docker run publishes only the two ports shown above. Add a -p flag for every protocol your devices actually use — see Port Reference.

6.3. Manual (dist package)

FTACS runs directly on a JDK 25 host from the distribution package. The database, ClickHouse, Hazelcast, and FT Configs Service must be reachable before starting the application — see Prerequisites.

6.3.1. Directory Structure

The extracted distribution package has this structure:

dist/
├── app/                 # Compiled application (exploded WAR) -- do NOT modify
│   ├── META-INF/        # JAR/WAR manifest and persistence descriptors
│   ├── org/             # Java bytecode packages (ACS core classes)
│   └── WEB-INF/         # Spring Boot web configuration and libraries
│
├── bin/                 # Executable scripts and environment definitions
│   ├── app.env          # Main environment file (DB, ports, JVM, etc.)
│   ├── env_to_cmd.cmd   # Helper: .env -> Windows CMD variables
│   ├── start.cmd        # Windows startup script
│   ├── start.sh         # Linux startup script
│   ├── stop.cmd         # Windows shutdown script
│   └── stop.sh          # Linux shutdown script
│
├── conf/                # Configuration files (editable by operators)
│   ├── ftacs.keystore   # SSL keystore for the HTTPS ports
│   ├── hazelcast-client.xml  # Hazelcast client -- also the FT Configs Service channel
│   └── logback-spring.xml    # Logging configuration
│                        # everything else moved to FT Configs Service in 6.5.2
│
├── ext/                 # External extensions / JARs loaded dynamically
├── flyway/              # ClickHouse migration scripts (managed by Flyway)
├── liquibase/           # Database migration scripts (managed by Liquibase)
├── logs/                # Runtime logs (created at startup)
└── patches/             # Hot-fix patches
  • app/ is static — do not modify; changes belong in conf/.

  • bin/ scripts resolve every path relative to APP_HOME, defined in app.env.

  • conf/ is safe to edit and should be version-controlled in ops repositories.

  • logs/ may be mounted to external storage for persistent log retention.

  • liquibase/ drives schema migrations; changes here must match release cycles.

  • ext/ enables plugin-like JARs or vendor integrations without rebuilding.

6.3.2. Install the JDK

Linux
wget https://download.oracle.com/java/25/latest/jdk-25_linux-x64_bin.tar.gz
tar -xzf jdk-25_linux-x64_bin.tar.gz
sudo mv jdk-25 /opt/jdk-25

# Add to ~/.bashrc or /etc/profile
export JAVA_HOME=/opt/jdk-25
export PATH=$JAVA_HOME/bin:$PATH

java -version
Windows
  1. Download the JDK 25 installer from the Oracle website.

  2. Run the installer (e.g. jdk-25_windows-x64_bin.exe).

  3. Set the JAVA_HOME environment variable to C:\Program Files\Java\jdk-25.

  4. Add %JAVA_HOME%\bin to PATH.

  5. Verify: open Command Prompt and run java -version.

Then extract the distribution package:

# Linux
tar -xzf ftacs-dist-<version>.tar.gz
cd dist/

# Windows
unzip ftacs-dist-<version>.zip
cd dist\

6.3.3. Environment Configuration

Edit bin/app.env in the extracted distribution package.

# === Base directories (all other paths derive from APP_HOME) ===
APP_HOME=/usr/local/ftacs/                            # <-- replace with your dist path
CONF_DIR=${APP_HOME}/conf
LOG_DIR=${APP_HOME}/logs
EXT_DIR=${APP_HOME}/ext
PATCHES_DIR=${APP_HOME}/patches
DB_LIQUIBASE_DIR=${APP_HOME}/liquibase

# === Database connection ===
DB_HOST=localhost                                     # <-- replace
DB_PASSWORD=<your-db-password>                        # <-- replace

# --- MySQL (default) ---
DB_VENDOR=mysql
MYSQL_PORT=3306
MYSQL_USER=ftacs

# --- Oracle (uncomment to switch) ---
#DB_VENDOR=oracle
#ORACLE_PORT=1521
#ORACLE_SERVICE=orclpdb1                              # <-- replace with your PDB name
#ORACLE_USER_FTACS=ftacs
#ORACLE_USER_QUARTZ=ftacs_quartz

# === ClickHouse ===
CLICKHOUSE_HOST=localhost                             # <-- replace
CLICKHOUSE_PORT=8123
CLICKHOUSE_SCHEMA=ftacs_qoe_ui_data
CLICKHOUSE_USERNAME=ftacs
CLICKHOUSE_PASWORD=<your-ch-password>                 # <-- replace

# === Hazelcast ===
HZ_MEMBERS=127.0.0.1:5701                             # <-- replace with cluster addresses
HZ_CLIENT_CONFIG=${CONF_DIR}/hazelcast-client.xml

# === TR-069 device URL mappings ===
WEB_TR069_NO_AUTH=/ftacs/ACS
WEB_TR069_BASIC=/ftacs-basic/ACS
WEB_TR069_DIGEST=/ftacs-digest/ACS

# === Web / SSL ===
HTTP_PORTS=8080,8181
HTTPS_PORTS=8443
SSL_KEYSTORE=${CONF_DIR}/ftacs.keystore
SSL_KEYSTORE_PASSWORD=<your-keystore-password>        # <-- replace

# === JVM options ===
JAVA_OPTS='-Xms1g -Xmx4g'

# === Additional configuration ===
SPRING_CONFIG_EXTRA=file:${CONF_DIR}/
LOGBACK_CONFIG=${CONF_DIR}/logback-spring.xml
CXF_PATH=/ACSServer-ACS
QUARTZ_PASSWORD=${DB_PASSWORD}

Set APP_HOME to the absolute path of your dist directory.

  • Linux example: APP_HOME=/usr/local/ftacs/

  • Windows example: APP_HOME=D:/ftacs/dist/

On Windows, use forward slashes (/) and include the trailing slash. Avoid double slashes.

6.3.4. Deploy and Verify

Linux
cd bin/
chmod +x start.sh
./start.sh

tail -f ../logs/server.log
Windows
cd bin
start.cmd

type ..\logs\server.log

To stop the application, run ./stop.sh (or stop.cmd) from bin/.

7. Verification

7.1. Startup Log

Wait 30—​60 seconds after start, then check the logs:

# Docker Compose
docker compose logs ftacs | tail -20

# Docker standalone
docker logs ftacs | tail -20

# Manual deployment
tail -f ${APP_HOME}/logs/server.log

A successful startup produces the following output (timestamps, version, and build number vary):

INFO  [c.f.a.config.AcsAppConfig] (main)
      ==================================================
          HTTP ports: 8080,8181
          HTTPS ports: 8443
          TR069 Device URLs:
              NOAUTH: /ftacs/ACS
              BASIC: /ftacs-basic/ACS
              DIGEST: /ftacs-digest/ACS
          WebServices:
              /ACSServer-ACS/ACSWebService
              /ACSServer-ACS/ACSWebServiceAdditional
              /ACSServer-ACS/ACSWebServiceConfiguration
              /ACSServer-ACS/ACSWebServiceEventSubscription
          Swagger UI: /rest/swagger-ui
      ==================================================

INFO  [c.f.i.c.s.FTIOTApplicationListener] (main) ==================== Friendly IOT Management Server V<version> Build <build>(<build-date>) started ====================
INFO  [com.friendly.acs.FtAcsApp] (main) Started FtAcsApp in <seconds> seconds (process running for <seconds>)

The last two lines are the "ready" markers: the FTIOTApplicationListener banner reports the running version and build, and Started FtAcsApp confirms the process is up.

ClusterManager: detected failed or restarted instances during startup is normal Quartz scheduler recovery and can be ignored.

Liquibase migration messages go to liquibase.log, not server.log.

The startup banner is produced by FTIOTApplicationListener; the version and build number are read from version.xml inside the container.

7.2. Endpoint Checks

# Application statistics -- HTTP Basic, use a web service user
curl -u <ws-user>:<ws-password> http://localhost:8080/acsstats

# TR-069 ACS endpoint -- POST only, so probe it with POST
curl -s -o /dev/null -w '%{http_code}\n' -X POST http://localhost:8080/ftacs/ACS

# SOAP WSDL -- HTTP Basic, use a web service user
curl -u <ws-user>:<ws-password> "http://localhost:8080/ACSServer-ACS/ACSWebService?wsdl"

# Swagger UI -- the entry point redirects, so follow the redirect
curl -sIL -o /dev/null -w '%{http_code}\n' http://localhost:8080/rest/swagger-ui

/acsstats and the SOAP WSDL must return HTTP 200. /ftacs/ACS is POST-only (see HTTP Endpoints) — any status other than 404 means the endpoint is mapped and served; a curl -I HEAD probe returns 405 on a healthy deployment, which is not a fault. /rest/swagger-ui redirects to the UI page, so it reaches 200 only when the redirect is followed (-L). /acsstats and the SOAP web services sit behind the ftacsws HTTP Basic realm; their users are published by FT Configs Service, so a 401 means either wrong credentials or no published configuration — see FTACS Starts but Has No Configuration. The web interface is reachable at http://localhost:8080 and https://localhost:8443.

7.3. Database Connectivity

Check that Liquibase created the schema tables:

MySQL
USE ftacs;
SHOW TABLES;
Oracle
SELECT table_name FROM user_tables ORDER BY table_name;

You should see multiple tables created by the Liquibase migrations.

8. Port Reference

Ports are written as published → container. Across hosts you connect to the published port; inside the Docker bridge you connect to the container port. All published ports are configurable through the .env variables shown in the compose file above.

8.1. HTTP / HTTPS

Port Protocol Purpose Exposure

8080 → 8080

HTTP

Main web interface: SOAP, REST, TR-069, statistics, bulk data (FTACS_HTTP_PORT)

Public (devices)

8181 → 8181

HTTP

Secondary HTTP port, the same services as 8080 (ACS_HTTP_ADDITIONAL)

Public (devices)

8443 → 8443

HTTPS

Secure web interface (FTACS_HTTPS_PORT)

Public (devices)

8.2. MQTT

Port Protocol Purpose Exposure

1883 → 1883

MQTT

MQTT broker, plain (ACS_MQTT)

Public (devices)

8883 → 8883

MQTT over TLS

MQTT broker, TLS (ACS_MQTT_TLS)

Public (devices)

8.3. USP (TR-369)

Port Protocol Purpose Exposure

8025 → 8025

TCP

USP WebSocket, plain (ACS_USP_WS)

Public (devices)

4430 → 4430

TCP

USP WebSocket Secure, WSS (ACS_USP_WSS)

Public (devices)

18881 → 18881

TCP

USP MQTT, plain (ACS_USP_MQTT)

Public (devices)

18882 → 18882

TCP

USP MQTT, TLS (ACS_USP_MQTT_TLS)

Public (devices)

18891 → 18891

TCP

USP Bulk Data receiver, plain (ACS_USP_BULK)

Public (devices)

61617 → 61617

TCP

USP STOMP, plain (ACS_USP_STOMP)

Public (devices)

61618 → 61618

TCP

USP STOMP, TLS (ACS_USP_STOMP_TLS)

Public (devices)

5783 → 5783

UDP

USP CoAP, plain (ACS_USP_CoAP)

Public (devices)

5784 → 5784

UDP

USP CoAP, DTLS

Public (devices)

8.4. LWM2M

Port Protocol Purpose Exposure

5680 → 5680

UDP

Bootstrap server, no security (BS_PORT_NOSEC)

Public (devices)

5681 → 5681

UDP

Bootstrap server, PSK (BS_PORT_PSK)

Public (devices)

5682 → 5682

UDP

Bootstrap server, X.509 (BS_PORT_X509)

Public (devices)

5880 → 5880

TCP

Bootstrap server, no security (BS_TCP_PORT_NOSEC)

Public (devices)

5683 → 5683

UDP

LWM2M server, no security (LWM2M_PORT_NOSEC)

Public (devices)

5684 → 5684

UDP

LWM2M server, PSK (LWM2M_PORT_PSK)

Public (devices)

5685 → 5685

UDP

LWM2M server, X.509 (LWM2M_PORT_X509)

Public (devices)

5883 → 5883

TCP

LWM2M server, no security (LWM2M_TCP_PORT_NOSEC)

Public (devices)

8.5. Outbound Connections

Ports FTACS dials on the existing infrastructure. These are the published ports of those services — confirm them against whoever operates them, and set the matching variables in .env.

Destination Port Protocol Purpose

Relational database

3306 (Oracle: 1521)

TCP

ACS and Quartz schemas (DB_HOST, DB_PORT)

ClickHouse HTTP interface

8123

HTTP

QoE and statistics data (CLICKHOUSE_HOST, CLICKHOUSE_HTTP_PORT)

Hazelcast cluster

5701, 5711

TCP

Distributed cache and the FT Configs Service configuration channel (HZ_MEMBERS)

8.6. HTTP Endpoints

Method Path Purpose Auth

GET

/acsstats

TR-069 protocol statistics

HTTP Basic (web service user)

POST

/ftacs/ACS

TR-069 endpoint

None

POST

/ftacs-basic/ACS

TR-069 endpoint

HTTP Basic (device credentials)

POST

/ftacs-digest/ACS

TR-069 endpoint

HTTP Digest (device credentials)

GET, POST

/ACSServer-ACS/ACSWebService

Primary SOAP web service; append ?wsdl for the contract

HTTP Basic (web service user)

GET, POST

/ACSServer-ACS/ACSWebServiceAdditional

Additional SOAP web service; append ?wsdl for the contract

HTTP Basic (web service user)

GET

/rest/swagger-ui

Swagger UI documentation interface

None

GET

/actuator/prometheus

Prometheus metrics scrape endpoint

None

The TR-069 endpoint paths are configurable through WEB_TR069_NO_AUTH, WEB_TR069_BASIC, and WEB_TR069_DIGEST; each accepts a comma-separated list, so endpoints can be added or removed.

Endpoint URL examples:

# No authentication
http://<your-server>:8080/ftacs/ACS

# Basic authentication
http://<your-server>:8080/ftacs-basic/ACS

# Digest authentication (HTTPS)
https://<your-server>:8443/ftacs-digest/ACS

# SOAP web service
http://<your-server>:8080/ACSServer-ACS/ACSWebService?wsdl

# Swagger UI
http://<your-server>:8080/rest/swagger-ui

Plain-text ports (no encryption): 1883, 5680—​5685, 5783, 5880, 5883, 8025, 8080, 8181, 18881, 18891, 61617.

Encrypted ports: 4430 (WSS), 5784 (CoAP DTLS), 8443 (HTTPS), 8883 (MQTT TLS), 18882 (USP MQTT TLS), 61618 (STOMP TLS).

In production, firewall the plain-text ports and route device traffic through the encrypted endpoints.

9. Stack Management

9.1. Logs

Command Description

docker compose logs -f ftacs

Follow the container log of the ftacs service

docker compose logs --tail 100 ftacs

Print the last 100 lines of the container log

tail -f ${APP_HOME}/logs/server.log

Follow the main application log in a manual deployment

# Follow logs for one service
docker compose logs -f ftacs

# Last 100 lines
docker compose logs --tail 100 ftacs

# Manual deployment
tail -f ${APP_HOME}/logs/server.log

The application writes several log files to ftacs/logs/ (Docker) or logs/ (manual):

Log file Purpose

server.log

Main application log

acsws.log

Web service calls and responses

liquibase.log

Database migration logs

sql.log

SQL query execution logs

bulk_process.log

Bulk operation processing

monitors.log

System monitoring events

db_session_killed.log

Database session management

externalTrace.log

External system integration traces

old_data_clean.log

Data cleanup operations

hikari.log

Connection pool diagnostics

9.2. Start, Stop, Restart

Command Description

docker compose ps

Show the status of every service in the stack

docker compose stop ftacs

Stop the ftacs container, keeping it defined

docker compose start ftacs

Start the stopped ftacs container again

docker compose restart ftacs

Restart the ftacs container

docker compose down

Stop and remove all containers of the stack

cd /usr/local/ft-system

docker compose ps                # status of every service
docker compose stop ftacs        # stop one service
docker compose start ftacs       # start it again
docker compose restart ftacs     # restart it
docker compose down              # stop and remove all containers

docker compose down -v additionally removes all named volumes, including database data. Use it only when you intend to start from scratch.

9.3. Shell Access

Command Description

docker exec -it ftacs bash

Open an interactive shell inside the ftacs container

docker exec ftacs ls -la /opt/FTACS/logs

Run a single command inside the container without a shell session

docker exec -it ftacs bash
docker exec ftacs ls -la /opt/FTACS/logs

9.4. Updating FTACS

See Docker Image Management Guide for the full image lifecycle.

cd /usr/local/ft-system

# 1. Back up the FTACS configuration and logs
tar -czf ftacs-backup-$(date +%Y%m%d).tar.gz ./ftacs

# 2. Update the image tag in compose.yml, then:
docker compose pull ftacs
docker compose up -d ftacs

Liquibase migrates the database schema automatically on the first start of the new version.

Back up the ACS database before pulling a new image — the Liquibase migrations are applied on first start and are not reversible from within FTACS. The database backup is taken on the database host, which is outside this stack.

For manual deployments, the upgrade procedure is:

  1. Back up the database, configuration files, and logs.

  2. Check for a data-backfill migration — see Upgrading a Large Installation. On a large installation this step is mandatory and is performed before stopping the application.

  3. Stop the application — run ./stop.sh from bin/.

  4. Extract the new version into a new directory.

  5. Migrate the configuration — copy and adapt app.env from the old version; review new options.

  6. Start the new version — Liquibase migrates the schema automatically.

  7. Verify device connections, web services, and monitoring dashboards.

  8. Roll back if needed — stop the new version, restore the database backup, start the old version.

9.5. Upgrading a Large Installation

Most Liquibase changeSets are schema changes and complete instantly regardless of installation size. A data-backfill changeSet is different: it rewrites existing rows, and its cost scales with the table. Release notes flag such releases explicitly; before upgrading one, check the size of the affected table:

SELECT COUNT(*) FROM cpe;

Up to roughly 1 million rows there is nothing to prepare — the backfill completes in seconds during startup. Above that, run the release’s pre-upgrade backfill script while the ACS is still on the old version. The script applies exactly the same predicate as the migration, in primary-key batches, so afterwards the changeSet finds nothing to do and only pays for a table scan.

Table 1. Measured on MySQL 8.0, backfill of cpe.location_id (the column is in no index, so only the clustered index is touched)
Rows in cpe Migration without the pre-upgrade script Migration after it

1 million

5.8 s

0.3 s

5 million

30.4 s

1.5 s

25 million

~2.5 min (extrapolated)

~8 s

On a large installation, skipping the pre-upgrade script does not merely make the upgrade slow — it takes the service down.

  • The whole table is closed for writes. Under MySQL’s default REPEATABLE READ isolation the backfill takes next-key locks on every row it scans, not only on the rows it changes. Once this exceeds innodb_lock_wait_timeout (50 s by default), any node still serving device traffic fails its writes. Stop the entire cluster for the upgrade — do not upgrade node by node.

  • One transaction carries the whole change set. Expect gigabytes of undo/redo/binlog, replicas falling behind, and on Oracle an UNDO tablespace large enough to hold it (ORA-30036 otherwise).

  • An interrupted migration blocks every later start. If the process is killed mid-transaction — a Kubernetes startup probe, an OOM kill, a manual restart — the InnoDB rollback is slower than the forward pass, and the Liquibase lock stays held. Every subsequent start then fails with Could not acquire change log lock until the lock is cleared manually:

    DELETE FROM DATABASECHANGELOGLOCK;

    A second cluster node waits only changelogLockWaitTimeInMinutes (5 by default) before failing the same way.

FTACS starts Liquibase in strict mode (app.liquibase.strict, default true): a failed migration aborts startup rather than letting the application run against a half-migrated schema. Allow enough time in the startup probe — or the manual start — for the migration to finish.

9.5.1. Release 7.1.2 (DEV-2749) — location_id Backfill

Release 7.1.2 is such a release: it backfills cpe.location_id and cpe_credentials.isp_id.

The backfill is irreversible — the previous values are preserved nowhere, in neither the manual script nor the migration. Before running either one:

  1. Back up cpe and cpe_credentials.

  2. Run the verification queries (Verification queries) and keep the output. Section 4 lists the rows that are about to change — export it, it is the only record of the previous values.

  3. Note the category F count. Those rows are ambiguous and are left untouched on purpose; see Known limitation.

Pre-upgrade backfill script

The backfill is delivered as a single script, dev2749-backfill.sh, which does the same work as the migration but in batches of existing rows: each statement locks only its own range, and between batches the table is free. It is safe to re-run and safe to interrupt — a repeat run picks up whatever is left.

It is published on the FT shared drive, next to the release it belongs to, as ACS/dev2749-backfill.sh:

Copy it to a host that can reach the database and make it executable:

chmod +x dev2749-backfill.sh

Requirements: the mysql client for MySQL, or sqlplus for Oracle, on that host. The password is read from the DB_PASSWORD environment variable, or prompted for — it is never passed on the command line, where it would be visible in ps and in the shell history.

Step 1 — report only, changes nothing:

# MySQL
DB_PASSWORD=secret ./dev2749-backfill.sh --check --engine mysql \
    --host db.example.com --port 3306 --user ftacs --database ftacs

# Oracle
DB_PASSWORD=secret ./dev2749-backfill.sh --check --engine oracle \
    --host db.example.com --port 1521 --user ftacs --service FREEPDB1

It prints the row count of cpe and the category breakdown for both tables — the same categories as Verification queries. Category A plus category B is the number of rows the upgrade would rewrite.

Step 2 — run the backfill (same options, --run instead of --check), with the ACS still on the old version:

DB_PASSWORD=secret ./dev2749-backfill.sh --run --engine mysql \
    --host db.example.com --port 3306 --user ftacs --database ftacs

--run asks for an explicit yes before touching anything, then:

  1. exports every row it is about to change to <outdir>/cpe-before-backfill.tsv — keep this file, it is the only record of the previous values;

  2. backfills both tables in batches, printing progress;

  3. prints the category breakdown again, where A and B must both be 0.

Everything it prints also goes to <outdir>/run.log. By default <outdir> is ./dev2749-<timestamp>; use --outdir to place it elsewhere, and --batch to change the batch size (default 50000, which keeps each statement at roughly a quarter of a second).

The script is safe on a small installation too — it simply reports that the migration will complete on its own and, in --run, that there is nothing to heal.

Verification queries

Run before the upgrade to size the work and record the previous values, and after it to confirm the outcome. Portable as written — works on both MySQL 8 and Oracle.

-- 1. cpe.location_id - one row per category
SELECT 'A null (heals to 0)'            AS category, COUNT(*) AS rows_affected FROM cpe c
 WHERE c.location_id IS NULL
UNION ALL
SELECT 'B poisoned (heals to 0)',       COUNT(*) FROM cpe c
 WHERE c.location_id IS NOT NULL AND c.location_id != 0
   AND NOT EXISTS (SELECT 1 FROM isp i WHERE i.id = c.location_id)
   AND     EXISTS (SELECT 1 FROM product_class_group g WHERE g.id = c.location_id)
UNION ALL
SELECT 'C real domain (untouched)',     COUNT(*) FROM cpe c
 WHERE c.location_id IS NOT NULL AND c.location_id != 0
   AND EXISTS (SELECT 1 FROM isp i WHERE i.id = c.location_id)
   AND NOT EXISTS (SELECT 1 FROM product_class_group g WHERE g.id = c.location_id)
UNION ALL
SELECT 'D orphaned domain (untouched)', COUNT(*) FROM cpe c
 WHERE c.location_id IS NOT NULL AND c.location_id != 0
   AND NOT EXISTS (SELECT 1 FROM isp i WHERE i.id = c.location_id)
   AND NOT EXISTS (SELECT 1 FROM product_class_group g WHERE g.id = c.location_id)
UNION ALL
SELECT 'E already super domain',        COUNT(*) FROM cpe c
 WHERE c.location_id = 0
UNION ALL
SELECT 'F AMBIGUOUS - needs review',    COUNT(*) FROM cpe c
 WHERE c.location_id IS NOT NULL AND c.location_id != 0
   AND EXISTS (SELECT 1 FROM isp i WHERE i.id = c.location_id)
   AND EXISTS (SELECT 1 FROM product_class_group g WHERE g.id = c.location_id);

-- 2. cpe_credentials.isp_id - same categories (isp_id is NOT NULL DEFAULT 0)
SELECT 'B poisoned (heals to 0)'        AS category, COUNT(*) AS rows_affected FROM cpe_credentials cc
 WHERE cc.isp_id != 0
   AND NOT EXISTS (SELECT 1 FROM isp i WHERE i.id = cc.isp_id)
   AND     EXISTS (SELECT 1 FROM product_class_group g WHERE g.id = cc.isp_id)
UNION ALL
SELECT 'C real domain (untouched)',     COUNT(*) FROM cpe_credentials cc
 WHERE cc.isp_id != 0
   AND EXISTS (SELECT 1 FROM isp i WHERE i.id = cc.isp_id)
   AND NOT EXISTS (SELECT 1 FROM product_class_group g WHERE g.id = cc.isp_id)
UNION ALL
SELECT 'D orphaned domain (untouched)', COUNT(*) FROM cpe_credentials cc
 WHERE cc.isp_id != 0
   AND NOT EXISTS (SELECT 1 FROM isp i WHERE i.id = cc.isp_id)
   AND NOT EXISTS (SELECT 1 FROM product_class_group g WHERE g.id = cc.isp_id)
UNION ALL
SELECT 'E already super domain',        COUNT(*) FROM cpe_credentials cc
 WHERE cc.isp_id = 0
UNION ALL
SELECT 'F AMBIGUOUS - needs review',    COUNT(*) FROM cpe_credentials cc
 WHERE cc.isp_id != 0
   AND EXISTS (SELECT 1 FROM isp i WHERE i.id = cc.isp_id)
   AND EXISTS (SELECT 1 FROM product_class_group g WHERE g.id = cc.isp_id);

-- 3. Context: how large are the two id spaces, and do they overlap?
SELECT 'isp rows'                 AS metric, COUNT(*) AS value FROM isp
UNION ALL
SELECT 'product_class_group rows',           COUNT(*) FROM product_class_group
UNION ALL
SELECT 'overlapping ids',                    COUNT(*) FROM isp i
 WHERE EXISTS (SELECT 1 FROM product_class_group g WHERE g.id = i.id);

-- 4. The rows that are about to change - EXPORT THIS BEFORE UPGRADING.
--    Their previous values are preserved nowhere else.
SELECT c.id, c.serial, c.location_id, c.product_class_id, c.protocol_id, c.updated
  FROM cpe c
 WHERE c.location_id IS NULL
    OR (c.location_id != 0
        AND NOT EXISTS (SELECT 1 FROM isp i WHERE i.id = c.location_id)
        AND EXISTS (SELECT 1 FROM product_class_group g WHERE g.id = c.location_id))
 ORDER BY c.id;

Expected after the upgrade: A = 0, B = 0, C and D unchanged, E = Ebefore + Abefore + Bbefore, F unchanged.

After the upgrade
  1. Re-run the verification queries. Categories A and B must both be 0.

  2. Verify device connections and domain assignment in the UI.

  3. If the ACS failed to start with Could not acquire change log lock, the previous attempt was interrupted mid-migration. Clear the lock and start again:

    DELETE FROM DATABASECHANGELOGLOCK;
Known limitation

A poisoned value that happens to equal a real domain id cannot be told apart from a legitimate assignment: both id sequences are independent and both start low, so such collisions are common rather than exceptional. Those rows are reported as category F by the verification queries, and neither the migration nor the runtime fix touches them — they need manual review.

10. Production Checklist

  • Change all default passwords in .env / app.env (DB_PASSWORD, CLICKHOUSE_PASWORD — single "S", see Environment Configuration — and SSL_KEYSTORE_PASSWORD).

  • Use a valid SSL certificate — replace the self-signed ftacs.keystore with a CA-issued certificate.

  • Configure firewall rules — allow only the required ports; restrict database access to ACS hosts only.

  • Review TR-069 authentication — disable the no-auth endpoint (/ftacs/ACS) if it is not required.

  • Secure configuration files — restrict permissions: chmod 600 bin/app.env.

  • Enable auto-restart — restart: unless-stopped in compose.yml, or --restart unless-stopped with docker run.

  • Verify health checks — the compose file defines them for every service; add --health-cmd for standalone docker run.

  • Set up monitoring — deploy Prometheus and Grafana against /actuator/prometheus and the ClickHouse data source.

  • Configure backups — regular database backups, a configuration directory backup, a ClickHouse backup strategy, and log rotation.

  • Deploy multiple FTACS instances for high availability — identical configuration, an external Hazelcast cluster with 2+ nodes, and a load balancer in front.

  • Tune JVM settings — match the JVM sizing table to your deployment size.

11. Troubleshooting

11.1. Container Fails to Start

Symptom: The ftacs container exits immediately after docker compose up.

Fix:

  1. Read the error message from the logs:

    docker compose logs ftacs
    docker events --filter container=ftacs
  2. Confirm the environment files are loaded:

    docker exec ftacs env | grep DB_
  3. Confirm ftacs/config/ exists and is populated — an empty mount at /opt/FTACS/conf is a common cause.

11.2. Database Connection Failure

Symptom: Startup fails with a JDBC connection error, or Liquibase cannot acquire a lock.

Fix:

  1. Test connectivity to the database. It is external to this stack and addressed by IP through DB_HOST (see Startup Dependencies), so there is no mysql container to reach or to read logs from — probe the address from the host and read the JDBC error in the FTACS log:

    # use the DB_HOST / DB_PORT values from .env
    nc -zv <db-host> <db-port>
    
    docker logs ftacs 2>&1 | grep -i -E 'jdbc|liquibase|hikari'
  2. Verify DB_HOST, DB_PORT, DB_USER, and DB_PASSWORD in .env match the database configuration.

  3. If the database was just created, wait for its health check to pass before starting FTACS.

11.3. ClickHouse Connections Fail

Symptom: QoE and statistics queries fail; the log reports a ClickHouse connection or protocol error.

Fix:

  1. Confirm FTACS targets the HTTP port 8123, not the native TCP port 9000 — FTACS uses the clickhouse-java HTTP client. In the compose file this is the CLICKHOUSE_PORT: "$\{CLICKHOUSE_HTTP_PORT:-8123}" remap.

  2. Verify ClickHouse answers, from inside the container so the check uses the same network path FTACS does:

    docker exec ftacs curl -s http://${CLICKHOUSE_HOST}:8123/ping
  3. If ClickHouse answers but federated queries against the relational database fail, the JDBC Bridge on the ClickHouse host is the likely cause — report it to whoever operates that stack.

11.4. FTACS Starts but Has No Configuration

Symptom: FTACS reaches Started FtAcsApp, but web service calls are rejected as unauthorised, ACS configuration values fall back to defaults, and the parameter-name cache rules have no effect. No error appears in server.log.

Cause: since 6.5.2 this configuration is owned by FT Configs Service and delivered through Hazelcast. FTACS finds no published snapshot — either the service is not running, or the two are on different Hazelcast clusters.

Fix:

  1. Verify FT Configs Service is healthy:

    curl -s http://<configs-service-host-ip>:8087/configs-service/actuator/health
  2. Compare HZ_MEMBERS in FTACS’s .env with the value used by FT Configs Service — both must resolve to the same cluster.

  3. Compare the cluster name in ftacs/config/hazelcast-client.xml with the one FT Configs Service uses (default: dev). A mismatch produces two separate clusters that never see each other.

  4. Confirm the configuration was actually saved in FT Configs UI — an empty service publishes empty snapshots, which looks identical from the FTACS side.

  5. Do not work around this by putting acs_configuration.xml and friends back into ftacs/config/ — FTACS no longer reads them from disk.

11.5. Hazelcast Cluster Issues

Symptom: The log shows Unable to connect to any address in the config or repeated Hazelcast client reconnects.

Fix:

  1. Verify HZ_MEMBERS in .env lists addresses this host can actually reach. The Hazelcast cluster is a prerequisite deployed outside this guide (see Prerequisites), so use the member host IPs — localhost or a bridge alias only works when a member runs on this host and on the same Docker network.

  2. Ensure ports 5701 and 5711 are open from this host to every Hazelcast member.

  3. Check that the cluster name in ftacs/config/hazelcast-client.xml — the only Hazelcast file this installation owns — matches the name the cluster was started with (default: dev). The member-side hazelcast.xml is not part of this installation; ask whoever operates the cluster.

11.6. Port Already in Use

Symptom: docker compose up fails with bind: address already in use.

Fix:

  1. Find the process holding the port:

    sudo lsof -i :8080
    netstat -tulpn | grep 8080
  2. Either stop that process, or change the published port through the corresponding .env variable (for example FTACS_HTTP_PORT=9080) and run docker compose up -d again.

11.7. Permission Denied on Mounted Volumes

Symptom: The container cannot write to ftacs/logs or the database cannot write to its data directory.

Fix:

sudo chown -R $(whoami):$(whoami) /usr/local/ft-system
chmod -R 755 /usr/local/ft-system

11.8. Out of Memory

Symptom: java.lang.OutOfMemoryError in server.log, or the container is killed by the OOM killer.

Fix:

  1. Raise the container memory limit (Docker Desktop: Settings → Resources → Memory).

  2. Increase the heap through FT_ACS_JAVA_RAM in .env (Docker) or JAVA_OPTS in app.env (manual), matching the JVM sizing table.

  3. If the problem persists, capture a heap dump and analyse it for leaks.

11.9. Containers Not Reachable From the Host

Symptom: The container is healthy and answers on localhost inside the container, published ports look correct (0.0.0.0:PORT→PORT), but requests from the host hang and time out — no ECONNREFUSED, only timeouts.

Cause: the host firewall blocks traffic on Docker bridge interfaces (br-*) in the host INPUT/OUTPUT path. This is not a Compose or application issue.

Fix:

  1. Allow host-to-container traffic on Docker bridges:

    # IPv4 (all Docker bridges)
    iptables -I INPUT  1 -i br+ -j ACCEPT
    iptables -I OUTPUT 1 -o br+ -j ACCEPT
    
    # IPv6 requires separate rules
    ip6tables -I INPUT  1 -i br+ -j ACCEPT
    ip6tables -I OUTPUT 1 -o br+ -j ACCEPT
    
    # Persist (Debian/Ubuntu)
    apt-get install -y iptables-persistent
    netfilter-persistent save
  2. Confirm the diagnosis step by step:

    # 1. Container works internally
    docker exec ftacs curl -s -o /dev/null -w '%{http_code}\n' http://localhost:8080
    
    # 2. Container IPv4 address
    docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' ftacs
    
    # 3. Host -> container IP
    nc -vz <container_ipv4> 8080
    
    # 4. Identify the Docker bridge
    ip -4 addr | grep '^.*br-'
    
    # 5. Inspect the firewall
    iptables -S INPUT
    iptables -S OUTPUT

IPv4 working does not imply IPv6 works — they are separate firewall stacks. Docker programs NAT and FORWARD rules but does not override host INPUT/OUTPUT policies.

11.10. Quartz Cluster Recovery Messages

Symptom: ClusterManager: detected failed or restarted instances appears during startup.

Fix: No action needed. This is normal when the application restarts or a previous instance was not shut down cleanly; Quartz recovers jobs from failed instances automatically.

11.11. Application Fails to Start (Manual Deployment)

Symptom: start.sh returns but no process is listening; server.log is empty or shows an early failure.

Fix:

  1. Verify Java: java -version must report JDK 25.

  2. Check JAVA_HOME and APP_HOME in app.env — on Windows APP_HOME must use forward slashes and a trailing slash (D:/ftacs/dist/).

  3. Verify the database is reachable: telnet <DB_HOST> <DB_PORT>.

  4. Check the credentials in app.env.

  5. Review logs/liquibase.log for SQL errors.

11.12. Slow Performance

Symptom: High response times, growing session backlog, or CPE session timeouts.

Fix:

  1. Check relational database query performance and index health.

  2. Verify Hazelcast cache hit ratios.

  3. Review the HikariCP pool sizes in ftacs/.env (FTACS_MAX_POOL, QUARTZ_MAX_POOL, CLICKHOUSE_MAX_POOL).

  4. Measure network latency to the database and to Hazelcast.

  5. Increase the JVM heap per the JVM sizing table.

  6. Inspect the Grafana dashboards fed by /actuator/prometheus.

11.13. Getting Support

If the issue persists, collect the following and contact Friendly Tech support:

  1. .env and ftacs/.env (or app.env) with passwords masked.

  2. logs/server.log — the last 1000 lines.

  3. logs/liquibase.log if the problem is database-related.

  4. java -version output.

  5. Operating system version.

  6. Database type and version.

  7. docker version output, if applicable.