Installation & Deployment

This guide covers deploying FTACS — the TR-069/TR-369 Auto Configuration Server — as a single service.

Two deployment methods are supported: Docker Compose (recommended) and manual installation from the distribution package on bare metal or a VM.

The infrastructure FTACS depends on — a relational database, ClickHouse, Hazelcast, and FT Configs Service — is assumed to be already deployed and reachable; this guide only points FTACS at it. To deploy that infrastructure, see All in one server deployment or Separate server deployment.
For the environment variable reference, see Configuration Guide — Environment Variables.

1. Overview

FTACS is a single container (ftacs) — or a single JVM process in a manual deployment — exposing the TR-069, USP, MQTT, LWM2M, SOAP, and REST endpoints. It is a plain Compose service on the ft-network bridge, deployed from /usr/local/ft-system/.

Deployment method What it installs Section

Docker Compose

The ftacs container, its configuration directory, and its logs

Docker Compose

Manual

The distribution package on a JDK 25 host, driven by bin/app.env

Manual (dist package)

Everything else in the diagram below is a prerequisite, not part of this installation:

CPE devices  ->  FTACS  ->  MySQL / Oracle  +  ClickHouse  +  Hazelcast  <-  FT Configs Service

FTACS uses the Compose project name ft-system and the directory root /usr/local/ft-system/ — the same layout as every deployment guide in the Friendly Tech portal. Keeping the layout identical means the ftacs service can be added to an existing ft-system stack without moving any directory. Use the project name to manage it from anywhere on the host, e.g. docker compose -p ft-system ps.

1.1. Choosing a Deployment Method

  • Docker Compose — recommended. Use this unless a policy forbids containers.

  • Manual — the distribution package runs directly on a JDK 25 host. Use it when Docker is unavailable, or when FTACS must run alongside existing bare-metal services.

Since version 6.5.2, FT Configs Service is a hard dependency — it owns the ACS runtime configuration that used to live in ftacs/config/ and publishes it through Hazelcast. FTACS is not usable without it. See Prerequisites.

Deploying it is out of scope here: use FT Configs deployment, or Separate server deployment — Server C, which brings up FTACS and FT Configs Service together on one host.

2. Prerequisites

2.1. Host Requirements

Component Minimum Recommended Notes

CPU

4 cores

 — 

Baseline for a single FTACS instance

RAM

8 GB

 — 

Must cover the JVM heap set through FT_ACS_JAVA_RAM — see the sizing table below

Disk

20 GB

 — 

Application, configuration, and logs. The database and ClickHouse keep their data on their own hosts

Network

1 Gbps interface

 — 

Device traffic plus the outbound connections listed in Network Requirements

Docker Engine

20.10

 — 

Docker deployment only; Docker Compose 2.0+ as well

JDK

25

 — 

Manual deployment only — the application is built and run on Java 25

JVM sizing by deployment scale:

Deployment size Devices JVM settings

Small

< 1,000

-Xms512m -Xmx2g

Medium

1,000—​5,000

-Xms1g -Xmx4g

Large

5,000—​20,000

-Xms2g -Xmx8g

Very large

20,000+

-Xms4g -Xmx16g

2.2. Required External Dependencies

These services must be installed, running, and reachable from this host before FTACS starts. None of them are deployed by this guide.

Component Minimum Version Why It Is Needed Port Required

MySQL or Oracle

MySQL 8.4 / Oracle 21c (the reference compose in this repo builds MySQL from mysql:8.4.6 and runs gvenzl/oracle-xe:21)

Holds the ftacs and ftacs_quartz schemas; Liquibase applies the FTACS migrations into them on first start. Without it FTACS does not start.

3306 (TCP) for MySQL, 1521 (TCP) for Oracle

Yes

ClickHouse

26.2

Stores the QoE and statistics data (schema ftacs_qoe_ui_data). FTACS connects with the clickhouse-jdbc client from clickhouse-java over HTTP, so the native TCP port is not sufficient. Without it QoE and statistics queries fail.

8123 (HTTP)

Yes

Hazelcast

5.5 (FTACS embeds the com.hazelcast:hazelcast 5.5.0 client; the reference compose image is built from hazelcast/hazelcast:5.5.0-slim)

Distributed cache and the only channel through which FT Configs Service delivers the ACS runtime configuration. Without it FTACS cannot initialise its Hazelcast client and receives no configuration.

5701, 5711 (TCP)

Yes

FT Configs Service

verify with the FT Configs Service owner

Owns the ACS runtime configuration since FTACS 6.5.2 and publishes it into the shared Hazelcast cluster. Without it FTACS starts with no ACS configuration, no web service users, and no parameter-name cache rules.

No direct port — delivered through Hazelcast; its own health endpoint is 8087 (HTTP)

Yes

Deploying that infrastructure is out of scope here — see All in one server deployment for a single-host stack, or Separate server deployment for a split topology, and FT Configs deployment for FT Configs Service.

The database schemas (ftacs, ftacs_quartz) and the ClickHouse schema (ftacs_qoe_ui_data) must exist and be accessible to the FTACS user before the first start.

26.2 is the supported ClickHouse version. The reference compose in this repo (FT_IoT/FTAPP/ftapp-build/docker/docker-compose.yml) still pins clickhouse/clickhouse-server:24.8; that pin is a development-stack artefact and is tracked in docs/decisions/LOG.md. Deploy 26.2 unless FT DevOps tells you otherwise.

FTACS and FT Configs Service must be clients of the same Hazelcast cluster — that cluster is the only channel between them. If HZ_MEMBERS on the two services points at different clusters, FTACS starts cleanly and then behaves as if no configuration had ever been set: this failure is silent, with no connection error in the log.

2.3. Supported Operating Systems

Deployment Operating system

Docker

Linux with Docker Engine 20.10+ (recommended), macOS, or Windows with WSL2

Manual

Linux (recommended), Windows Server — JDK 25 required

2.4. Registry Access

Network access to the hub.friendly-tech.com Docker registry (or offline image archives — see Offline Servers).

Access to Friendly Tech SharePoint (FT_DISK) for the deployment files (compose.yml, .env, ftacs/.env) and the pre-configured ftacs-conf package.

3. Network Requirements

Outbound connections FTACS opens to its infrastructure, and inbound connections it accepts. Each row is a connection that must be open through firewalls when the peer is on another host; traffic to peers on the same Docker bridge needs no rule.

Destination Port Protocol Purpose

Database

3306 (Oracle: 1521)

TCP

Outbound: relational database connection

ClickHouse

8123

HTTP

Outbound: ClickHouse HTTP API — FTACS uses the clickhouse-jdbc HTTP client

Hazelcast

5701, 5711

TCP

Outbound: Hazelcast client and member discovery

FTACS

8080, 8181, 8443

TCP

Inbound from CPE devices: TR-069 (CWMP) sessions, SOAP, REST

FTACS

1883, 8883

TCP

Inbound from CPE devices: MQTT and MQTT over TLS

FTACS

8025, 4430, 18881, 18882, 18891, 61617, 61618

TCP

Inbound from CPE devices: USP (TR-369) — WebSocket, MQTT, Bulk Data, STOMP

FTACS

5783, 5784

UDP

Inbound from CPE devices: USP CoAP and CoAP over DTLS

FTACS

5680—​5685

UDP

Inbound from CPE devices: LWM2M Bootstrap and LWM2M Server

FTACS

8080

TCP

Inbound from monitoring: Prometheus scrapes /actuator/prometheus

For a quick connectivity check from any host:

# Replace with target host IP and port
nc -zv <target-ip> <port>

3.1. Docker Networking

The compose file in Docker Compose puts FTACS on the ft-network bridge with container_name: ftacs, so peers on that same bridge reach it as ftacs and need no firewall rule. The infrastructure services are addressed the same way when they share the bridge — that is what the comment on DB_HOST in the environment file means by "use the container name if it shares this bridge".

Peers on another host must be given a routable address instead: DB_HOST, HZ_MEMBERS and CLICKHOUSE_HOST take an IP or DNS name, and every port in the table above has to be open through the firewall. Mixing the two is the usual cause of a container that starts and then cannot reach its database: a container name only resolves inside the bridge that defines it.

If a peer runs on the same host but outside Docker, use host.docker.internal rather than localhost — inside a container localhost is the container itself:

DB_HOST=host.docker.internal
HZ_MEMBERS=host.docker.internal:5701
CLICKHOUSE_HOST=host.docker.internal

host.docker.internal resolves to the host’s gateway address. On Docker Desktop (macOS, Windows) it works out of the box; on Linux it exists only if the container maps it, which the compose file in Docker Compose does:

extra_hosts:
  - "host.docker.internal:host-gateway"

The alternative is the host’s real LAN IP address.

4. Registry Authentication

All FTACS images are pulled from hub.friendly-tech.com. Authenticate once per host before the first docker compose up.

docker login hub.friendly-tech.com

Enter the read-only pull credentials when prompted:

Field Value

Username

readonly

Password

fokxuw-fymte1-taSxyc

The readonly account provides pull-only access to the published images. It cannot push.

Alternatively, log in non-interactively:

echo "fokxuw-fymte1-taSxyc" | docker login hub.friendly-tech.com -u readonly --password-stdin

Verify authentication:

docker info | grep -A 5 Registry

4.1. Offline Servers

When the FTACS host cannot reach hub.friendly-tech.com, pull the image on a machine that does have registry access, export it to an archive, transfer it, and load it on the offline host.

The connected machine can run Linux, macOS, or Windows — commands are given for both shells below. It does not need to be the same platform as the offline host.

An explicit --platform matching the offline host’s architecture is required. The images in Harbor are multi-arch (linux/amd64, linux/arm64); without --platform, docker pull selects the host architecture, which may not match the target. On an Apple Silicon (arm64) Mac or an arm64 Windows machine without --platform, the resulting archive will be arm64 and will fail with a platform does not match warning on amd64 servers. The examples below use linux/amd64; step 1 shows how to read the correct value off the offline host.

  1. On the offline host, find out which architecture it runs — this is the value you will pass as PLATFORM below. Ask Docker itself, since it reports what the daemon will actually accept:

    docker version --format '{{.Server.Arch}}'

    If Docker is not installed there yet, use the operating system instead — uname -m on Linux, or echo $env:PROCESSOR_ARCHITECTURE in PowerShell on Windows. Map the result:

    docker version reports uname -m / Windows reports Use as PLATFORM

    amd64

    x86_64 / AMD64

    linux/amd64

    arm64

    aarch64 / ARM64

    linux/arm64

    On Windows with Docker Desktop, {{.Server.Arch}} reports the architecture of the Linux VM that actually runs the containers — which is the value you want, not the Windows host’s own architecture.

  2. On a machine with registry access, log in. Run this on its own — it prompts for a password, so anything pasted after it on the same go would be swallowed as input:

    docker login hub.friendly-tech.com

  3. Pull and export the image. Paste the whole block as-is; the only lines to change are PLATFORM and TAG on top.

    Linux / macOS (bash):

    PLATFORM=linux/amd64
TAG=latest

docker pull --platform "$PLATFORM" "hub.friendly-tech.com/ftacs/ftacs:$TAG"
docker save "hub.friendly-tech.com/ftacs/ftacs:$TAG" | gzip > "ftacs-$TAG.tar.gz"

    Windows (PowerShell):

    $PLATFORM = "linux/amd64"
$TAG = "latest"

docker pull --platform $PLATFORM "hub.friendly-tech.com/ftacs/ftacs:$TAG"
docker save -o "ftacs-$TAG.tar" "hub.friendly-tech.com/ftacs/ftacs:$TAG"

    On Windows, always write the archive with docker save -o <file>. Piping or redirecting docker save from PowerShell (docker save …​ > file.tar) corrupts the archive, because the PowerShell pipeline re-encodes the stream as text instead of passing raw bytes. docker load then fails with unexpected EOF or invalid tar header. To compress for transfer, use the bundled tar.exe (Windows 10 1803+ / Server 2019+): tar.exe -czf ftacs.tar.gz ftacs-$TAG.tar.

  4. Transfer the archive to the offline host, together with compose.yml, .env, ftacs/.env, and the three files in ftacs/config/.

  5. On the offline host, load the archive and start the service. Set TAG to the same value you used above:

    Linux (bash):

    TAG=latest

gzip -dc "ftacs-$TAG.tar.gz" | docker load
docker compose up -d ftacs

    Windows (PowerShell):

    $TAG = "latest"

docker load -i "ftacs-$TAG.tar"
docker compose up -d ftacs

Confirm the image is present before starting, so a missing or mis-architected image fails here rather than mid-startup:

docker images hub.friendly-tech.com/ftacs/ftacs

Upgrades use the same flow: pull the new tag on the connected machine, transfer and load the archive, then docker compose up -d ftacs.

This covers the FTACS image only. The database, ClickHouse, Hazelcast, and FT Configs Service are separate images on their own hosts — for the offline procedure covering a whole stack, see All in one server deployment — Offline Servers, which derives the image list from compose.yml itself.

5. Preparation

Create the directories FTACS needs on the host and point it at the infrastructure listed in Prerequisites.

5.1. First: Check the Existing Database (7.1.2 and later)

Applies whenever 7.1.2 or later is brought up against a database that already contains devices — an upgrade, a re-install on an existing schema, or a migration of the database to a new host. A fresh, empty database needs none of this.

Release 7.1.2 carries a data-backfill migration: unlike a schema change, it rewrites existing rows, so its cost scales with the size of the cpe table. Before anything else, find out how big that is:

SELECT COUNT(*) FROM cpe;
Devices in cpe What this means for the installation

Up to ~1 million

Nothing to prepare. The migration completes in seconds during startup. Continue with Directory Structure.

More than ~1 million

A manual step is required before starting the new version, and the cluster must be stopped for the upgrade rather than updated node by node. Read Upgrading a Large Installation now — it explains the backfill script, the verification queries and the failure modes of skipping it.

On a large installation, starting 7.1.2 without that preparation is not merely slow: the whole cpe table is closed for writes while the migration runs, and a migration killed mid-transaction leaves the Liquibase lock held, so every later start fails until it is cleared by hand. See Upgrading a Large Installation.

5.2. Directory Structure

mkdir -p /usr/local/ft-system/ftacs/{config,logs}
cd /usr/local/ft-system

5.2.1. Directory Layout

FTACS is one slice of the platform layout described in Prepare Working Directory — the stack root holds compose.yml and the shared .env, and each service owns a directory below it:

/usr/local/ft-system/
├── compose.yml                             # from FT_DISK
├── .env                                    # shared stack environment
└── ftacs/
    ├── .env                                # per-service environment
    ├── sensitive_params.xml                # optional, mounted separately
    ├── config/                             # -> /opt/FTACS/conf (read-write)
    │   ├── ftacs.keystore
    │   ├── hazelcast-client.xml
    │   └── logback-spring.xml
    └── logs/                               # -> /opt/FTACS/logs
Path Content Backup

compose.yml

Stack definition; the FTACS service block is shown in Deployment

Yes

.env

Shared stack environment — database connection, ClickHouse, Hazelcast, timezone

Yes

ftacs/.env

FTACS-only environment — published host ports, HikariCP pool sizes, JVM settings

Yes

ftacs/config/

The three files FTACS reads from disk, mounted read-write at /opt/FTACS/conf

Yes

ftacs/sensitive_params.xml

Optional; mounted individually into /opt/FTACS/app/WEB-INF/lib/, not part of config/ — see Sensitive Parameters

Yes

ftacs/logs/

Application file logs, mounted at /opt/FTACS/logs; written by the container at runtime

No

Download the following from FT_DISK:

Component FT_DISK Target directory Files

FTACS environment

FT_DISK — ftacs

ftacs/.env

Only one file: the .env in the folder root

FTACS configuration

FT_DISK — ftacs-conf

ftacs/config/

Only three files: ftacs.keystore, hazelcast-client.xml, logback-spring.xml

The ftacs/config/ directory is mounted read-write at /opt/FTACS/conf inside the container and holds the three files FTACS must read from disk before it can reach anything else:

File Purpose

ftacs.keystore

SSL keystore for the HTTPS ports

hazelcast-client.xml

Hazelcast cluster client configuration — also the transport FTACS receives its runtime configuration over

logback-spring.xml

Logging configuration

Mount this directory read-write, not :ro. The three files above are only read at startup, but at runtime the QoE module also writes mac_path.csv and gpv_models.csv into this same directory to sync MAC-to-path and GPV-model data. Both writers only log an IOException on failure rather than crashing the process, so a read-only mount does not fail loudly — it just leaves those two files stale, silently. This is unrelated to the startup version banner, which is read from outside this directory.

Since version 6.5.2 the remaining runtime configuration lives in FT Configs Service, not on disk.

acs_configuration.xml, bulk_data.xml, ctn_info.properties, external_trace.properties, fcc.properties, force_qoe_stop.properties, ftacs_task.properties, ftacsws-roles.properties, ftacsws-users.properties, hardcoded_event_soap_template.xml, and parameterNamesCacheRules.yaml were moved there. Do not place them in ftacs/config/ — edit them through FT Configs UI instead.

FT Configs Service publishes each configuration snapshot into the shared Hazelcast cluster, and FTACS consumes it from there; there is no direct URL from FTACS to the service. This is why both must be clients of the same Hazelcast cluster — see Prerequisites.

See Configuration Guide for the reference on each configuration item, and FT Configs deployment for deploying the service that now owns them.

5.3. Environment Configuration

The stack uses two environment layers, and every service — FTACS included — reads both:

File Holds

.env

Shared by the whole stack: database connection (DB_HOST, DB_PORT, DB_USER, DB_PASSWORD), database vendor selection (DB_PROFILE, SPRING_PROFILES_ACTIVE, DB_SERVICE), ClickHouse, Hazelcast, timezone

ftacs/.env

FTACS only: published host ports, HikariCP pool sizes (FTACS_MAX_POOL, QUARTZ_MAX_POOL, CLICKHOUSE_MAX_POOL), JVM settings, SSL_KEYSTORE_PASSWORD

There is no per-vendor environment file. MySQL is the default; switching to Oracle means changing DB_PROFILE, SPRING_PROFILES_ACTIVE, DB_HOST, DB_PORT and DB_SERVICE inside the single root .env — see Switching to Oracle. compose.yml maps these generic DB_* names onto the vendor-specific names FTACS reads, through the x-db-env anchor shown in Deployment.

# /usr/local/ft-system/.env
TZ=Europe/Kyiv

# Database (existing)
DB_HOST=<db-host-ip>                       # <-- replace; use the container name if it shares this bridge
DB_PORT=3306
DB_USER=ftacs
DB_PASSWORD=<your-db-password>             # <-- replace
DB_SERVICE=XEPDB1
DB_PROFILE=mysql
SPRING_PROFILES_ACTIVE=mysql

# Hazelcast (existing)
HZ_MEMBERS=<hazelcast-host-ip>:5701        # <-- replace

# ClickHouse (existing)
CLICKHOUSE_HOST=<clickhouse-host-ip>       # <-- replace
CLICKHOUSE_PORT=9000
CLICKHOUSE_HTTP_PORT=8123
CLICKHOUSE_SCHEMA=ftacs_qoe_ui_data
CLICKHOUSE_USERNAME=ftacs
CLICKHOUSE_PASSWORD=<your-ch-password>     # <-- replace

# FTACS
FT_ACS_JAVA_RAM=-Xms1g -Xmx4g

The port changes together with the host, not only the hostname. Across hosts you connect to the published port (the left side of ports:); inside the Docker bridge you connect to the container port (the right side). ClickHouse is 8123 → 8123 and 9000 → 9000, so both forms match here — but FTACS must always target the HTTP port 8123, never the native TCP port 9000, because it uses the clickhouse-java HTTP client.

FTACS reads CLICKHOUSE_SCHEMA, not CLICKHOUSE_DATABASE. CLICKHOUSE_DATABASE is what ft-device-network-service reads, so both names can legitimately appear in the same .env — but the schema FTACS connects to comes from CLICKHOUSE_SCHEMA alone. (The clickhouse/clickhouse-server image reads neither; its own bootstrap variables are CLICKHOUSE_DB, CLICKHOUSE_USER and CLICKHOUSE_PASSWORD.)

The variables FTACS itself reads are defined in this repository by FT_IoT/FTAPP/ftapp-build/docker/ftacs.env (Docker), FT_IoT/FTAPP/ftapp-build/bin/app.env (manual) and FT_IoT/FTAPP/ftapp-build/docker/docker-entrypoint.sh:

Variable Description Default Required

APP_HOME

Installation root; conf/, logs/, ext/ and patches/ derive from it

/opt/ftacs in the entrypoint; the reference compose sets /opt/FTACS

No

DB_VENDOR

Database vendor; selects the active Spring profile (mysql or oracle)

mysql

No

DB_HOST

Hostname or IP of the database server

none

Yes

DB_PASSWORD

Password of the FTACS database user

none

Yes

MYSQL_PORT

MySQL listener port

3306

Yes (MySQL)

MYSQL_USER

MySQL user owning the ftacs schema

ftacs

Yes (MySQL)

ORACLE_PORT

Oracle listener port

1521

Yes (Oracle)

ORACLE_SERVICE

Oracle service name

orclpdb1 (application.yaml line 123)

Yes (Oracle)

ORACLE_USER_FTACS

Oracle user owning the ftacs schema

ftacs

Yes (Oracle)

ORACLE_USER_QUARTZ

Oracle user owning the ftacs_quartz schema

ftacs_quartz

Yes (Oracle)

HZ_MEMBERS

Hazelcast member list as host:port; the only channel to FT Configs Service

none

Yes

CLICKHOUSE_HOST

Hostname or IP of the ClickHouse server

none

Yes

CLICKHOUSE_PORT

ClickHouse port FTACS connects to. FTACS uses the HTTP client, so this must be the HTTP port 8123, not the native TCP port 9000

8123 in bin/app.env; docker/ftacs.env still ships 9000 — tracked in docs/decisions/LOG.md

Yes

CLICKHOUSE_USERNAME

ClickHouse user

ftacs

Yes

CLICKHOUSE_PASSWORD

ClickHouse password. The misspelled CLICKHOUSE_PASWORD (single "S") is still read as a deprecated fallback when CLICKHOUSE_PASSWORD is absent — rename it in your .env / app.env, because a future release will stop reading it

ftacs

Yes

CLICKHOUSE_SCHEMA

ClickHouse database FTACS writes QoE data to. FTACS does not read CLICKHOUSE_DATABASE

ftacs_qoe_ui_data

Yes

HTTP_PORTS

HTTP listener ports; comma or space separated

8080,8181 in application.yaml and bin/app.env; the Docker entrypoint falls back to 8080 when the variable is unset

No

HTTPS_PORTS

HTTPS listener port passed to the application as app.web.https-ports

8443

No

SSL_KEYSTORE

Path to the Java keystore serving the TLS ports

empty

No

SSL_KEYSTORE_PASSWORD

Password of that keystore

empty

No

WEB_TR069_NO_AUTH

Servlet path of the unauthenticated TR-069 endpoint

/ftacs/ACS

No

WEB_TR069_BASIC

Servlet path of the HTTP Basic TR-069 endpoint

/ftacs-basic/ACS

No

WEB_TR069_DIGEST

Servlet path of the HTTP Digest TR-069 endpoint

/ftacs-digest/ACS

No

JAVA_OPTS

JVM options passed to the java command

-Xms1g -Xmx4g -Duser.timezone=Europe/Kiev

No
The names used in the shared .env above (DB_PORT, DB_USER, DB_SERVICE, DB_PROFILE, CLICKHOUSE_HTTP_PORT, FT_ACS_JAVA_RAM) come from the portal-wide deployment bundle and do not appear in this repository’s env files; verify with FT DevOps which set applies to your build.

Create ftacs/.env with the per-service variables documented in Configuration Guide — Environment Variables; the full platform-wide reference is FTACS Environment Variables.

Click to expand a commented ftacs/.env reference 
# =============================================================================
# HikariCP Connection Pools
# =============================================================================

# Shared pool defaults (apply to all datasources unless overridden below)
DB_MAX_POOL_SIZE=10
DB_MIN_IDLE=5
DB_CONNECTION_TIMEOUT_MS=30000

# Main FTACS datasource pool. Handles CPE management, parameters,
# provisioning operations. Needs a larger pool due to high concurrency.
FTACS_MAX_POOL=30
FTACS_MIN_POOL=5

# Quartz job scheduler datasource. Used for scheduled task persistence
# (UpdateGroup, QoE, Events). Lower load than the main datasource.
QUARTZ_MAX_POOL=10
QUARTZ_MIN_POOL=2

# ClickHouse analytics datasource. Used for QoE data and statistics.
CLICKHOUSE_MAX_POOL=10
CLICKHOUSE_MIN_POOL=2

# =============================================================================
# ACS Deployment Mode
# Controls which functionality is active. Used for distributed deployments
# where provisioning and management run on separate nodes.
# =============================================================================

# ALL    = full functionality (provisioning + management) -- single-node default
# PROV   = provisioning only (device bootstrap, session handling)
# MNGMNT = management only (device monitoring, configuration)
# In MNGMNT mode, FTACS connects to a separate Hazelcast "provision" cluster.
# FTACS_MODE=ALL

# =============================================================================
# Server IP Address
# Advertised to CPE devices for callbacks and connection requests.
# Auto-detected from the hostname if not set. Set explicitly in Docker/NAT
# environments where auto-detection returns an unreachable address.
# =============================================================================

# IPv4 address advertised to CPE devices (e.g. 192.168.1.100)
# FTACS_IP_ADDRESS=

# IPv6 address for dual-stack deployments. Brackets are added automatically.
# FTACS_IPV6_ADDRESS=

# =============================================================================
# Event Subscription Parameters
# Control when parameter change notifications are sent to subscriber URLs.
# =============================================================================

# 0 = notify on every parameter update (EACH_VALUE trigger)
# 1 = notify only when the value actually changes (VALUE_CHANGE trigger)
# FTACS_SUBSCRIPTION_PARAMETER_ONLY_VALUE_CHANGE=0

# 0 = include all parameter values (including null/empty)
# 1 = skip notifications for null or empty values
# FTACS_SUBSCRIPTION_PARAMETER_NOT_NULL_ONLY=0

# =============================================================================
# TR-069 Endpoint URL Mappings
# Each defines the servlet URL path for a different authentication mode.
# Multiple paths can be comma-separated.
# =============================================================================

# No authentication -- CPE connects without credentials
WEB_TR069_NO_AUTH=/ftacs/ACS

# HTTP Basic Authentication (RFC 7617) -- CPE sends base64(user:password)
WEB_TR069_BASIC=/ftacs-basic/ACS

# HTTP Digest Authentication (RFC 2617) -- challenge-response, no plaintext password
WEB_TR069_DIGEST=/ftacs-digest/ACS

# =============================================================================
# MQTT Protocol (optional)
# Activates the embedded MQTT broker for TR-369/USP MQTT-MTP transport.
# =============================================================================

MQTT_ENABLE=false

# Additional MQTT transport connector URI (ActiveMQ format).
# Format: mqtt+nio://<host>:<port> or mqtt+nio+ssl://<host>:<port>
# FT_MQTT_ADDITIONAL_CONNECTOR_URI=

# Active MQTT API versions. Incoming requests are validated against this list.
# MQTT_ACTIVE_API=api-v1,api-v2

# =============================================================================
# USP Protocol (optional)
# =============================================================================

# Path to the USP data model XML specification (BBF TR-181-2).
# Loaded from the filesystem first; falls back to a classpath resource.
# USP_SERVER_DM_PATH=tr-181-2-16-0-usp-full.xml

# =============================================================================
# X509 Certificates -- USP TLS/DTLS (optional)
# Used for secure USP MTP connections (WebSocket, STOMP, CoAP over DTLS).
# Format: JKS or PKCS12 keystore files.
# =============================================================================
# FT_USP_X509_KEYSTORE_PATH=        # Server keystore with private key and certificate
# FT_USP_X509_KEYSTORE_PASSWORD=    # Keystore file password
# FT_USP_X509_KEYSTORE_PK_PASSWORD= # Private key password (if it differs)
# FT_USP_X509_TRUSTSTORE_PATH=      # Truststore with trusted CA certificates
# FT_USP_X509_TRUSTSTORE_PASSWORD=  # Truststore file password

# =============================================================================
# X509 Certificates -- MQTT TLS (optional)
# Used for MQTTS on port 8883. Enables mutual TLS when a truststore is set.
# =============================================================================
# FT_MQTT_X509_KEYSTORE_PATH=
# FT_MQTT_X509_KEYSTORE_PASSWORD=
# FT_MQTT_X509_PK_PASSWORD=
# FT_MQTT_X509_TRUSTSTORE_PATH=
# FT_MQTT_X509_TRUSTSTORE_PASSWORD=

# =============================================================================
# General Async Executor (used by @Async methods without an explicit executor)
# =============================================================================

# Minimum threads always running. Idle core threads are reclaimed after
# ASYNC_KEEPALIVE seconds (allowCoreThreadTimeout=true).
ASYNC_CORE=10

# Maximum threads under load. When ASYNC_QUEUE=0, threads spawn immediately
# up to this limit before tasks go to an external 10,000-task overflow queue.
ASYNC_MAX=200

# Internal queue capacity before spawning new threads.
# 0 = no internal queue (SynchronousQueue); tasks spawn threads immediately.
# An external OverflowingExecutor with a 10,000-task queue wraps the pool as a
# safety net, blocking on overflow to provide backpressure.
ASYNC_QUEUE=0

# Seconds an idle thread stays alive before termination.
ASYNC_KEEPALIVE=60

# Thread pool size for @Scheduled methods (virtual threads enabled).
# Handles periodic tasks: statistics flush (every 60s), counter reset (daily).
# Quartz jobs use a separate JDBC-based scheduler, not this pool.
SCHED_POOL=300

# =============================================================================
# HikariCP Tuning (optional)
# Applies to ALL three datasources (ftacs, quartz, clickhouse).
# =============================================================================

# Max time (ms) a connection can sit idle before removal. Default: 60000
# HIKARI_IDLE_TIMEOUT_MS=60000

# Max time (ms) to wait for a connection from the pool. Default: 45000
# HIKARI_CONN_TIMEOUT_MS=45000

# Max age (ms) of a connection before forced rotation. Default: 180000
# Must be greater than HIKARI_IDLE_TIMEOUT_MS.
# HIKARI_MAX_LIFETIME_MS=180000

# Max time (ms) for the connection validation query (SELECT 1). Default: 3000
# HIKARI_VALIDATION_TIMEOUT_MS=3000

# Register HikariCP pool metrics as JMX MBeans (true/false).
# HIKARI_REGISTER_MBEANS=false

5.4. TLS Keystore (optional)

FTACS serves its HTTPS port (8443) and the other TLS ports from a single Java keystore, ftacs.keystore. The bundle on FT_DISK ships a self-signed keystore — suitable for testing only. This step applies to both deployment methods; only the target directory and the file holding the password differ.

To use a CA-issued or otherwise custom certificate:

  1. Generate the keystore (or import an existing certificate into one):

    keytool -genkey -alias ftacs -keyalg RSA -keysize 2048 -keystore ftacs.keystore

  2. Place ftacs.keystore in the FTACS configuration directory:

    • Docker: ftacs/config/ftacs.keystore (mounted read-write at /opt/FTACS/conf).

    • Manual: conf/ftacs.keystore in the distribution package.

  3. Set the keystore password. SSL_KEYSTORE_PASSWORD must match the password used when the keystore was created:

    • Docker: in ftacs/.env.

    • Manual: in bin/app.env, alongside SSL_KEYSTORE=${CONF_DIR}/ftacs.keystore.

Both paths resolve to ${CONF_DIR}/ftacs.keystore inside the application — /opt/FTACS/conf in the container, ${APP_HOME}/conf in a manual deployment. Replacing the keystore requires a restart; it is not hot-reloaded.

6. Deployment

Choose one method: Docker Compose (recommended) or the distribution package on a JDK 25 host.

6.1. Startup Dependencies

FTACS has no depends_on on its infrastructure — the database, ClickHouse, Hazelcast, and FT Configs Service live outside this stack, so Compose cannot gate on their health. All of them must be up before FTACS launches: Liquibase applies the schema migrations on first start, the Hazelcast client connects during context initialisation, and FT Configs Service must already have published its configuration snapshots into Hazelcast.

Wait for each of them explicitly before starting the service:

# Wait for the database
until nc -z <db-host-ip> 3306; do sleep 2; done

# Wait for ClickHouse (HTTP port)
until curl -sf http://<clickhouse-host-ip>:8123/ping; do sleep 2; done

# Wait for Hazelcast
until nc -z <hazelcast-host-ip> 5701; do sleep 2; done

# Wait for FT Configs Service to be healthy (it warms up its Hazelcast snapshots on start)
until curl -sf http://<configs-service-host-ip>:8087/configs-service/actuator/health; do sleep 2; done

# Then start FTACS
docker compose up -d ftacs

FTACS itself restarts cleanly at any time once the infrastructure is up — restart: unless-stopped reconnects it after a host reboot.

6.2. Docker Compose

Click to expand compose.yml (FTACS) 
x-db-env: &db-env
  DB_VENDOR: ${DB_PROFILE}
  DB_HOST: ${DB_HOST}
  MYSQL_PORT: ${DB_PORT}
  MYSQL_USER: ${DB_USER}
  MYSQL_SCHEMA_FTACS: ftacs
  MYSQL_SCHEMA_QUARTZ: ftacs_quartz
  FTACS_PASSWORD: ${DB_PASSWORD}
  QUARTZ_PASSWORD: ${DB_PASSWORD}

services:

  ftacs:
    image: hub.friendly-tech.com/ftacs/ftacs:v7.1.1-b0.0.6
    container_name: ftacs
    # The database, ClickHouse and Hazelcast are external to this stack, so there
    # is no depends_on. They must be healthy before ftacs starts -- DB_HOST,
    # CLICKHOUSE_HOST and HZ_MEMBERS in .env point at them.
    env_file:
      - .env
      - ./ftacs/.env
    environment:
      <<: *db-env
      USE_NOHUP: 0
      # ftacs uses the clickhouse-java HTTP client, so it must target the HTTP
      # port (8123), not the native TCP port (9000) that CLICKHOUSE_PORT defaults
      # to in the shared .env. Without this remap, ClickHouse connections fail.
      CLICKHOUSE_PORT: "${CLICKHOUSE_HTTP_PORT:-8123}"
      JAVA_OPTS: "${FT_ACS_JAVA_RAM:--Xms1g -Xmx4g} -Duser.timezone=${TZ:-Europe/Kyiv}"
    ports:
      # TR-069 / REST / SOAP
      - "${FTACS_HTTP_PORT:-8080}:8080"
      - "${FTACS_HTTPS_PORT:-8443}:8443"
      - "${ACS_HTTP_ADDITIONAL:-8181}:8181"
      # MQTT
      - "${ACS_MQTT:-1883}:1883"
      - "${ACS_MQTT_TLS:-8883}:8883"
      # USP (TR-369)
      - "${ACS_USP_WS:-8025}:8025"
      - "${ACS_USP_WSS:-4430}:4430"
      - "${ACS_USP_MQTT:-18881}:18881"
      - "${ACS_USP_MQTT_TLS:-18882}:18882"
      - "${ACS_USP_BULK:-18891}:18891"
      - "${ACS_USP_STOMP:-61617}:61617"
      - "${ACS_USP_STOMP_TLS:-61618}:61618"
      - "${ACS_USP_CoAP:-5783}:5783/udp"
      - "5784:5784/udp"
      # LWM2M Bootstrap
      - "${BS_PORT_NOSEC:-5680}:5680/udp"
      - "${BS_PORT_PSK:-5681}:5681/udp"
      - "${BS_PORT_X509:-5682}:5682/udp"
      - "${BS_TCP_PORT_NOSEC:-5880}:5880"
      # LWM2M Server
      - "${LWM2M_PORT_NOSEC:-5683}:5683/udp"
      - "${LWM2M_PORT_PSK:-5684}:5684/udp"
      - "${LWM2M_PORT_X509:-5685}:5685/udp"
      - "${LWM2M_TCP_PORT_NOSEC:-5883}:5883"
    volumes:
      - ./ftacs/config:/opt/FTACS/conf
      - ./ftacs/logs:/opt/FTACS/logs
    healthcheck:
      test: ["CMD", "pgrep", "-f", "com.friendly.acs.FtAcsApp"]
      interval: 20s
      timeout: 5s
      retries: 10
      start_period: 60s
    restart: unless-stopped
    stop_grace_period: 45s
    extra_hosts:
      - "host.docker.internal:host-gateway"   # peers running on this host outside Docker
    networks:
      - ft-network

For Oracle, keep the same two env_file entries — there is no per-vendor environment file — set the vendor variables in the root .env as described in Switching to Oracle, and change the x-db-env anchor to Oracle-specific mappings:

x-db-env: &db-env
  DB_VENDOR: ${DB_PROFILE}
  DB_HOST: ${DB_HOST}
  ORACLE_PORT: ${DB_PORT}
  ORACLE_SERVICE: ${DB_SERVICE}
  ORACLE_USER_FTACS: ${DB_USER}
  ORACLE_USER_QUARTZ: ftacs_quartz
  FTACS_PASSWORD: ${DB_PASSWORD}
  QUARTZ_PASSWORD: ${DB_PASSWORD}

The anchors above map the shared .env names onto the names FTACS actually reads. The database host is DB_HOST for both vendors (application.yaml line 94, vars.host) — there is no MYSQL_HOST or ORACLE_HOST. Schemas and users are per role: MYSQL_SCHEMA_FTACS / MYSQL_SCHEMA_QUARTZ and ORACLE_USER_FTACS / ORACLE_USER_QUARTZ (application.yaml lines 115-116 and 122-123) — there is no MYSQL_SCHEMA or ORACLE_USER. The database password reaches the datasources as FTACS_PASSWORD and QUARTZ_PASSWORD (application.yaml lines 95-96); bin/app.env lines 101-102 derive both from DB_PASSWORD, but that derivation lives in the env file, not in the application, so a compose file must set them explicitly.

6.2.1. Sensitive Parameters

To run FTACS with a sensitive_params.xml file, mount it from the host into the container as an additional volume:

    volumes:
      - ./ftacs/config:/opt/FTACS/conf
      - ./ftacs/logs:/opt/FTACS/logs
      - ./ftacs/sensitive_params.xml:/opt/FTACS/app/WEB-INF/lib/sensitive_params.xml:ro

6.2.2. Start the Stack

Start FTACS only after the database, ClickHouse, Hazelcast, and FT Configs Service accept connections — see Startup Dependencies.

 
cd /usr/local/ft-system
docker login hub.friendly-tech.com
docker compose up -d ftacs
docker compose ps
# FTACS takes 1-2 minutes to start
docker compose logs -f ftacs | grep -i "started"
curl -s -u <ws-user>:<ws-password> http://localhost:8080/acsstats

To run FTACS as a single container without Compose:

cd /usr/local/ft-system

docker run -d \
  --name ftacs \
  --restart unless-stopped \
  --env-file .env --env-file ./ftacs/.env \
  -p 8080:8080 \
  -p 8443:8443 \
  -v ./ftacs/config:/opt/FTACS/conf \
  -v ./ftacs/logs:/opt/FTACS/logs \
  hub.friendly-tech.com/ftacs/ftacs:v7.1.1-b0.0.6

docker run publishes only the two ports shown above. Add a -p flag for every protocol your devices actually use — see Port Reference.

6.3. Manual (dist package)

FTACS runs directly on a JDK 25 host from the distribution package. The database, ClickHouse, Hazelcast, and FT Configs Service must be reachable before starting the application — see Prerequisites.

6.3.1. Directory Structure

The extracted distribution package has this structure:

dist/
├── app/                 # Compiled application (exploded WAR) -- do NOT modify
│   ├── META-INF/        # JAR/WAR manifest and persistence descriptors
│   ├── org/             # Java bytecode packages (ACS core classes)
│   └── WEB-INF/         # Spring Boot web configuration and libraries
│
├── bin/                 # Executable scripts and environment definitions
│   ├── app.env          # Main environment file (DB, ports, JVM, etc.)
│   ├── env_to_cmd.cmd   # Helper: .env -> Windows CMD variables
│   ├── start.cmd        # Windows startup script
│   ├── start.sh         # Linux startup script
│   ├── stop.cmd         # Windows shutdown script
│   └── stop.sh          # Linux shutdown script
│
├── conf/                # Configuration files (editable by operators)
│   ├── ftacs.keystore   # SSL keystore for the HTTPS ports
│   ├── hazelcast-client.xml  # Hazelcast client -- also the FT Configs Service channel
│   └── logback-spring.xml    # Logging configuration
│                        # everything else moved to FT Configs Service in 6.5.2
│
├── ext/                 # External extensions / JARs loaded dynamically
├── flyway/              # ClickHouse migration scripts (managed by Flyway)
├── liquibase/           # Database migration scripts (managed by Liquibase)
├── logs/                # Runtime logs (created at startup)
├── patches/             # Hot-fix patches
└── version.xml          # Product version/build metadata, read by FTIOTApplicationListener

  • app/ is static — do not modify; changes belong in conf/.

  • bin/ scripts resolve every path relative to APP_HOME, defined in app.env.

  • conf/ is safe to edit and should be version-controlled in ops repositories.

  • logs/ may be mounted to external storage for persistent log retention.

  • liquibase/ drives schema migrations; changes here must match release cycles.

  • ext/ enables plugin-like JARs or vendor integrations without rebuilding.

  • version.xml sits at the top level of dist/, next to app/ — not in conf/. It is produced by the build and read from the process working directory only; see Startup Log for what a missing or unparsable file looks like in the startup banner.

6.3.2. Install the JDK

Linux
wget https://download.oracle.com/java/25/latest/jdk-25_linux-x64_bin.tar.gz
tar -xzf jdk-25_linux-x64_bin.tar.gz
sudo mv jdk-25 /opt/jdk-25

# Add to ~/.bashrc or /etc/profile
export JAVA_HOME=/opt/jdk-25
export PATH=$JAVA_HOME/bin:$PATH

java -version
Windows

  1. Download the JDK 25 installer from the Oracle website.

  2. Run the installer (e.g. jdk-25_windows-x64_bin.exe).

  3. Set the JAVA_HOME environment variable to C:\Program Files\Java\jdk-25.

  4. Add %JAVA_HOME%\bin to PATH.

  5. Verify: open Command Prompt and run java -version.

Then extract the distribution package:

# Linux
tar -xzf ftacs-dist-<version>.tar.gz
cd dist/

# Windows
unzip ftacs-dist-<version>.zip
cd dist\

6.3.3. Environment Configuration

Edit bin/app.env in the extracted distribution package.

# === Base directories (all other paths derive from APP_HOME) ===
APP_HOME=/usr/local/ftacs/                            # <-- replace with your dist path
CONF_DIR=${APP_HOME}/conf
LOG_DIR=${APP_HOME}/logs
EXT_DIR=${APP_HOME}/ext
PATCHES_DIR=${APP_HOME}/patches
DB_LIQUIBASE_DIR=${APP_HOME}/liquibase

# === Database connection ===
DB_HOST=localhost                                     # <-- replace
DB_PASSWORD=<your-db-password>                        # <-- replace

# --- MySQL (default) ---
DB_VENDOR=mysql
MYSQL_PORT=3306
MYSQL_USER=ftacs

# --- Oracle (uncomment to switch) ---
#DB_VENDOR=oracle
#ORACLE_PORT=1521
#ORACLE_SERVICE=orclpdb1                              # <-- replace with your PDB name
#ORACLE_USER_FTACS=ftacs
#ORACLE_USER_QUARTZ=ftacs_quartz

# === ClickHouse ===
CLICKHOUSE_HOST=localhost                             # <-- replace
CLICKHOUSE_PORT=8123
CLICKHOUSE_SCHEMA=ftacs_qoe_ui_data
CLICKHOUSE_USERNAME=ftacs
CLICKHOUSE_PASSWORD=<your-ch-password>                # <-- replace

# === Hazelcast ===
HZ_MEMBERS=127.0.0.1:5701                             # <-- replace with cluster addresses
HZ_CLIENT_CONFIG=${CONF_DIR}/hazelcast-client.xml

# === TR-069 device URL mappings ===
WEB_TR069_NO_AUTH=/ftacs/ACS
WEB_TR069_BASIC=/ftacs-basic/ACS
WEB_TR069_DIGEST=/ftacs-digest/ACS

# === Web / SSL ===
HTTP_PORTS=8080,8181
HTTPS_PORTS=8443
SSL_KEYSTORE=${CONF_DIR}/ftacs.keystore
SSL_KEYSTORE_PASSWORD=<your-keystore-password>        # <-- replace

# === JVM options ===
JAVA_OPTS='-Xms1g -Xmx4g'

# === Additional configuration ===
SPRING_CONFIG_EXTRA=file:${CONF_DIR}/
LOGBACK_CONFIG=${CONF_DIR}/logback-spring.xml
CXF_PATH=/ACSServer-ACS
QUARTZ_PASSWORD=${DB_PASSWORD}

Set APP_HOME to the absolute path of your dist directory.

  • Linux example: APP_HOME=/usr/local/ftacs/

  • Windows example: APP_HOME=D:/ftacs/dist/

On Windows, use forward slashes (/) and include the trailing slash. Avoid double slashes.

6.3.4. Deploy and Verify

Linux
cd bin/
chmod +x start.sh
./start.sh

tail -f ../logs/server.log
Windows
cd bin
start.cmd

type ..\logs\server.log

To stop the application, run ./stop.sh (or stop.cmd) from bin/.

7. Verification

7.1. Startup Log

Wait 30—​60 seconds after start, then check the logs:

# Docker Compose
docker compose logs ftacs | tail -20

# Docker standalone
docker logs ftacs | tail -20

# Manual deployment
tail -f ${APP_HOME}/logs/server.log

A successful startup produces the following output (timestamps, version, and build number vary):

INFO  [c.f.a.config.AcsAppConfig] (main)
      ==================================================
          HTTP ports: 8080,8181
          HTTPS ports: 8443
          TR069 Device URLs:
              NOAUTH: /ftacs/ACS
              BASIC: /ftacs-basic/ACS
              DIGEST: /ftacs-digest/ACS
          WebServices:
              /ACSServer-ACS/ACSWebService
              /ACSServer-ACS/ACSWebServiceAdditional
              /ACSServer-ACS/ACSWebServiceConfiguration
              /ACSServer-ACS/ACSWebServiceEventSubscription
          Swagger UI: /rest/swagger-ui
      ==================================================

INFO  [c.f.i.c.s.FTIOTApplicationListener] (main) ==================== Friendly IOT Management Server V<version> Build <build>(<build-date>) started ====================
INFO  [com.friendly.acs.FtAcsApp] (main) Started FtAcsApp in <seconds> seconds (process running for <seconds>)

The last two lines are the "ready" markers: the FTIOTApplicationListener banner reports the running version and build, and Started FtAcsApp confirms the process is up.

ClusterManager: detected failed or restarted instances during startup is normal Quartz scheduler recovery and can be ignored.

Liquibase migration messages go to liquibase.log, not server.log.

The startup banner is produced by FTIOTApplicationListener, which reads version.xml from the process working directory only — ${APP_HOME}/version.xml (/opt/FTACS/version.xml in the reference compose), next to app/, not inside conf/. The build places the file there; a config directory mounted over conf/ can no longer shadow it.

If the banner instead reads Friendly IOT Management Server Vunknown Build unknown(unknown), version.xml could not be found or parsed at that path — see Startup Banner Shows Unknown Version.

7.2. Endpoint Checks

# Application statistics -- HTTP Basic, use a web service user
curl -u <ws-user>:<ws-password> http://localhost:8080/acsstats

# TR-069 ACS endpoint -- POST only, so probe it with POST
curl -s -o /dev/null -w '%{http_code}\n' -X POST http://localhost:8080/ftacs/ACS

# SOAP WSDL -- HTTP Basic, use a web service user
curl -u <ws-user>:<ws-password> "http://localhost:8080/ACSServer-ACS/ACSWebService?wsdl"

# Swagger UI -- the entry point redirects, so follow the redirect
curl -sIL -o /dev/null -w '%{http_code}\n' http://localhost:8080/rest/swagger-ui

/acsstats and the SOAP WSDL must return HTTP 200. /ftacs/ACS is POST-only (see HTTP Endpoints) — any status other than 404 means the endpoint is mapped and served; a curl -I HEAD probe returns 405 on a healthy deployment, which is not a fault. /rest/swagger-ui redirects to the UI page, so it reaches 200 only when the redirect is followed (-L). /acsstats and the SOAP web services sit behind the ftacsws HTTP Basic realm; their users are published by FT Configs Service, so a 401 means either wrong credentials or no published configuration — see FTACS Starts but Has No Configuration. The web interface is reachable at http://localhost:8080 and https://localhost:8443.

7.3. Database Connectivity

Check that Liquibase created the schema tables:

MySQL
USE ftacs;
SHOW TABLES;
Oracle
SELECT table_name FROM user_tables ORDER BY table_name;

You should see multiple tables created by the Liquibase migrations.

8. Port Reference

Ports are written as published → container. Across hosts you connect to the published port; inside the Docker bridge you connect to the container port. All published ports are configurable through the .env variables shown in the compose file above.

8.1. HTTP / HTTPS

Port Protocol Purpose Exposure

8080 → 8080

HTTP

Main web interface: SOAP, REST, TR-069, statistics, bulk data (FTACS_HTTP_PORT)

Public (devices)

8181 → 8181

HTTP

Secondary HTTP port, the same services as 8080 (ACS_HTTP_ADDITIONAL)

Public (devices)

8443 → 8443

HTTPS

Secure web interface (FTACS_HTTPS_PORT)

Public (devices)

8.2. MQTT

Port Protocol Purpose Exposure

1883 → 1883

MQTT

MQTT broker, plain (ACS_MQTT)

Public (devices)

8883 → 8883

MQTT over TLS

MQTT broker, TLS (ACS_MQTT_TLS)

Public (devices)

8.3. USP (TR-369)

Port Protocol Purpose Exposure

8025 → 8025

TCP

USP WebSocket, plain (ACS_USP_WS)

Public (devices)

4430 → 4430

TCP

USP WebSocket Secure, WSS (ACS_USP_WSS)

Public (devices)

18881 → 18881

TCP

USP MQTT, plain (ACS_USP_MQTT)

Public (devices)

18882 → 18882

TCP

USP MQTT, TLS (ACS_USP_MQTT_TLS)

Public (devices)

18891 → 18891

TCP

USP Bulk Data receiver, plain (ACS_USP_BULK)

Public (devices)

61617 → 61617

TCP

USP STOMP, plain (ACS_USP_STOMP)

Public (devices)

61618 → 61618

TCP

USP STOMP, TLS (ACS_USP_STOMP_TLS)

Public (devices)

5783 → 5783

UDP

USP CoAP, plain (ACS_USP_CoAP)

Public (devices)

5784 → 5784

UDP

USP CoAP, DTLS

Public (devices)

8.4. LWM2M

Port Protocol Purpose Exposure

5680 → 5680

UDP

Bootstrap server, no security (BS_PORT_NOSEC)

Public (devices)

5681 → 5681

UDP

Bootstrap server, PSK (BS_PORT_PSK)

Public (devices)

5682 → 5682

UDP

Bootstrap server, X.509 (BS_PORT_X509)

Public (devices)

5880 → 5880

TCP

Bootstrap server, no security (BS_TCP_PORT_NOSEC)

Public (devices)

5683 → 5683

UDP

LWM2M server, no security (LWM2M_PORT_NOSEC)

Public (devices)

5684 → 5684

UDP

LWM2M server, PSK (LWM2M_PORT_PSK)

Public (devices)

5685 → 5685

UDP

LWM2M server, X.509 (LWM2M_PORT_X509)

Public (devices)

5883 → 5883

TCP

LWM2M server, no security (LWM2M_TCP_PORT_NOSEC)

Public (devices)

8.5. Outbound Connections

Ports FTACS dials on the existing infrastructure. These are the published ports of those services — confirm them against whoever operates them, and set the matching variables in .env.

Destination Port Protocol Purpose

Relational database

3306 (Oracle: 1521)

TCP

ACS and Quartz schemas (DB_HOST, DB_PORT)

ClickHouse HTTP interface

8123

HTTP

QoE and statistics data (CLICKHOUSE_HOST, CLICKHOUSE_HTTP_PORT)

Hazelcast cluster

5701, 5711

TCP

Distributed cache and the FT Configs Service configuration channel (HZ_MEMBERS)

8.6. HTTP Endpoints

Method Path Purpose Auth

GET

/acsstats

TR-069 protocol statistics

HTTP Basic (web service user)

POST

/ftacs/ACS

TR-069 endpoint

None

POST

/ftacs-basic/ACS

TR-069 endpoint

HTTP Basic (device credentials)

POST

/ftacs-digest/ACS

TR-069 endpoint

HTTP Digest (device credentials)

GET, POST

/ACSServer-ACS/ACSWebService

Primary SOAP web service; append ?wsdl for the contract

HTTP Basic (web service user)

GET, POST

/ACSServer-ACS/ACSWebServiceAdditional

Additional SOAP web service; append ?wsdl for the contract

HTTP Basic (web service user)

GET

/rest/swagger-ui

Swagger UI documentation interface

None

GET

/actuator/prometheus

Prometheus metrics scrape endpoint

None

The TR-069 endpoint paths are configurable through WEB_TR069_NO_AUTH, WEB_TR069_BASIC, and WEB_TR069_DIGEST; each accepts a comma-separated list, so endpoints can be added or removed.

Endpoint URL examples:

# No authentication
http://<your-server>:8080/ftacs/ACS

# Basic authentication
http://<your-server>:8080/ftacs-basic/ACS

# Digest authentication (HTTPS)
https://<your-server>:8443/ftacs-digest/ACS

# SOAP web service
http://<your-server>:8080/ACSServer-ACS/ACSWebService?wsdl

# Swagger UI
http://<your-server>:8080/rest/swagger-ui

Plain-text ports (no encryption): 1883, 5680—​5685, 5783, 5880, 5883, 8025, 8080, 8181, 18881, 18891, 61617.

Encrypted ports: 4430 (WSS), 5784 (CoAP DTLS), 8443 (HTTPS), 8883 (MQTT TLS), 18882 (USP MQTT TLS), 61618 (STOMP TLS).

In production, firewall the plain-text ports and route device traffic through the encrypted endpoints.

9. Stack Management

9.1. Logs

Command Description

docker compose logs -f ftacs

Follow the container log of the ftacs service

docker compose logs --tail 100 ftacs

Print the last 100 lines of the container log

tail -f ${APP_HOME}/logs/server.log

Follow the main application log in a manual deployment

 
# Follow logs for one service
docker compose logs -f ftacs

# Last 100 lines
docker compose logs --tail 100 ftacs

# Manual deployment
tail -f ${APP_HOME}/logs/server.log

The application writes several log files to ftacs/logs/ (Docker) or logs/ (manual):

Log file Purpose

server.log

Main application log

acsws.log

Web service calls and responses

liquibase.log

Database migration logs

sql.log

SQL query execution logs

bulk_process.log

Bulk operation processing

monitors.log

System monitoring events

db_session_killed.log

Database session management

externalTrace.log

External system integration traces

old_data_clean.log

Data cleanup operations

hikari.log

Connection pool diagnostics

9.2. Start, Stop, Restart

Command Description

docker compose ps

Show the status of every service in the stack

docker compose stop ftacs

Stop the ftacs container, keeping it defined

docker compose start ftacs

Start the stopped ftacs container again

docker compose restart ftacs

Restart the ftacs container

docker compose down

Stop and remove all containers of the stack

 
cd /usr/local/ft-system

docker compose ps                # status of every service
docker compose stop ftacs        # stop one service
docker compose start ftacs       # start it again
docker compose restart ftacs     # restart it
docker compose down              # stop and remove all containers

docker compose down -v additionally removes all named volumes, including database data. Use it only when you intend to start from scratch.

9.3. Shell Access

Command Description

docker exec -it ftacs bash

Open an interactive shell inside the ftacs container

docker exec ftacs ls -la /opt/FTACS/logs

Run a single command inside the container without a shell session

 
docker exec -it ftacs bash
docker exec ftacs ls -la /opt/FTACS/logs

9.4. Updating FTACS

See Docker Image Management Guide for the full image lifecycle.

cd /usr/local/ft-system

# 1. Back up the FTACS configuration and logs
tar -czf ftacs-backup-$(date +%Y%m%d).tar.gz ./ftacs

# 2. Update the image tag in compose.yml, then:
docker compose pull ftacs
docker compose up -d ftacs

Liquibase migrates the database schema automatically on the first start of the new version.

Back up the ACS database before pulling a new image — the Liquibase migrations are applied on first start and are not reversible from within FTACS. The database backup is taken on the database host, which is outside this stack.

For manual deployments, the upgrade procedure is:

  1. Back up the database, configuration files, and logs.

  2. Check for a data-backfill migration — see Upgrading a Large Installation. On a large installation this step is mandatory and is performed before stopping the application.

  3. Stop the application — run ./stop.sh from bin/.

  4. Extract the new version into a new directory.

  5. Migrate the configuration — copy and adapt app.env from the old version; review new options.

  6. Start the new version — Liquibase migrates the schema automatically.

  7. Verify device connections, web services, and monitoring dashboards.

  8. Roll back if needed — stop the new version, restore the database backup, start the old version.

9.5. Upgrading a Large Installation

Most Liquibase changeSets are schema changes and complete instantly regardless of installation size. A data-backfill changeSet is different: it rewrites existing rows, and its cost scales with the table. Release notes flag such releases explicitly; before upgrading one, check the size of the affected table:

SELECT COUNT(*) FROM cpe;

Up to roughly 1 million rows there is nothing to prepare — the backfill completes in seconds during startup. Above that, run the release’s pre-upgrade backfill script while the ACS is still on the old version. The script applies exactly the same predicate as the migration, in primary-key batches, so afterwards the changeSet finds nothing to do and only pays for a table scan.

Table 1. Measured on MySQL 8.0, backfill of cpe.location_id (the column is in no index, so only the clustered index is touched)
Rows in cpe Migration without the pre-upgrade script Migration after it

1 million

5.8 s

0.3 s

5 million

30.4 s

1.5 s

25 million

~2.5 min (extrapolated)

~8 s

On a large installation, skipping the pre-upgrade script does not merely make the upgrade slow — it takes the service down.

  • The whole table is closed for writes. Under MySQL’s default REPEATABLE READ isolation the backfill takes next-key locks on every row it scans, not only on the rows it changes. Once this exceeds innodb_lock_wait_timeout (50 s by default), any node still serving device traffic fails its writes. Stop the entire cluster for the upgrade — do not upgrade node by node.

  • One transaction carries the whole change set. Expect gigabytes of undo/redo/binlog, replicas falling behind, and on Oracle an UNDO tablespace large enough to hold it (ORA-30036 otherwise).

  • An interrupted migration blocks every later start. If the process is killed mid-transaction — a Kubernetes startup probe, an OOM kill, a manual restart — the InnoDB rollback is slower than the forward pass, and the Liquibase lock stays held. Every subsequent start then fails with Could not acquire change log lock until the lock is cleared manually:

    Stop every ACS node first — clearing the lock while a node still holds it lets two nodes migrate concurrently. In a container deployment that means stopping the container, not just the process, which would otherwise restart straight back into the same migration.

DELETE FROM DATABASECHANGELOGLOCK;

+ A second cluster node waits only changelogLockWaitTimeInMinutes (5 by default) before failing the same way.

FTACS starts Liquibase in strict mode (app.liquibase.strict, default true): a failed migration aborts startup rather than letting the application run against a half-migrated schema. Allow enough time in the startup probe — or the manual start — for the migration to finish.

9.5.1. Release 7.1.2 (DEV-2749) — location_id Backfill

Release 7.1.2 is such a release: it backfills cpe.location_id and cpe_credentials.isp_id.

The migration is irreversible — it preserves the previous values nowhere.

The pre-upgrade script is recoverable: it exports every row it is about to change, and dev2749-restore.sh replays those exports (Undoing a backfill). That recovery exists only if you keep both export files, and only for rows nothing has written since.

Before running either one:

  1. Back up cpe and cpe_credentials.

  2. Run the verification queries (Verification queries) and keep the output. Sections 4a and 4b list the rows that are about to change in both tables — export both, they are the only record of the previous values. Exporting only cpe leaves cpe_credentials.isp_id overwritten with no record. Skip section 4 entirely if you are running dev2749-backfill.sh: it writes both exports itself.

  3. Note the category F count. Those rows are ambiguous and are left untouched on purpose; see Known limitation. Section 6 of the verification queries lists them individually — it is the only place that does.
Pre-upgrade backfill script

The backfill is delivered as a single script, dev2749-backfill.sh, which does the same work as the migration but in batches of existing rows: each statement locks only its own range, and between batches the table is free. Re-running is safe: the predicate is self-limiting, so a second run only touches what is left. Interrupting is safe for the database — every batch is committed — but there is no checkpoint: a re-run rescans from the lowest id, and it must be given a fresh --outdir, which the script enforces. After an interruption the first run’s export directory is the complete record.

Three files are published together on the FT shared drive, at the top level of the release folder (not in the ACS subfolder inside it, which holds deployment config): dev2749-backfill.sh, dev2749-restore.sh and DEV-2749-verify.sql.

chmod +x dev2749-backfill.sh dev2749-restore.sh

If the files travelled through Windows on their way to an air-gapped host, strip the line endings first — a CRLF script fails with bad interpreter: /bin/bash^M:

sed -i 's/\r$//' dev2749-backfill.sh dev2749-restore.sh
Where to run it, and what has to be installed

The scripts need only a database client: mysql for MySQL, sqlplus for Oracle. Everything else they use is bash plus coreutils (printf, awk, sed, grep, sort, tr, tail, head, wc, tee, mktemp, date, mkdir, chmod, rm), which every Linux server already has. Nothing is installed on the database itself, and nothing is fetched from the internet at run time.

On an offline site, run the scripts on the database server. That is the zero-installation route:

  • Oracle — sqlplus is not part of any Linux distribution, but the database server already has it at $ORACLE_HOME/bin/sqlplus. The scripts connect through //host:port/service, so running them on the database server with --host localhost behaves identically.

  • MySQL — the mysql client is on the database server too, and is an ordinary distribution package (mysql-client / mariadb-client) if you would rather use another host.

If the scripts must run on a host that has no client, install Oracle Instant Client there: the Basic (or Basic Lite) package plus the SQL*Plus package — two archives unpacked into one directory, with PATH and LD_LIBRARY_PATH pointed at it. Download them once on a connected machine and carry them across with the release; no network is needed while the scripts run.

A missing client is reported before any connection is attempted — after the password has been supplied and the output directory created, so a run.log already exists by then: ERROR: the 'mysql' client is not on PATH / ERROR: 'sqlplus' is not on PATH.

Privileges. --check needs SELECT on cpe, cpe_credentials, isp and product_class_group; --run additionally needs UPDATE on cpe and cpe_credentials, which it verifies before asking for confirmation. dev2749-restore.sh needs the same rights, plus SELECT on DATABASECHANGELOG — without it the post-upgrade guard cannot fire and the script says so and continues (Undoing a backfill).

The password is read from the DB_PASSWORD environment variable, or prompted for. It is never placed in argv, so it cannot be read out of ps. Prefer the prompt: writing DB_PASSWORD=secret ./dev2749-backfill.sh …​ on the command line does put it in the shell history and in the environment of the process and its children.

Running the backfill
Table 2. How long it takes
Phase 1M rows in cpe 10M rows in cpe

--check (the category report, twelve full scans)

~1 s

~48 s

--run backfill (300k and 3M rows healed)

~4 s

~81 s

dev2749-restore.sh --run on the same export

~10 s

~215 s

Each figure is the whole command, wall clock, start to finish — for --run that includes the category report, both exports and the closing report, not the batches alone. Measured on MySQL 8.0/8.4 with the production schema, 30% of rows affected, and the default --batch 50000. Cost is linear in the size of cpe, so scale the 10M column for a larger fleet.

Oracle was verified functionally at the same sizes but is materially slower — assume several times the MySQL figure and size the maintenance window accordingly. No Oracle number is quoted here because the measurements available were taken through a test harness whose per-statement overhead dominates them, and a misleading number is worse than none. The restore costs more than the backfill: it makes two passes over the exported rows, one to determine exactly which of them are still restorable and one to write them.

Step 1 — report only, changes nothing:

# MySQL
./dev2749-backfill.sh --check --engine mysql \
    --host db.example.com --port 3306 --user ftacs --database ftacs

# Oracle
./dev2749-backfill.sh --check --engine oracle \
    --host db.example.com --port 1521 --user ftacs --service FREEPDB1

It prints the row count of cpe and the category breakdown for both tables — the same categories as Verification queries. Category A plus category B is the number of rows the upgrade would rewrite. The report costs twelve full table scans, so expect it to take a while on a large installation before the first numbers appear; the script says so as it starts.

Step 2 — run the backfill (same options, --run instead of --check), with the ACS still on the old version:

./dev2749-backfill.sh --run --engine mysql \
    --host db.example.com --port 3306 --user ftacs --database ftacs

--run asks for an explicit yes before touching anything, then:

  1. exports every row it is about to change to <outdir>/cpe-before-backfill.tsv and <outdir>/cpe_credentials-before-backfill.tsv — keep both files until the post-upgrade verification (After the upgrade) has passed; together they are the only record of the previous values and dev2749-restore.sh refuses to run without both;

  2. writes <outdir>/manifest.txt, which records the engine, host and schema the export came from;

  3. backfills both tables in batches, printing progress and writing a durable checkpoint to run.log roughly every million rows scanned (a progress trace for support — not a resume point; see the re-run rule above);

  4. prints the category breakdown again.

--run requires a terminal, because it asks for the typed confirmation before an irreversible change. nohup and < /dev/null are refused with a message. A plain & is not detected: the run gets as far as the confirmation prompt and the shell then stops it on SIGTTIN, with no message at all. Use screen or tmux for a long run. --check is fully non-interactive with DB_PASSWORD set.

Almost everything it prints also goes to <outdir>/run.log, together with the database client’s own error output — that log is what support should be asked for. Two things are not in it: the rewriting progress line, which is terminal-only by design, and any failure that happens before --outdir exists (a missing client, a bad password, a refused non-fresh --outdir), which stays on the terminal because there is no log file yet. By default <outdir> is ./dev2749-<timestamp>; use --outdir to place it elsewhere, and --batch to change the batch size (default 50000). A re-run must be given a fresh --outdir; the script refuses to overwrite an earlier export, and refuses in the first second rather than after the scans.

Allow roughly 70 bytes per affected row across the two export files — about 250 MB for a 12-million-row installation with 30% affected. The script prints its own estimate before exporting, and aborts if the export comes out shorter than the count it just took.

A and B reach 0 only on a quiesced database. This script is meant to run while the old ACS is still serving devices, and the old ACS keeps writing the defect, so a small residue in categories A and B is normal and expected in the closing report. The 7.1.2 migration clears whatever is left — and now finds so little to do that it costs only the table scan. A and B must be 0 when you run the verification after the upgrade (After the upgrade), which is a different check at a different time.

The script is safe on a small installation too. It reports that the migration would complete on its own — but note that --run still heals whatever it finds, so use --check if you only want the report.
Undoing a backfill

dev2749-backfill.sh --run cannot undo itself, but the two *-before-backfill.tsv files it writes are a complete record of the previous values, and dev2749-restore.sh replays them. (The two files are cpe-before-backfill.tsv and cpe_credentials-before-backfill.tsv.)

Never restore after the 7.1.2 upgrade. The migration is already recorded in DATABASECHANGELOG and will not run again, while the upgraded runtime re-heals each device on check-in — so a post-upgrade restore is undone slowly and invisibly, leaving a partially restored fleet. The script checks DATABASECHANGELOG and refuses unless --force-after-upgrade is given.

The check is best-effort and is not a guarantee. Where DATABASECHANGELOG is missing, or the connecting user has no SELECT on it, the script prints WARNING: could not read DATABASECHANGELOG …​ the post-upgrade guard therefore did NOT run and continues. Treat that warning as no protection at all: establish from the change log yourself whether 7.1.2 has been applied before you type the confirmation.

--force-after-upgrade exists for Friendly Technologies support. Do not use it on your own judgement.

 
# report only - validates the files and names exactly which rows would come back
./dev2749-restore.sh --check --indir ./dev2749-20260831-120000 --engine mysql \
    --host db.example.com --port 3306 --user ftacs --database ftacs

# restore (the confirmation word is "restore", not the backfill's "yes")
./dev2749-restore.sh --run --indir ./dev2749-20260831-120000 --engine mysql \
    --host db.example.com --port 3306 --user ftacs --database ftacs

--indir is the backfill run’s --outdir. Both TSV files must be present; a directory holding only one is refused.

The safety rule

A row is restored only if it is still at 0 and still carries the updated timestamp the export recorded. The backfill does not touch updated, so a row that still has its exported timestamp is a row nothing has written since.

This is deliberately stricter than a bare 0 check, and on purpose: an administrator who moved a device to the super domain after the backfill also leaves it at 0, and that assignment must not be reverted. Rows skipped for any reason are listed by id in left-alone-cpe.txt and left-alone-cpe_credentials.txt, so there is always a definite answer to "which rows still need a human".

Provenance. The backfill writes manifest.txt naming the engine, host and schema the export came from, and the restore refuses an export taken from a different database — device ids are autoincrements starting at 1 on every installation, so a wrong --indir would rewrite unrelated devices with plausible-looking counts and no error. Override with --ignore-source-mismatch only when you know why they differ.

Table 3. What it cannot recover
Case Why

Rows the live old ACS poisoned after the export was taken

The backfill healed them, but no file holds their previous value. run.log reports how many.

Rows whose device was deleted after the backfill

The id no longer exists.

A run whose export files were lost

There is no second copy.

Category F (ambiguous) rows

The backfill never touched them, so there is nothing to restore. See Known limitation.

Other order-of-operations rules:

  • After an interrupted backfill, restore from the first run’s directory — but keep the later one too. Rows the live old ACS poisoned between the two runs are recorded only in the later export, so neither directory alone is complete. Restore from the first, then from the later one; the safety rule makes the second pass a no-op for anything already restored.

  • Restoring twice is safe and a no-op the second time.

  • After a restore, re-run the backfill before upgrading — a restored database carries the defect again.

  • After a restore, re-run the verification queries and read left-alone-cpe.txt / left-alone-cpe_credentials.txt. Those are the rows the restore deliberately skipped, and the only ones still needing a decision.

Output goes to ./dev2749-restore-<timestamp>/: restore.log, the two left-alone-*.txt lists, would-restore-*.txt (the ids the probe matched), and restore-*.sql / probe-*.sql — the exact statements executed. The SQL carries a provenance header, is re-runnable by hand, and is safe to re-run because every statement only touches rows still at 0 with their exported timestamp.

After a failed restore, support should be given: restore.log, both restore-*.sql, both left-alone-*.txt and would-restore-*.txt, and the backfill run’s run.log, manifest.txt and both *-before-backfill.tsv.

Verification queries

Run before the upgrade to size the work and record the previous values, and after it to confirm the outcome. Portable as written — works on both MySQL 8 and Oracle.

-- 1. cpe.location_id - one row per category
SELECT 'A null (heals to 0)'            AS category, COUNT(*) AS rows_affected FROM cpe c
 WHERE c.location_id IS NULL
UNION ALL
SELECT 'B poisoned (heals to 0)',       COUNT(*) FROM cpe c
 WHERE c.location_id IS NOT NULL AND c.location_id != 0
   AND NOT EXISTS (SELECT 1 FROM isp i WHERE i.id = c.location_id)
   AND     EXISTS (SELECT 1 FROM product_class_group g WHERE g.id = c.location_id)
UNION ALL
SELECT 'C real domain (untouched)',     COUNT(*) FROM cpe c
 WHERE c.location_id IS NOT NULL AND c.location_id != 0
   AND EXISTS (SELECT 1 FROM isp i WHERE i.id = c.location_id)
   AND NOT EXISTS (SELECT 1 FROM product_class_group g WHERE g.id = c.location_id)
UNION ALL
SELECT 'D orphaned domain (untouched)', COUNT(*) FROM cpe c
 WHERE c.location_id IS NOT NULL AND c.location_id != 0
   AND NOT EXISTS (SELECT 1 FROM isp i WHERE i.id = c.location_id)
   AND NOT EXISTS (SELECT 1 FROM product_class_group g WHERE g.id = c.location_id)
UNION ALL
SELECT 'E already super domain',        COUNT(*) FROM cpe c
 WHERE c.location_id = 0
UNION ALL
SELECT 'F AMBIGUOUS - needs review',    COUNT(*) FROM cpe c
 WHERE c.location_id IS NOT NULL AND c.location_id != 0
   AND EXISTS (SELECT 1 FROM isp i WHERE i.id = c.location_id)
   AND EXISTS (SELECT 1 FROM product_class_group g WHERE g.id = c.location_id);

-- 2. cpe_credentials.isp_id - same categories (isp_id is NOT NULL DEFAULT 0)
SELECT 'B poisoned (heals to 0)'        AS category, COUNT(*) AS rows_affected FROM cpe_credentials cc
 WHERE cc.isp_id != 0
   AND NOT EXISTS (SELECT 1 FROM isp i WHERE i.id = cc.isp_id)
   AND     EXISTS (SELECT 1 FROM product_class_group g WHERE g.id = cc.isp_id)
UNION ALL
SELECT 'C real domain (untouched)',     COUNT(*) FROM cpe_credentials cc
 WHERE cc.isp_id != 0
   AND EXISTS (SELECT 1 FROM isp i WHERE i.id = cc.isp_id)
   AND NOT EXISTS (SELECT 1 FROM product_class_group g WHERE g.id = cc.isp_id)
UNION ALL
SELECT 'D orphaned domain (untouched)', COUNT(*) FROM cpe_credentials cc
 WHERE cc.isp_id != 0
   AND NOT EXISTS (SELECT 1 FROM isp i WHERE i.id = cc.isp_id)
   AND NOT EXISTS (SELECT 1 FROM product_class_group g WHERE g.id = cc.isp_id)
UNION ALL
SELECT 'E already super domain',        COUNT(*) FROM cpe_credentials cc
 WHERE cc.isp_id = 0
UNION ALL
SELECT 'F AMBIGUOUS - needs review',    COUNT(*) FROM cpe_credentials cc
 WHERE cc.isp_id != 0
   AND EXISTS (SELECT 1 FROM isp i WHERE i.id = cc.isp_id)
   AND EXISTS (SELECT 1 FROM product_class_group g WHERE g.id = cc.isp_id);

-- 3. Context: how large are the two id spaces, and do they overlap?
SELECT 'isp rows'                 AS metric, COUNT(*) AS value FROM isp
UNION ALL
SELECT 'product_class_group rows',           COUNT(*) FROM product_class_group
UNION ALL
SELECT 'overlapping ids',                    COUNT(*) FROM isp i
 WHERE EXISTS (SELECT 1 FROM product_class_group g WHERE g.id = i.id);

-- 4. The rows that are about to change - EXPORT THESE BEFORE UPGRADING.
--    Their previous values are preserved nowhere else. BOTH tables are rewritten, so
--    BOTH have to be exported: exporting only cpe leaves cpe_credentials.isp_id
--    irreversibly overwritten with no record.
--    4a. cpe.location_id
SELECT c.id, c.serial, c.location_id, c.product_class_id, c.protocol_id, c.updated
  FROM cpe c
 WHERE c.location_id IS NULL
    OR (c.location_id != 0
        AND NOT EXISTS (SELECT 1 FROM isp i WHERE i.id = c.location_id)
        AND EXISTS (SELECT 1 FROM product_class_group g WHERE g.id = c.location_id))
 ORDER BY c.id;

--    4b. cpe_credentials.isp_id - the other half of the irreversible change.
--    Note cpe_deleted_at: rows of soft-deleted devices are rewritten too, which moves
--    them into a different cpe_credentials_retention_policy.isp_id bucket.
SELECT cc.id, cc.serial, cc.isp_id, cc.group_id, cc.cpe_id, cc.cpe_deleted_at, cc.updated
  FROM cpe_credentials cc
 WHERE cc.isp_id != 0
   AND NOT EXISTS (SELECT 1 FROM isp i WHERE i.id = cc.isp_id)
   AND EXISTS (SELECT 1 FROM product_class_group g WHERE g.id = cc.isp_id)
 ORDER BY cc.id;

-- 5. dev2749-backfill.sh produces exactly these two exports itself, as
--    cpe-before-backfill.tsv and cpe_credentials-before-backfill.tsv. If you run the
--    script, you do not need to run section 4 by hand.

-- 6. Category F, LISTED rather than counted. Every other place - sections 1 and 2 above,
--    dev2749-backfill.sh --check, the changelog comment and the PR - reports category F
--    as a COUNT and tells you to correct those rows by hand. This is the only place that
--    names them. Neither the migration, nor the pre-upgrade script, nor the runtime fix
--    touches them: the value is both a real domain and a product_class_group id, so a
--    poisoned row is indistinguishable from a legitimate assignment. A human decides.
SELECT c.id, c.serial, c.location_id, c.product_class_id, c.protocol_id, c.updated
  FROM cpe c
 WHERE c.location_id IS NOT NULL AND c.location_id != 0
   AND EXISTS (SELECT 1 FROM isp i WHERE i.id = c.location_id)
   AND EXISTS (SELECT 1 FROM product_class_group g WHERE g.id = c.location_id)
 ORDER BY c.id;

SELECT cc.id, cc.serial, cc.isp_id, cc.group_id, cc.cpe_id, cc.cpe_deleted_at, cc.updated
  FROM cpe_credentials cc
 WHERE cc.isp_id != 0
   AND EXISTS (SELECT 1 FROM isp i WHERE i.id = cc.isp_id)
   AND EXISTS (SELECT 1 FROM product_class_group g WHERE g.id = cc.isp_id)
 ORDER BY cc.id;

The same queries ship as DEV-2749-verify.sql next to the scripts. Capture the output to a file rather than reading it on screen — sections 4a, 4b and 6 return one row per affected device:

mysql --batch ... < DEV-2749-verify.sql > dev2749-verify-$(date +%F).tsv

On Oracle the shipped file has no SET, SPOOL or EXIT of its own, so run it from a wrapper that adds them — otherwise the output wraps across the terminal and you are left at a SQL> prompt:

SET LINESIZE 32767 PAGESIZE 0 FEEDBACK OFF
SPOOL dev2749-verify.txt
@DEV-2749-verify.sql
SPOOL OFF
EXIT

Expected after the upgrade: A = 0, B = 0, C and D unchanged, E = Ebefore + Abefore + Bbefore, F unchanged.

After the upgrade

From this point on, do not run dev2749-restore.sh. The migration is recorded in DATABASECHANGELOG and will not run again, while the upgraded runtime re-heals each device on check-in — so a restore is undone slowly and invisibly. The script detects this and refuses — unless it cannot read DATABASECHANGELOG, in which case it warns and continues; see Undoing a backfill.

  1. Re-run the verification queries (Verification queries). Categories A and B must both be 0 now — the upgraded ACS no longer writes the defect, so unlike the pre-upgrade script’s closing report there is no residue to expect.

  2. Compare against the output you kept from before the upgrade: E should equal E_before + A_before + B_before, while C, D and F are unchanged.

  3. Category F is still non-zero if it was before. Those rows were never touched by anything; section 6 of the verification queries lists them individually so they can be corrected by hand. See Known limitation.

  4. Verify device connections and domain assignment in the UI. If a device turns out to be in the wrong domain, see A device is in the wrong domain after the upgrade.

  5. If the ACS failed to start with Could not acquire change log lock, the previous attempt was interrupted mid-migration. Clear the lock and start again:

    Stop every ACS node first — clearing the lock while a node still holds it lets two nodes migrate concurrently. In a container deployment that means stopping the container, not just the process, which would otherwise restart straight back into the same migration.

DELETE FROM DATABASECHANGELOGLOCK;
A device is in the wrong domain after the upgrade

This is the question support is most likely to be asked, and it is answerable from the artefacts the pre-upgrade run left behind.

  1. Was the device changed by the backfill, and from what? Look the serial up in cpe-before-backfill.tsv from the pre-upgrade run. Column 1 is the id, column 2 the serial and column 3 the location_id the backfill replaced with 0:

    grep -P "^\d+\t<serial>\t" cpe-before-backfill.tsv

    A serial present in that file was changed, and column 3 is its previous value. Check cpe_credentials-before-backfill.tsv the same way for isp_id.

  2. Absent from both files? Then the backfill did not touch it. Two things still could have: category F (Known limitation), which nothing corrects automatically — run section 6 of Verification queries and look for the id; or a row the still-live old ACS poisoned after the export was taken, which no file records. run.log reports how many of those there were, but not which.

  3. If a restore was run, left-alone-cpe.txt lists the ids it deliberately did not put back.

The export files are tab-separated with no quoting. A NULL is an empty field, and inside a text field a backslash, TAB, LF, CR and NUL appear as \\, \t, \n, \r, \0. The two engines produce byte-identical files.

Known limitation

A poisoned value that happens to equal a real domain id cannot be told apart from a legitimate assignment: both id sequences are independent and both start low, so such collisions are common rather than exceptional. Those rows are reported as category F by the verification queries and by dev2749-backfill.sh --check, and neither the migration, nor the pre-upgrade script, nor the runtime fix touches them — they need manual review. Every one of those places reports only a count; section 6 of Verification queries is the only place that lists the rows, so start there when you have to correct them. dev2749-restore.sh cannot help either: the backfill never changed these rows, so there is nothing to restore.

Deciding a row means comparing its location_id against the domain the device’s subscriber actually belongs to — the customer’s own account record, or the domain of its sibling devices on the same account. A value that matches the device’s product-class grouping rather than its subscriber’s domain is the poisoned case. No query can decide this; each row needs the customer’s own tenancy data. If the count is large enough that per-row review is impractical, raise it with Friendly Technologies support before upgrading, quoting the overlapping ids figure from section 3 of Verification queries.

10. Production Checklist

  • Change all default passwords in .env / app.env (DB_PASSWORD, CLICKHOUSE_PASSWORD and SSL_KEYSTORE_PASSWORD).

  • Use a valid SSL certificate — replace the self-signed ftacs.keystore with a CA-issued certificate.

  • Configure firewall rules — allow only the required ports; restrict database access to ACS hosts only.

  • Review TR-069 authentication — disable the no-auth endpoint (/ftacs/ACS) if it is not required.

  • Secure configuration files — restrict permissions: chmod 600 bin/app.env.

  • Enable auto-restart — restart: unless-stopped in compose.yml, or --restart unless-stopped with docker run.

  • Verify health checks — the compose file defines them for every service; add --health-cmd for standalone docker run.

  • Set up monitoring — deploy Prometheus and Grafana against /actuator/prometheus and the ClickHouse data source.

  • Configure backups — regular database backups, a configuration directory backup, a ClickHouse backup strategy, and log rotation.

  • Deploy multiple FTACS instances for high availability — identical configuration, an external Hazelcast cluster with 2+ nodes, and a load balancer in front.

  • Tune JVM settings — match the JVM sizing table to your deployment size.

11. Troubleshooting

11.1. Container Fails to Start

Symptom: The ftacs container exits immediately after docker compose up.

Fix:

  1. Read the error message from the logs:

    docker compose logs ftacs
docker events --filter container=ftacs

  2. Confirm the environment files are loaded:

    docker exec ftacs env | grep DB_

  3. Confirm ftacs/config/ exists and is populated — an empty mount at /opt/FTACS/conf is a common cause.

11.2. Database Connection Failure

Symptom: Startup fails with a JDBC connection error, or Liquibase cannot acquire a lock.

Fix:

  1. Test connectivity to the database. It is external to this stack and addressed by IP through DB_HOST (see Startup Dependencies), so there is no mysql container to reach or to read logs from — probe the address from the host and read the JDBC error in the FTACS log:

    # use the DB_HOST / DB_PORT values from .env
nc -zv <db-host> <db-port>

docker logs ftacs 2>&1 | grep -i -E 'jdbc|liquibase|hikari'

  2. Verify DB_HOST, DB_PORT, DB_USER, and DB_PASSWORD in .env match the database configuration.

  3. If the database was just created, wait for its health check to pass before starting FTACS.

11.3. ClickHouse Connections Fail

Symptom: QoE and statistics queries fail; the log reports a ClickHouse connection or protocol error.

Fix:

  1. Confirm FTACS targets the HTTP port 8123, not the native TCP port 9000 — FTACS uses the clickhouse-java HTTP client. In the compose file this is the CLICKHOUSE_PORT: "$\{CLICKHOUSE_HTTP_PORT:-8123}" remap.

  2. Verify ClickHouse answers, from inside the container so the check uses the same network path FTACS does:

    docker exec ftacs curl -s http://${CLICKHOUSE_HOST}:8123/ping

  3. If ClickHouse answers but federated queries against the relational database fail, the JDBC Bridge on the ClickHouse host is the likely cause — report it to whoever operates that stack.

11.4. FTACS Starts but Has No Configuration

Symptom: FTACS reaches Started FtAcsApp, but web service calls are rejected as unauthorised, ACS configuration values fall back to defaults, and the parameter-name cache rules have no effect. No error appears in server.log.

Cause: since 6.5.2 this configuration is owned by FT Configs Service and delivered through Hazelcast. FTACS finds no published snapshot — either the service is not running, or the two are on different Hazelcast clusters.

Fix:

  1. Verify FT Configs Service is healthy:

    curl -s http://<configs-service-host-ip>:8087/configs-service/actuator/health

  2. Compare HZ_MEMBERS in FTACS’s .env with the value used by FT Configs Service — both must resolve to the same cluster.

  3. Compare the cluster name in ftacs/config/hazelcast-client.xml with the one FT Configs Service uses (default: dev). A mismatch produces two separate clusters that never see each other.

  4. Confirm the configuration was actually saved in FT Configs UI — an empty service publishes empty snapshots, which looks identical from the FTACS side.

  5. Do not work around this by putting acs_configuration.xml and friends back into ftacs/config/ — FTACS no longer reads them from disk.

11.5. Startup Banner Shows Unknown Version

Symptom: The startup banner in server.log reads Friendly IOT Management Server Vunknown Build unknown(unknown) started instead of an actual version and build number. The application otherwise starts normally.

Cause: FTIOTApplicationListener reads version.xml from the process working directory (${APP_HOME}/version.xml, /opt/FTACS/version.xml in the reference compose) and degrades every field to unknown if the file is missing, unreadable, or fails to parse. There is deliberately no fallback to any other location — in particular, a version.xml placed under conf/ is never read.

Fix:

  1. Search server.log for the FTIOTApplicationListener WARN logged at the same startup — it names the exception class (missing file vs. parse failure) and the resolved path that was tried.

  2. Confirm the file actually shipped with the image or distribution:

    # Docker
docker exec ftacs ls -l /opt/FTACS/version.xml

# Manual deployment
ls -l ${APP_HOME}/version.xml

    Its absence points at a build or packaging problem, not a runtime configuration issue — version.xml is produced by the Maven build and copied into the distribution/image; it is not something an operator edits or mounts.

  3. A version.xml left over under conf/ (or /opt/FTACS/conf/) from an older installation is inert — it is no longer read and does not need to be removed, though it can be deleted to avoid confusion.

11.6. Hazelcast Cluster Issues

Symptom: The log shows Unable to connect to any address in the config or repeated Hazelcast client reconnects.

Fix:

  1. Verify HZ_MEMBERS in .env lists addresses this host can actually reach. The Hazelcast cluster is a prerequisite deployed outside this guide (see Prerequisites), so use the member host IPs — localhost or a bridge alias only works when a member runs on this host and on the same Docker network.

  2. Ensure ports 5701 and 5711 are open from this host to every Hazelcast member.

  3. Check that the cluster name in ftacs/config/hazelcast-client.xml — the only Hazelcast file this installation owns — matches the name the cluster was started with (default: dev). The member-side hazelcast.xml is not part of this installation; ask whoever operates the cluster.

11.7. Port Already in Use

Symptom: docker compose up fails with bind: address already in use.

Fix:

  1. Find the process holding the port:

    sudo lsof -i :8080
netstat -tulpn | grep 8080

  2. Either stop that process, or change the published port through the corresponding .env variable (for example FTACS_HTTP_PORT=9080) and run docker compose up -d again.

11.8. Permission Denied on Mounted Volumes

Symptom: The container cannot write to ftacs/logs or the database cannot write to its data directory.

Fix:

sudo chown -R $(whoami):$(whoami) /usr/local/ft-system
chmod -R 755 /usr/local/ft-system

11.9. Config Directory Mounted Read-Only

Symptom: mac_path.csv and gpv_models.csv in ftacs/config/ never update, or stay stale, even though QoE MAC-path and GPV-model data has changed. server.log shows an IOException (or similar file-write failure) logged from MacPathsSyncUtil or GpvModelsSyncUtil. On startup with an older image, docker logs ftacs may also show:

cp: cannot create regular file '/opt/FTACS/conf/version.xml': Read-only file system
WARN: could not sync version.xml into /opt/FTACS/conf

Cause: /opt/FTACS/conf (ftacs/config/ on the host) is mounted :ro. Three files are read from there at startup, but the QoE module also writes mac_path.csv and gpv_models.csv into the same directory at runtime. Both writers only log the IOException rather than failing startup, so the process keeps running and the failure is easy to miss — see Directory Layout for what this directory holds.

Fix:

  1. Remove the :ro suffix from the ftacs/config volume mount in compose.yml or the docker run command and restart the container.

  2. Confirm the fix by checking that mac_path.csv / gpv_models.csv timestamps advance and that no further IOException is logged from the QoE sync utilities.

11.10. Out of Memory

Symptom: java.lang.OutOfMemoryError in server.log, or the container is killed by the OOM killer.

Fix:

  1. Raise the container memory limit (Docker Desktop: Settings → Resources → Memory).

  2. Increase the heap through FT_ACS_JAVA_RAM in .env (Docker) or JAVA_OPTS in app.env (manual), matching the JVM sizing table.

  3. If the problem persists, capture a heap dump and analyse it for leaks.

11.11. Containers Not Reachable From the Host

Symptom: The container is healthy and answers on localhost inside the container, published ports look correct (0.0.0.0:PORT→PORT), but requests from the host hang and time out — no ECONNREFUSED, only timeouts.

Cause: the host firewall blocks traffic on Docker bridge interfaces (br-*) in the host INPUT/OUTPUT path. This is not a Compose or application issue.

Fix:

  1. Allow host-to-container traffic on Docker bridges:

    # IPv4 (all Docker bridges)
iptables -I INPUT  1 -i br+ -j ACCEPT
iptables -I OUTPUT 1 -o br+ -j ACCEPT

# IPv6 requires separate rules
ip6tables -I INPUT  1 -i br+ -j ACCEPT
ip6tables -I OUTPUT 1 -o br+ -j ACCEPT

# Persist (Debian/Ubuntu)
apt-get install -y iptables-persistent
netfilter-persistent save

  2. Confirm the diagnosis step by step:

    # 1. Container works internally
docker exec ftacs curl -s -o /dev/null -w '%{http_code}\n' http://localhost:8080

# 2. Container IPv4 address
docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' ftacs

# 3. Host -> container IP
nc -vz <container_ipv4> 8080

# 4. Identify the Docker bridge
ip -4 addr | grep '^.*br-'

# 5. Inspect the firewall
iptables -S INPUT
iptables -S OUTPUT

IPv4 working does not imply IPv6 works — they are separate firewall stacks. Docker programs NAT and FORWARD rules but does not override host INPUT/OUTPUT policies.

11.12. Quartz Cluster Recovery Messages

Symptom: ClusterManager: detected failed or restarted instances appears during startup.

Fix: No action needed. This is normal when the application restarts or a previous instance was not shut down cleanly; Quartz recovers jobs from failed instances automatically.

11.13. Application Fails to Start (Manual Deployment)

Symptom: start.sh returns but no process is listening; server.log is empty or shows an early failure.

Fix:

  1. Verify Java: java -version must report JDK 25.

  2. Check JAVA_HOME and APP_HOME in app.env — on Windows APP_HOME must use forward slashes and a trailing slash (D:/ftacs/dist/).

  3. Verify the database is reachable: telnet <DB_HOST> <DB_PORT>.

  4. Check the credentials in app.env.

  5. Review logs/liquibase.log for SQL errors.

11.14. Slow Performance

Symptom: High response times, growing session backlog, or CPE session timeouts.

Fix:

  1. Check relational database query performance and index health.

  2. Verify Hazelcast cache hit ratios.

  3. Review the HikariCP pool sizes in ftacs/.env (FTACS_MAX_POOL, QUARTZ_MAX_POOL, CLICKHOUSE_MAX_POOL).

  4. Measure network latency to the database and to Hazelcast.

  5. Increase the JVM heap per the JVM sizing table.

  6. Inspect the Grafana dashboards fed by /actuator/prometheus.

11.15. Getting Support

If the issue persists, collect the following and contact Friendly Tech support:

  1. .env and ftacs/.env (or app.env) with passwords masked.

  2. logs/server.log — the last 1000 lines.

  3. logs/liquibase.log if the problem is database-related.

  4. java -version output.

  5. Operating system version.

  6. Database type and version.

  7. docker version output, if applicable.