Troubleshooting

Hazelcast connectivity

Symptoms:

  • Startup fails in cache services that treat Hazelcast as mandatory (e.g., TabCacheService throws on init failure).

  • Some caches may disable themselves when Hazelcast is offline (ColumnDefinitionCacheService catches HazelcastClientOfflineException and sets cache to null).

Checks:

  • Ensure Hazelcast members are reachable from the service network.

  • Verify HZ_MEMBERS is set correctly in the deployment environment.

  • Verify hazelcast-client.cluster-name matches the cluster (dev in production for this system).

Liquibase migration failures

Symptoms:

  • Application fails at startup with Liquibase errors.

Checks:

  • Inspect db/changelog/db.changelog-master.yaml ordering and the failing change set.

  • Confirm DB user has DDL permissions for initial install and migrations.

502 Bad Gateway from the UI proxy

Symptoms:

  • The UI loads (login page, static assets, fonts return 200), but every API call under /configs-service/** (auth/me, csrf, auth/login, environment/snapshot) returns 502 Bad Gateway from nginx.

  • The UI container log shows, for each proxied request: [error] <host> could not be resolved (2: Server failure) (DNS SERVFAIL).

Cause:

  • The UI’s nginx is configured to proxy the API to a public FQDN (e.g. BACKEND_URL=http://qa63.friendly-tech.com:8086). nginx must resolve that name before proxying, but the resolution fails inside the container, so nginx has no upstream to forward to and returns 502.

  • Static assets are served from local files (no DNS needed), which is why only the proxied API calls fail — the backend process itself is healthy and listening (verify with ss -ltnp | grep 8080 on the host and curl http://localhost:8080/configs-service/actuator/health).

  • A common root trigger: the public FQDN resolves to IPv6 only (an AAAA record, no A record), while the Docker bridge network is IPv4-only. The container’s embedded DNS (127.0.0.11) returns SERVFAIL for the IPv4 (A) lookup. Confirm with getent hosts <fqdn> on the host (shows an IPv6 address) versus docker exec <ui-container> getent hosts <fqdn> (returns SERVFAIL).

Fixes:

  • Preferred — point the UI at the backend by its internal Compose/Kubernetes service name instead of the public FQDN, so resolution stays on the container network’s DNS and never leaves it:

    BACKEND_URL=http://ft-configs-service:8080

    This is the cleanest fix and removes the dependency on external DNS entirely. The UI and backend must share a Docker network (same Compose project / namespace).

  • If the public FQDN must be kept, restore the missing A (IPv4) record in DNS, or add an extra_hosts entry to the UI container mapping the FQDN to the host’s IPv4 (e.g. "<fqdn>:host-gateway"). A missing A record affects every IPv4 consumer of that name, not just this UI — fix it in DNS rather than only working around it per container.

After repointing the UI, make sure CORS_ALLOWED_ORIGINS on the backend lists the origin the browser uses — including the https:// UI origin, not only http:// (see Invalid CORS request below). The internal service name (ft-configs-service) is never a browser origin and must not be added to CORS.

Invalid CORS request

Symptoms:

  • API requests are blocked by the browser with Invalid CORS request, or preflight OPTIONS calls fail, even though the backend is reachable.

Cause:

  • CORS_ALLOWED_ORIGINS does not contain the exact origin (scheme + host + port) shown in the browser address bar. Origins are scheme-sensitive: http:// and https:// are distinct origins.

  • The UI is served on both HTTP and HTTPS (default ports 3001 for HTTP and 3443 for HTTPS), but only the http:// origin was allow-listed, so HTTPS page loads are blocked.

Fixes:

  • List every origin the UI is reachable on — both the http:// and https:// variants:

    CORS_ALLOWED_ORIGINS=http://<ui-host>:3001,https://<ui-host>:3443

  • The value must match the browser URL exactly; do not use the internal Docker service name here.

Authentication issues

Symptoms:

  • 401 responses; UI cannot stay logged in.

Checks:

  • Confirm JWT_SECRET is Base64-encoded (decoded via Decoders.BASE64 in JwtService).

  • Confirm cookies are sent (consider cookie.secure and HTTPS termination).

  • Confirm CSRF token is fetched from GET /configs-service/csrf and sent via header expected by Spring Security.

Mail mode / deletion email

Symptoms:

  • Account deletion request fails with error code USER_DELETION_EMAIL_REQUIRED ("User email is required to request account deletion").

  • No emails are being sent (onboarding, deletion OTP), or you expected offline behavior but the service still requires email.

Causes:

  • The service is in online mode but the initiating admin has no email address (the deletion OTP is sent to that admin).

  • You expected offline mode, but MAIL_MODE/SMTP config resolves to online. A common pitfall: under MAIL_MODE=AUTO, a profile that referenced ${MAIL_PASSWORD} without an empty default left the literal placeholder ${MAIL_PASSWORD} (non-blank), so AUTO wrongly inferred online. All three profiles now use empty defaults (${MAIL_USERNAME:} / ${MAIL_PASSWORD:}).

A configured-but-unreachable SMTP server under MAIL_MODE=AUTO no longer fails the deletion: it degrades to the no-OTP path (request returns otpRequired=false, tokenless confirm deletes the user).

Fixes:

  • For air-gapped installs, set MAIL_MODE=OFFLINE. Temporary passwords are then returned in the create-user/reset-password API responses (passwordDelivery=RETURNED) and deletion OTP is disabled (the request returns otpRequired=false and the account is deleted in the confirm step, called without an OTP).

  • For email delivery, give the initiating admin an email address and configure real, reachable SMTP credentials (MAIL_HOST/MAIL_USERNAME/MAIL_PASSWORD) under MAIL_MODE=AUTO.

  • Check the startup log line Mail delivery: mode=…​, smtpConfigured=…​, effectiveOffline=…​ to confirm the resolved mode.

See Configuration — Delivery modes for details.

Cache correctness (known issues)

  • tabs Hazelcast map is not refreshed after /tabs/** writes/imports in current code (consumer may observe stale data).

  • Many cache refreshes occur inside write transactions; if rollbacks occur, Hazelcast may temporarily contain uncommitted snapshots.