Deployment

The chart’s default resources.limits.memory is 1Gi, and the entrypoint reads it through -XX:MaxRAMPercentage=65 to size the heap: roughly a 665 MiB max heap, leaving about 360 MiB for everything outside it. At the more common 75%, that remainder was only about 256 MiB, and what has to fit in it here is not small — Spring Boot 3.5 metaspace plus code cache alone is commonly 150-220 MiB, and on top of that sit three Hikari pools, a lazily created ClickHouse pool, a Hazelcast client with its own IO threads and buffers, virtual-thread carrier stacks, and Liquibase plus Flyway at boot. An OOMKill is the one kill path the liveness/readiness design on this page cannot see at all — it never goes through a probe, so the DEGRADED / liveness split (Probes, and Why Liveness and Readiness Are Split) cannot soften it, and a pod that OOMKills mid-migration looks nothing like the graceful-shutdown or ClickHouse-outage cases this page otherwise accounts for. The cost of the extra headroom is about 100 MiB of heap a 10 000-entry result buffer does not need.

To override the percentage, use extraEnv with _JAVA_OPTIONS, never JAVA_TOOL_OPTIONS. The JVM reads JAVA_TOOL_OPTIONS before the command line, so a MaxRAMPercentage set there loses to the flag the entrypoint already sets; _JAVA_OPTIONS is read after the command line and wins. JAVA_TOOL_OPTIONS remains the right place for flags the entrypoint does not set at all (-XX:+HeapDumpOnOutOfMemoryError, agents). Full reasoning and the values-file syntax: helm/ft-fcc-service/README.md, "Resources, replicas, shutdown".

With a read-only root filesystem the JVM still needs a writable temp directory (hsperfdata, NIO temp files), and Spring spools multipart parts to the Java temp directory — PUT /FCC/item/import, PUT /FCC/devices/import and POST /FCC/devices/resolve (and POST /FCC/devices/count) all take a file upload, each bounded at FCC_SERIAL_FILE_MAX_SIZE (default 10 MB). tmpDir.sizeLimit (default 128Mi) is set comfortably above that ceiling to absorb concurrent uploads plus whatever else the JVM parks there; it counts against the node’s ephemeral storage, which is what makes it worth capping at all rather than leaving unbounded. tmpDir.medium can be set to Memory for a tmpfs, at the cost of counting against the pod’s memory limit instead.

The aggregate GET /actuator/health maps the custom status DEGRADED to HTTP 200, on purpose (Health). ClickHouse being unreachable, or reachable with its schema behind pending migrations (SCHEMA_BEHIND), is a state this service is explicitly designed to survive: the migration retries on ft.fcc.clickhouse.migration.retry-interval (default 60s) and heals itself, and CRUD against ftfcc keeps working throughout — only new result-producing work (activations, reports, the legacy import) is refused with 503/9503.

A liveness probe pointed at that aggregate endpoint would restart the pod precisely while it is waiting for ClickHouse to come back — and the retry that would have healed it never gets to finish, because the JVM does not live long enough between restarts to let it. A single-instance ClickHouse outage would turn into a cluster-wide CrashLoopBackOff that reads, from the outside, exactly like a bug in this service.

A degraded pod therefore leaves the Service endpoints — so the load balancer stops sending it new work that would only fail against a store it cannot reach — but is not restarted, so the migration retry that would heal it is never interrupted.

The default is readiness.includeClickHouse: false (rendering FCC_READINESS_INCLUDE=readinessState,db), and this section is why. With true, ClickHouse being shared and cluster-wide means a ClickHouse outage takes every replica un-ready at the same instant. The Service then has no endpoints at all, and ui-backend gets connection failures — for plain campaign CRUD too, not only for report and activation traffic — even though decision Q3' promises exactly that CRUD keeps working while the service is DEGRADED.

The default therefore keeps that promise: the pods stay in rotation, CRUD keeps working, and report/activation traffic answers an honest 503/9503 from a pod that is reachable rather than not being reachable at all.

The two-line decision rule:

readiness.include overrides the rendered list outright when neither boolean expresses the combination you need.

startupProbe (initialDelaySeconds: 10, periodSeconds: 10, failureThreshold: 90) budgets 910 seconds (10 + 90 x 10) of boot time. Verified against helm/ft-fcc-service/values.yaml:

910s covers that with roughly 45% headroom. The item that actually forces the number is the 300s Liquibase lock wait: with replicaCount > 1, every replica but one legitimately sits idle behind the one running the migration, and a short startup budget would kill and restart exactly the replica that is behaving correctly — the pod doing nothing wrong except waiting its legitimate turn. This is the same trap DEGRADED sets one layer up (The Trap), one layer down in the boot sequence rather than in steady-state operation.

Being generous here is precisely what lets liveness stay tight (failureThreshold: 3, periodSeconds: 10, timeoutSeconds: 3 — about 30s of a non-responding JVM before a restart): liveness does not run at all until the startup probe has succeeded once, so a startup budget sized against the worst case never costs steady-state responsiveness anything.

Kubernetes runs no probe of any kind against the application container while the wait-for-hazelcast init container is running (The Hazelcast Init Container, and the Two ACS Failures It Cannot Fix) — init containers run to completion before the kubelet starts the main container, and a container that has not been started is never probed. So the init wait can never cause the application container to be restarted; the only bound on that phase is the init container’s own timeoutSeconds.

The startup budget above (910s) governs when the kubelet gives up on a single pod. It says nothing about when the Deployment controller gives up on the rollout as a whole, and that is a second clock, progressDeadlineSeconds, whose Kubernetes default — 600 seconds — sits below the 910s this page spends two tables justifying. On the exact scenario the 910s budget exists for — replicaCount > 1 against a cold ftfcc, every replica but one legitimately queued behind the 300s Liquibase changelog-lock wait — the pods are not killed by anything, but the Deployment reports ProgressDeadlineExceeded at 600s regardless, and kubectl rollout status (which NOTES.txt tells an operator to run) exits non-zero. A pipeline running helm upgrade --wait --atomic reads that non-zero exit as a failed rollout and rolls back in the middle of a Liquibase migration — on precisely the replica set that was behaving correctly. The chart now sets progressDeadlineSeconds: 1200, above the startup budget, for the same reason the startup budget itself is generous: a pod correctly waiting its turn must never look, to any layer of tooling, like a pod that failed.

helm upgrade --wait (or --atomic) needs a matching --timeout of at least 20m. Helm’s own default --timeout is 5 minutes, independent of both Kubernetes clocks above, and expires long before either one — an --atomic rollback triggered by that timeout interrupts a migration exactly as the ProgressDeadlineExceeded case above did, for a reason that has nothing to do with either progressDeadlineSeconds or the startup probe.

terminationGracePeriodSeconds (default 45) delays the kill — the kubelet sends SIGTERM at t=0 regardless and only sends SIGKILL at t=grace. It does not delay SIGTERM itself, and a comment that used to stand on this chart said it did. Because SIGTERM and endpoint removal are dispatched by the control plane at the same instant, they race each other across every kube-proxy in the cluster: without something to close that race, Spring can begin its graceful shutdown — server.shutdown: graceful, spring.lifecycle.timeout-per-shutdown-phase: 20s — while some nodes are still routing live traffic to a pod that is no longer accepting it, and ui-backend sees connection-refused on every rollout. maxUnavailable: 0 does not help here: the pod being torn down is the old one, which is already in the `Service’s endpoints by definition.

The chart now runs a preStop hook (preStopSleepSeconds, default 5, exec + /bin/sleep) before SIGTERM is sent — the container is not signalled until the hook returns — which is what actually closes the gap: the sleep is the propagation delay, not additional shutdown time, but it does count against terminationGracePeriodSeconds, which is why raising one number usually means raising the other. The default’s arithmetic is 5 (preStop) + 20 (the graceful-shutdown phase timeout) + slack for the Hazelcast take() consumers to unwind, comfortably inside the default 45. A large export in flight is still cut off at the deadline regardless — that remains expected, and the operator reruns it.

With the ftfcc database unreachable, DataSourceHealthIndicator blocks inside getConnection() for the whole Hikari connection-timeout before it can answer — about 45 seconds. readinessProbe.timeoutSeconds is set to 5, far below that, and Kubernetes reads a probe timeout as a probe failure, i.e. "not ready" — the same verdict the 503 would eventually have carried, reached about 40 seconds sooner.

Raising probes.readiness.timeoutSeconds to 45 "to let the health check finish" is the wrong fix: the only effect would be that a pod whose database is genuinely dead keeps receiving live traffic it cannot serve for three quarters of a minute per probe cycle, before anything notices and routes around it.