Troubleshooting Guide

This guide provides solutions to common issues encountered in the FT Services platform, organized by category.

Quick Diagnosis

Start with these commands to quickly identify issues:

# Check pod status
kubectl get pods -n <namespace>

# View recent events
kubectl get events -n <namespace> --sort-by='.lastTimestamp' | tail -20

# Check logs
kubectl logs -n <namespace> <pod-name> --tail=100

Pod Issues

Pod Stuck in Pending

Symptoms:

  • Pod status shows Pending

  • Pod never starts running

Common Causes:

  1. Insufficient resources

  2. Node selector not matching

  3. PVC not bound

  4. Image pull errors

Diagnosis:

kubectl describe pod <pod-name> -n <namespace>

Look for messages like:

  • 0/3 nodes are available: insufficient cpu/memory

  • FailedScheduling

  • Unbound PersistentVolumeClaims

Solutions:

Cause Solution

Insufficient CPU/Memory

 
# Check node resources
kubectl describe nodes

# Reduce resource requests temporarily
kubectl edit deployment <name> -n <namespace>

# Or scale down other services
kubectl scale deployment <other-service> --replicas=0 -n <namespace>

PVC not bound

 
# Check PVC status
kubectl get pvc -n <namespace>

# Check storage class
kubectl get storageclass

# If PVC is pending, check provisioner logs
kubectl logs -n kube-system <provisioner-pod>

Node selector mismatch

 
# Check node labels
kubectl get nodes --show-labels

# Remove or update node selector in deployment
kubectl edit deployment <name> -n <namespace>

Pod CrashLoopBackOff

Symptoms:

  • Pod status shows CrashLoopBackOff

  • Pod restarts repeatedly

Diagnosis:

# View current logs
kubectl logs <pod-name> -n <namespace>

# View previous crash logs
kubectl logs <pod-name> -n <namespace> --previous

# Check resource limits
kubectl describe pod <pod-name> -n <namespace> | grep -A 5 Limits

Common Issues:

Database Connection Failures

# Check database pod status
kubectl get pods -n <namespace> -l component=database

# Test database connectivity
kubectl run -it --rm debug --image=busybox --restart=Never -n <namespace> \
  -- nc -zv mysql 3306

# Check database logs
kubectl logs -n <namespace> mysql-0

Solution:

# Ensure initContainers wait for database
initContainers:
- name: wait-for-db
  image: busybox:1.35
  command:
  - sh
  - -c
  - |
    until nc -z mysql 3306; do
      echo "Waiting for MySQL..."
      sleep 2
    done

Out of Memory (OOM)

Symptoms:

  • Pod logs show OOMKilled

  • Java heap space errors

Diagnosis:

# Check memory usage
kubectl top pod <pod-name> -n <namespace>

# Check OOM events
kubectl get events -n <namespace> | grep OOM

Solutions:

# Increase memory limits
resources:
  limits:
    memory: "4Gi"  # Increase from 2Gi

# For Java apps, tune heap size
env:
- name: JAVA_OPTS
  value: "-Xms512m -Xmx2g"  # Adjust as needed

Application Startup Failures

Symptoms:

  • Pod fails health checks immediately

  • Application exits with error code

Diagnosis:

# Check application logs
kubectl logs <pod-name> -n <namespace> --tail=200

# Check configuration
kubectl describe configmap <name> -n <namespace>
kubectl describe secret <name> -n <namespace>

Common Issues:

Error Solution

Configuration file not found

Verify ConfigMap is mounted:

kubectl describe pod <pod> -n <ns> | grep -A 10 Mounts

Connection refused to database

Check database host name in config:

# Should be service name, not localhost
DB_HOST: mysql  # not localhost

Secret key not found

Verify secret exists and has correct keys:

kubectl get secret <name> -n <ns> -o yaml

Pod Not Ready

Symptoms:

  • Pod is Running but not Ready

  • Health check failures

Diagnosis:

# Check readiness probe
kubectl describe pod <pod-name> -n <namespace> | grep -A 10 Readiness

# Test health endpoint manually
kubectl exec -it <pod-name> -n <namespace> -- curl localhost:8080/health

Solutions:

# Increase initialDelaySeconds
readinessProbe:
  httpGet:
    path: /health
    port: 8080
  initialDelaySeconds: 60  # Increase from 30
  periodSeconds: 10
  failureThreshold: 5  # Allow more failures

Don’t set failureThreshold too high in production - it delays detection of real failures.

FT FCC Service: Liveness Probe Must Not Use Aggregate /actuator/health

Symptoms:

  • FT FCC Service pod restarts repeatedly while ClickHouse is unreachable or its schema is behind

  • FT FCC Service pod is killed mid-rollout during an otherwise normal deployment

Cause:

FT FCC Service’s aggregate /actuator/health answers DEGRADED as HTTP 200 when ClickHouse is unreachable or its schema is behind, and in that state the service is deliberately still running: the ClickHouse migration retries on a 60s timer and heals itself, campaign CRUD keeps working, and only new activations are refused. The aggregate also answers 503 for up to 20s of every rolling deploy, because enabling the Boot probes registers readinessState as a contributor to the aggregate, and it reports REFUSING_TRAFFIC during graceful shutdown.

A livenessProbe or startupProbe pointed at the aggregate would kill the pod exactly while it is recovering, or during ordinary shutdown. The chart’s startupProbe and livenessProbe target /actuator/health/liveness; only readinessProbe targets /actuator/health/readiness:

startupProbe:
  httpGet:
    path: /actuator/health/liveness
    port: 8080
livenessProbe:
  httpGet:
    path: /actuator/health/liveness
    port: 8080
readinessProbe:
  httpGet:
    path: /actuator/health/readiness
    port: 8080

Diagnosis:

# Confirm which path the probes actually use
kubectl get deployment ft-fcc-service -n <namespace> \
  -o jsonpath='{.spec.template.spec.containers[0].livenessProbe.httpGet.path}{"\n"}{.spec.template.spec.containers[0].readinessProbe.httpGet.path}'

# Compare the aggregate against the split endpoints directly
kubectl exec -it <pod> -n <namespace> -- curl -s localhost:8080/actuator/health
kubectl exec -it <pod> -n <namespace> -- curl -s localhost:8080/actuator/health/liveness
kubectl exec -it <pod> -n <namespace> -- curl -s localhost:8080/actuator/health/readiness

Solution:

If a Deployment override has pointed either probe at the bare /actuator/health, repoint it at the split endpoint above. Do not "fix" this by lowering failureThreshold or raising delays — the aggregate genuinely reports states (DEGRADED, REFUSING_TRAFFIC) that are correct for the service but wrong for a liveness signal.

ClickHouse is also deliberately kept out of the readiness group (FCC_READINESS_INCLUDE defaults to readinessState,db). ClickHouse is one shared cluster every replica reaches over the network, so the only realistic outage is cluster-wide — including it in readiness would take every replica out of rotation at the same moment, leaving the Service with no endpoints for plain CRUD that is promised to keep working while degraded. Set ftFccService.readinessIncludeClickHouse: true only for topologies where ClickHouse can fail per pod.

ArgoCD Issues

Application Out of Sync

Symptoms:

  • ArgoCD shows application as OutOfSync

  • Changes in Git not reflected in cluster

Diagnosis:

# Check application status
kubectl get application <app-name> -n argocd -o yaml

# View diff
argocd app diff <app-name>

# Check sync status
argocd app get <app-name>

Solutions:

Cause Solution

Manual sync required

 
# Sync manually
argocd app sync <app-name>

# Or via kubectl
kubectl patch application <app-name> -n argocd \
  --type merge \
  -p '{"operation":{"sync":{}}}'

Helm rendering errors

 
# Test Helm template locally
helm template <release> <chart> -f values.yaml

# Check for syntax errors
helm lint <chart>

Resource conflicts

 
# Delete conflicting resource
kubectl delete <resource> <name> -n <namespace>

# Then sync again
argocd app sync <app-name>

Application Stuck in Progressing

Symptoms:

  • ArgoCD shows Progressing for extended period

  • Sync operation doesn’t complete

Diagnosis:

# Check sync operation
argocd app get <app-name>

# View operation details
kubectl describe application <app-name> -n argocd

# Check for hooks
kubectl get pods -n <namespace> -l app.kubernetes.io/instance=<app-name>

Common Causes:

  1. Helm hooks not completing

  2. Resource quotas exceeded

  3. Webhook timeout

  4. Image pull timeout

Solutions:

# Delete stuck hooks
kubectl delete pod -n <namespace> -l hook=pre-install

# Check resource quotas
kubectl describe quota -n <namespace>

# Terminate stuck sync operation
argocd app terminate-op <app-name>

# Retry sync
argocd app sync <app-name>

Application Health Degraded

Symptoms:

  • ArgoCD shows health as Degraded

  • Some resources unhealthy

Diagnosis:

# View health details
argocd app get <app-name> --hard-refresh

# Check individual resources
kubectl get all -n <namespace>

Solutions:

See Pod Issues for specific pod troubleshooting.

Networking Issues

Service Not Accessible

Symptoms:

  • Cannot reach service from outside cluster

  • 502/503 errors from ingress

Diagnosis:

# Check service
kubectl get svc <service-name> -n <namespace>

# Check endpoints
kubectl get endpoints <service-name> -n <namespace>

# Check ingress
kubectl get ingress -n <namespace>
kubectl describe ingress <name> -n <namespace>

Solutions:

Issue Solution

Service has no endpoints

 
# Check if pods are ready
kubectl get pods -n <namespace> -l app=<app-name>

# Fix pod issues (see Pod Issues section)

Ingress not configured

 
# Check IngressRoute (Traefik)
kubectl get ingressroute -n <namespace>

# Verify host matches
kubectl get ingressroute <name> -n <ns> -o yaml | grep host

Wrong service port

 
# Verify port in service matches pod
apiVersion: v1
kind: Service
spec:
  ports:
  - port: 8080
    targetPort: 8080  # Must match container port

Pod Cannot Reach Other Services

Symptoms:

  • Connection timeout between services

  • Network policy blocking traffic

Diagnosis:

# Test connectivity from pod
kubectl exec -it <pod-name> -n <namespace> -- \
  curl -v http://<service-name>:8080

# Check network policies
kubectl get networkpolicy -n <namespace>

# Describe policy
kubectl describe networkpolicy <policy-name> -n <namespace>

Solutions:

# Temporarily remove network policy to test
kubectl delete networkpolicy <policy-name> -n <namespace>

# If that fixes it, update policy to allow traffic
kubectl edit networkpolicy <policy-name> -n <namespace>

Always restore network policies after testing. Don’t leave cluster without network policies in production.

DNS Resolution Failures

Symptoms:

  • Name or service not known errors

  • Cannot resolve service names

Diagnosis:

# Test DNS from pod
kubectl exec -it <pod-name> -n <namespace> -- nslookup kubernetes.default

# Check CoreDNS pods
kubectl get pods -n kube-system -l k8s-app=kube-dns

# Check CoreDNS logs
kubectl logs -n kube-system -l k8s-app=kube-dns

Solutions:

# Restart CoreDNS
kubectl rollout restart deployment/coredns -n kube-system

# Check service name format
# Should be: <service-name>.<namespace>.svc.cluster.local
# Or just: <service-name> (if same namespace)

# Verify /etc/resolv.conf in pod
kubectl exec <pod> -n <ns> -- cat /etc/resolv.conf

Database Issues

Database Connection Pool Exhausted

Symptoms:

  • Cannot get connection from pool errors

  • Pool is exhausted messages

Diagnosis:

# Check application logs
kubectl logs <pod-name> -n <namespace> | grep -i "pool\|connection"

# Check database connections
kubectl exec -it mysql-0 -n <namespace> -- \
  mysql -uroot -p -e "SHOW PROCESSLIST;"

Solutions:

# Increase connection pool size
spring:
  datasource:
    hikari:
      maximum-pool-size: 20  # Increase from 10
      connection-timeout: 30000
# Or scale up application pods
kubectl scale deployment <app-name> -n <namespace> --replicas=3

Database Performance Issues

Symptoms:

  • Slow query responses

  • High database CPU/memory usage

Diagnosis:

# Check database resource usage
kubectl top pod mysql-0 -n <namespace>

# Check slow query log (MySQL)
kubectl exec mysql-0 -n <namespace> -- \
  mysql -uroot -p -e "SELECT * FROM mysql.slow_log LIMIT 10;"

# Check running queries
kubectl exec mysql-0 -n <namespace> -- \
  mysql -uroot -p -e "SHOW FULL PROCESSLIST;"

Solutions:

# Identify slow queries and add indexes
kubectl exec -it mysql-0 -n <namespace> -- mysql -uroot -p ftacs

# Inside MySQL
SHOW INDEX FROM <table>;
CREATE INDEX idx_name ON table_name(column);

# Increase database resources
kubectl edit statefulset mysql -n <namespace>

Database Disk Space Full

Symptoms:

  • No space left on device errors

  • Database won’t start

Diagnosis:

# Check PVC usage
kubectl exec mysql-0 -n <namespace> -- df -h /var/lib/mysql

# Check PVC size
kubectl get pvc -n <namespace>

Solutions:

# Expand PVC (if storage class supports it)
kubectl edit pvc mysql-data-mysql-0 -n <namespace>
# Increase storage size

# Or clean up old data
kubectl exec -it mysql-0 -n <namespace> -- \
  mysql -uroot -p -e "PURGE BINARY LOGS BEFORE NOW() - INTERVAL 7 DAY;"

See Database Migration Runbook for expanding storage.

FT FCC Service: ftfcc Schema Missing After Deploy

Symptoms:

  • FT FCC Service pod fails its readiness probe and never goes Ready

  • Logs show the datasource unable to reach the ftfcc schema, or FtacsSchemaCheckRunner aborting startup and naming a missing table.column in ftacs

Cause:

The MySQL/Oracle schema init SQL for FT FCC Service runs only once, against an empty database volume, same as every other service’s init SQL in this chart:

  • K3s MySQL: helm/ft-services/config-templates/mysql-chain/03-ftfcc.sql (this environment’s copy — values.yaml sets mysql.configTemplateDir: "mysql-chain"; config-templates/mysql-initdb/04-ftfcc.sql is the OpenShift equivalent)

  • Oracle: config-templates/{oracle-chain,oracle-initdb}/ftfcc/create_ftfcc.sql, invoked from createFriendlyDB.sql

An environment whose MySQL/Oracle data volume already holds data — most long-lived dev-mysql/dev-oracle-style environments, or any partial environment pointed at an existing parent database — never re-runs that init SQL.

On the K3s/ArgoCD path this is handled automatically by helm/ft-services/templates/jobs/ft-fcc-db-init.yaml: an idempotent ArgoCD Sync-hook Job (ft-fcc-db-init for MySQL, ft-fcc-db-init-oracle for Oracle) that runs the same statements against an already-initialized database on every sync. So the usual cause of this symptom is that the Job failed, not that it is absent — check it first. The OpenShift product has no equivalent hook, so there the schema does have to be created by hand.

Diagnosis:

# Does the schema exist at all?
kubectl exec -it mysql-0 -n <namespace> -- \
  mysql -uroot -p -e "SHOW DATABASES LIKE 'ftfcc';"

# Readiness failure detail
kubectl logs <ft-fcc-service-pod> -n <namespace> --previous | grep -i "ftfcc\|liquibase\|FtacsSchemaCheckRunner"

# Did the provisioning Job succeed? (deleted on success by the hook-delete-policy,
# so "not found" after a healthy sync is expected)
kubectl get job -n <namespace> ft-fcc-db-init ft-fcc-db-init-oracle
kubectl logs -n <namespace> job/ft-fcc-db-init

Solution:

On K3s, re-sync the ArgoCD Application so the ft-fcc-db-init hook Job runs again, and read its logs if it fails — it is idempotent, so re-running it is always safe. Failing that (or on OpenShift, which has no hook), create the ftfcc schema by hand against the existing database by running the statements from the init SQL file above, or recreate the environment so the init SQL runs against a fresh, empty volume — only a brand-new database volume picks up init SQL, and editing the SQL file has no effect on an existing one. If instead the failure is FtacsSchemaCheckRunner complaining that ACS hasn’t migrated its ug_fcc* tables yet, either wait for FTACS to finish its own Liquibase run, or temporarily set ftFccService.ftacsSchemaCheck: false to let the pod start anyway.

Performance Issues

High CPU Usage

Diagnosis:

# Check CPU usage
kubectl top pods -n <namespace>
kubectl top nodes

# Identify CPU-intensive processes
kubectl exec <pod> -n <ns> -- top

Solutions:

Cause Solution

Insufficient CPU limits

 
resources:
  limits:
    cpu: "2000m"  # Increase

Inefficient code

Review application logs and profiling data

Too many pods on one node

 
# Spread pods across nodes
kubectl cordon <node-name>
kubectl drain <node-name> --ignore-daemonsets

High Memory Usage

Diagnosis:

# Check memory usage
kubectl top pods -n <namespace>

# Check for memory leaks
kubectl exec <pod> -n <ns> -- free -h

Solutions:

# Increase memory limits
resources:
  limits:
    memory: "4Gi"

# For Java apps, enable heap dumps on OOM
env:
- name: JAVA_OPTS
  value: "-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp"

Image Pull Errors

ImagePullBackOff

Symptoms:

  • Pod status shows ImagePullBackOff or ErrImagePull

Diagnosis:

# Check pod events
kubectl describe pod <pod-name> -n <namespace> | grep -A 10 Events

# Common errors:
# - "unauthorized: authentication required"
# - "manifest unknown"
# - "pull access denied"

Solutions:

Error Solution

Authentication required

 
# Check if secret exists
kubectl get secret harbor-secret -n <namespace>

# Recreate secret
kubectl create secret docker-registry harbor-secret \
  --docker-server=hub.friendly-tech.com \
  --docker-username=<username> \
  --docker-password=<password> \
  -n <namespace>

# Update deployment to use secret
kubectl edit deployment <name> -n <namespace>

Image not found

 
# Verify image exists in Harbor
# Check image name and tag
kubectl get deployment <name> -n <ns> -o yaml | grep image

# Common issues:
# - Wrong registry URL
# - Typo in image name
# - Tag doesn't exist

Rate limit exceeded

Wait or use image pull secrets for authenticated pulls

Sealed Secrets Issues

Secret Not Decrypting

Symptoms:

  • SealedSecret exists but Secret not created

  • Pods cannot read secret values

Diagnosis:

# Check SealedSecret
kubectl get sealedsecret -n <namespace>

# Check if Secret was created
kubectl get secret <name> -n <namespace>

# Check sealed-secrets controller logs
kubectl logs -n sealed-secrets -l name=sealed-secrets-controller

Solutions:

# Verify sealed-secrets controller is running
kubectl get pods -n sealed-secrets

# If not running, reinstall
kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.24.0/controller.yaml

# Check if SealedSecret is properly scoped
kubectl get sealedsecret <name> -n <ns> -o yaml | grep scope

Environment Manager Issues

Environment Creation Fails

Symptoms:

  • Error message on form submit

  • Environment not appearing in list

Diagnosis:

# Check backend logs
docker-compose logs backend

# Check if files were created
ls -la helm/ft-services/values-dynamic-*.yaml
ls -la argocd/applications/dynamic/*.yaml

# Check git status
cd /path/to/ft-deployments && git status

Solutions:

Error Solution

"Environment already exists"

Choose a different name or delete existing environment

"Failed to trigger build"

Check GitHub Actions logs, verify GITHUB_TOKEN is valid

"Image not found in Harbor"

Wait for build to complete, check Harbor UI

Git commit fails

Check git credentials, verify repository write access

Duplicate Name Error

Symptoms:

  • "Environment already exists" error for seemingly new name

Diagnosis:

# Check for existing files
find helm/ft-services -name "values-dynamic-*" | grep <name>
find argocd/applications/dynamic -name "*.yaml" | grep <name>

# Check ArgoCD
kubectl get application -n argocd | grep <name>

Solutions:

# Clean up orphaned resources
argocd app delete <name> --cascade -y
rm helm/ft-services/values-dynamic-<name>.yaml
rm argocd/applications/dynamic/<name>.yaml
git add -A && git commit -m "Clean up orphaned environment"
git push

Build Takes Too Long

Symptoms:

  • UI shows "Building…​" for extended period

  • No progress updates

Diagnosis:

# Check GitHub Actions
# Navigate to: https://github.com/{owner}/{repo}/actions

# Check backend logs for build status
docker-compose logs backend | grep -i build

Solutions:

Cause Solution

Branch not found

Verify branch name exists in Git repository

Build error

Check GitHub Actions workflow logs for errors

Rate limit

Wait and retry, check GitHub API limits

See Environment Manager Documentation for more details.

Feature Branch Issues

Feature Environment Not Creating

Symptoms:

  • create-feature-env.sh script fails

  • ArgoCD application not appearing

Diagnosis:

# Check if application was created
kubectl get application -n argocd | grep feature

# Check ArgoCD controller logs
kubectl logs -n argocd -l app.kubernetes.io/name=argocd-application-controller

Solutions:

# Verify branch name is valid
# Must be lowercase, alphanumeric, and hyphens only

# Check quota
kubectl get resourcequota -n <feature-namespace>

# Use Environment Manager Web UI instead
# Navigate to: http://localhost:8080

Feature Branch Not Deploying

Symptoms:

  • Images tagged with branch name don’t exist

  • Pods use wrong image tag

Diagnosis:

# Check if CI/CD built the image
# In Harbor registry, verify image exists:
# hub.friendly-tech.com/ftacs/ftacs:<branch-name>

# Check deployment image
kubectl get deployment ftacs -n feature-<branch> -o yaml | grep image:

Solutions:

  1. Use Environment Manager to trigger builds with "Branch" source option

  2. Verify CI/CD pipeline ran successfully

  3. Check image naming matches branch name

Useful Debug Commands

Pod Inspection

# Execute shell in pod
kubectl exec -it <pod-name> -n <namespace> -- /bin/bash

# Copy files from pod
kubectl cp <namespace>/<pod-name>:/path/to/file ./local-file

# Port forward to pod
kubectl port-forward -n <namespace> <pod-name> 8080:8080

Log Inspection

# Follow logs in real-time
kubectl logs -f <pod-name> -n <namespace>

# Logs from all pods in deployment
kubectl logs -n <namespace> -l app=<app-name> --all-containers=true

# Logs from specific time range
kubectl logs <pod> -n <ns> --since=10m

# Save logs to file
kubectl logs <pod> -n <ns> > logs.txt

Resource Inspection

# Get all resources in namespace
kubectl get all -n <namespace>

# Describe all pods
kubectl describe pods -n <namespace>

# Get events sorted by time
kubectl get events -n <namespace> --sort-by='.lastTimestamp'

# Check resource usage
kubectl top nodes
kubectl top pods -n <namespace>

Getting Help

If you can’t resolve an issue:

  1. Check this troubleshooting guide

  2. Review System Architecture

  3. Consult relevant Runbook

  4. Gather diagnostics and contact DevOps team

Diagnostic Checklist:

  • kubectl get pods output

  • kubectl describe pod output

  • kubectl logs output

  • kubectl get events output

  • kubectl top output (if performance issue)

  • ArgoCD application status (if deployment issue)

Related Documentation

Last updated: 2026-10-14 19:00:11 +0200