Troubleshooting Guide

This guide provides solutions to common issues encountered in the FT Services platform, organized by category.

Quick Diagnosis

Start with these commands to quickly identify issues:

# Check pod status
kubectl get pods -n <namespace>

# View recent events
kubectl get events -n <namespace> --sort-by='.lastTimestamp' | tail -20

# Check logs
kubectl logs -n <namespace> <pod-name> --tail=100

Pod Issues

Pod Stuck in Pending

Symptoms:

  • Pod status shows Pending

  • Pod never starts running

Common Causes:

  1. Insufficient resources

  2. Node selector not matching

  3. PVC not bound

  4. Image pull errors

Diagnosis:

kubectl describe pod <pod-name> -n <namespace>

Look for messages like:

  • 0/3 nodes are available: insufficient cpu/memory

  • FailedScheduling

  • Unbound PersistentVolumeClaims

Solutions:

Cause Solution

Insufficient CPU/Memory

 
# Check node resources
kubectl describe nodes

# Reduce resource requests temporarily
kubectl edit deployment <name> -n <namespace>

# Or scale down other services
kubectl scale deployment <other-service> --replicas=0 -n <namespace>

PVC not bound

 
# Check PVC status
kubectl get pvc -n <namespace>

# Check storage class
kubectl get storageclass

# If PVC is pending, check provisioner logs
kubectl logs -n kube-system <provisioner-pod>

Node selector mismatch

 
# Check node labels
kubectl get nodes --show-labels

# Remove or update node selector in deployment
kubectl edit deployment <name> -n <namespace>

Pod CrashLoopBackOff

Symptoms:

  • Pod status shows CrashLoopBackOff

  • Pod restarts repeatedly

Diagnosis:

# View current logs
kubectl logs <pod-name> -n <namespace>

# View previous crash logs
kubectl logs <pod-name> -n <namespace> --previous

# Check resource limits
kubectl describe pod <pod-name> -n <namespace> | grep -A 5 Limits

Common Issues:

Database Connection Failures

# Check database pod status
kubectl get pods -n <namespace> -l component=database

# Test database connectivity
kubectl run -it --rm debug --image=busybox --restart=Never -n <namespace> \
  -- nc -zv mysql 3306

# Check database logs
kubectl logs -n <namespace> mysql-0

Solution:

# Ensure initContainers wait for database
initContainers:
- name: wait-for-db
  image: busybox:1.35
  command:
  - sh
  - -c
  - |
    until nc -z mysql 3306; do
      echo "Waiting for MySQL..."
      sleep 2
    done

Out of Memory (OOM)

Symptoms:

  • Pod logs show OOMKilled

  • Java heap space errors

Diagnosis:

# Check memory usage
kubectl top pod <pod-name> -n <namespace>

# Check OOM events
kubectl get events -n <namespace> | grep OOM

Solutions:

# Increase memory limits
resources:
  limits:
    memory: "4Gi"  # Increase from 2Gi

# For Java apps, tune heap size
env:
- name: JAVA_OPTS
  value: "-Xms512m -Xmx2g"  # Adjust as needed

Application Startup Failures

Symptoms:

  • Pod fails health checks immediately

  • Application exits with error code

Diagnosis:

# Check application logs
kubectl logs <pod-name> -n <namespace> --tail=200

# Check configuration
kubectl describe configmap <name> -n <namespace>
kubectl describe secret <name> -n <namespace>

Common Issues:

Error Solution

Configuration file not found

Verify ConfigMap is mounted:

kubectl describe pod <pod> -n <ns> | grep -A 10 Mounts

Connection refused to database

Check database host name in config:

# Should be service name, not localhost
DB_HOST: mysql  # not localhost

Secret key not found

Verify secret exists and has correct keys:

kubectl get secret <name> -n <ns> -o yaml

Pod Not Ready

Symptoms:

  • Pod is Running but not Ready

  • Health check failures

Diagnosis:

# Check readiness probe
kubectl describe pod <pod-name> -n <namespace> | grep -A 10 Readiness

# Test health endpoint manually
kubectl exec -it <pod-name> -n <namespace> -- curl localhost:8080/health

Solutions:

# Increase initialDelaySeconds
readinessProbe:
  httpGet:
    path: /health
    port: 8080
  initialDelaySeconds: 60  # Increase from 30
  periodSeconds: 10
  failureThreshold: 5  # Allow more failures

Don’t set failureThreshold too high in production - it delays detection of real failures.

ArgoCD Issues

Application Out of Sync

Symptoms:

  • ArgoCD shows application as OutOfSync

  • Changes in Git not reflected in cluster

Diagnosis:

# Check application status
kubectl get application <app-name> -n argocd -o yaml

# View diff
argocd app diff <app-name>

# Check sync status
argocd app get <app-name>

Solutions:

Cause Solution

Manual sync required

 
# Sync manually
argocd app sync <app-name>

# Or via kubectl
kubectl patch application <app-name> -n argocd \
  --type merge \
  -p '{"operation":{"sync":{}}}'

Helm rendering errors

 
# Test Helm template locally
helm template <release> <chart> -f values.yaml

# Check for syntax errors
helm lint <chart>

Resource conflicts

 
# Delete conflicting resource
kubectl delete <resource> <name> -n <namespace>

# Then sync again
argocd app sync <app-name>

Application Stuck in Progressing

Symptoms:

  • ArgoCD shows Progressing for extended period

  • Sync operation doesn’t complete

Diagnosis:

# Check sync operation
argocd app get <app-name>

# View operation details
kubectl describe application <app-name> -n argocd

# Check for hooks
kubectl get pods -n <namespace> -l app.kubernetes.io/instance=<app-name>

Common Causes:

  1. Helm hooks not completing

  2. Resource quotas exceeded

  3. Webhook timeout

  4. Image pull timeout

Solutions:

# Delete stuck hooks
kubectl delete pod -n <namespace> -l hook=pre-install

# Check resource quotas
kubectl describe quota -n <namespace>

# Terminate stuck sync operation
argocd app terminate-op <app-name>

# Retry sync
argocd app sync <app-name>

Application Health Degraded

Symptoms:

  • ArgoCD shows health as Degraded

  • Some resources unhealthy

Diagnosis:

# View health details
argocd app get <app-name> --hard-refresh

# Check individual resources
kubectl get all -n <namespace>

Solutions:

See Pod Issues for specific pod troubleshooting.

Networking Issues

Service Not Accessible

Symptoms:

  • Cannot reach service from outside cluster

  • 502/503 errors from ingress

Diagnosis:

# Check service
kubectl get svc <service-name> -n <namespace>

# Check endpoints
kubectl get endpoints <service-name> -n <namespace>

# Check ingress
kubectl get ingress -n <namespace>
kubectl describe ingress <name> -n <namespace>

Solutions:

Issue Solution

Service has no endpoints

 
# Check if pods are ready
kubectl get pods -n <namespace> -l app=<app-name>

# Fix pod issues (see Pod Issues section)

Ingress not configured

 
# Check IngressRoute (Traefik)
kubectl get ingressroute -n <namespace>

# Verify host matches
kubectl get ingressroute <name> -n <ns> -o yaml | grep host

Wrong service port

 
# Verify port in service matches pod
apiVersion: v1
kind: Service
spec:
  ports:
  - port: 8080
    targetPort: 8080  # Must match container port

Pod Cannot Reach Other Services

Symptoms:

  • Connection timeout between services

  • Network policy blocking traffic

Diagnosis:

# Test connectivity from pod
kubectl exec -it <pod-name> -n <namespace> -- \
  curl -v http://<service-name>:8080

# Check network policies
kubectl get networkpolicy -n <namespace>

# Describe policy
kubectl describe networkpolicy <policy-name> -n <namespace>

Solutions:

# Temporarily remove network policy to test
kubectl delete networkpolicy <policy-name> -n <namespace>

# If that fixes it, update policy to allow traffic
kubectl edit networkpolicy <policy-name> -n <namespace>

Always restore network policies after testing. Don’t leave cluster without network policies in production.

DNS Resolution Failures

Symptoms:

  • Name or service not known errors

  • Cannot resolve service names

Diagnosis:

# Test DNS from pod
kubectl exec -it <pod-name> -n <namespace> -- nslookup kubernetes.default

# Check CoreDNS pods
kubectl get pods -n kube-system -l k8s-app=kube-dns

# Check CoreDNS logs
kubectl logs -n kube-system -l k8s-app=kube-dns

Solutions:

# Restart CoreDNS
kubectl rollout restart deployment/coredns -n kube-system

# Check service name format
# Should be: <service-name>.<namespace>.svc.cluster.local
# Or just: <service-name> (if same namespace)

# Verify /etc/resolv.conf in pod
kubectl exec <pod> -n <ns> -- cat /etc/resolv.conf

Database Issues

Database Connection Pool Exhausted

Symptoms:

  • Cannot get connection from pool errors

  • Pool is exhausted messages

Diagnosis:

# Check application logs
kubectl logs <pod-name> -n <namespace> | grep -i "pool\|connection"

# Check database connections
kubectl exec -it mysql-0 -n <namespace> -- \
  mysql -uroot -p -e "SHOW PROCESSLIST;"

Solutions:

# Increase connection pool size
spring:
  datasource:
    hikari:
      maximum-pool-size: 20  # Increase from 10
      connection-timeout: 30000
# Or scale up application pods
kubectl scale deployment <app-name> -n <namespace> --replicas=3

Database Performance Issues

Symptoms:

  • Slow query responses

  • High database CPU/memory usage

Diagnosis:

# Check database resource usage
kubectl top pod mysql-0 -n <namespace>

# Check slow query log (MySQL)
kubectl exec mysql-0 -n <namespace> -- \
  mysql -uroot -p -e "SELECT * FROM mysql.slow_log LIMIT 10;"

# Check running queries
kubectl exec mysql-0 -n <namespace> -- \
  mysql -uroot -p -e "SHOW FULL PROCESSLIST;"

Solutions:

# Identify slow queries and add indexes
kubectl exec -it mysql-0 -n <namespace> -- mysql -uroot -p ftacs

# Inside MySQL
SHOW INDEX FROM <table>;
CREATE INDEX idx_name ON table_name(column);

# Increase database resources
kubectl edit statefulset mysql -n <namespace>

Database Disk Space Full

Symptoms:

  • No space left on device errors

  • Database won’t start

Diagnosis:

# Check PVC usage
kubectl exec mysql-0 -n <namespace> -- df -h /var/lib/mysql

# Check PVC size
kubectl get pvc -n <namespace>

Solutions:

# Expand PVC (if storage class supports it)
kubectl edit pvc mysql-data-mysql-0 -n <namespace>
# Increase storage size

# Or clean up old data
kubectl exec -it mysql-0 -n <namespace> -- \
  mysql -uroot -p -e "PURGE BINARY LOGS BEFORE NOW() - INTERVAL 7 DAY;"

See Database Migration Runbook for expanding storage.

Performance Issues

High CPU Usage

Diagnosis:

# Check CPU usage
kubectl top pods -n <namespace>
kubectl top nodes

# Identify CPU-intensive processes
kubectl exec <pod> -n <ns> -- top

Solutions:

Cause Solution

Insufficient CPU limits

 
resources:
  limits:
    cpu: "2000m"  # Increase

Inefficient code

Review application logs and profiling data

Too many pods on one node

 
# Spread pods across nodes
kubectl cordon <node-name>
kubectl drain <node-name> --ignore-daemonsets

High Memory Usage

Diagnosis:

# Check memory usage
kubectl top pods -n <namespace>

# Check for memory leaks
kubectl exec <pod> -n <ns> -- free -h

Solutions:

# Increase memory limits
resources:
  limits:
    memory: "4Gi"

# For Java apps, enable heap dumps on OOM
env:
- name: JAVA_OPTS
  value: "-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp"

Image Pull Errors

ImagePullBackOff

Symptoms:

  • Pod status shows ImagePullBackOff or ErrImagePull

Diagnosis:

# Check pod events
kubectl describe pod <pod-name> -n <namespace> | grep -A 10 Events

# Common errors:
# - "unauthorized: authentication required"
# - "manifest unknown"
# - "pull access denied"

Solutions:

Error Solution

Authentication required

 
# Check if secret exists
kubectl get secret harbor-secret -n <namespace>

# Recreate secret
kubectl create secret docker-registry harbor-secret \
  --docker-server=hub.friendly-tech.com \
  --docker-username=<username> \
  --docker-password=<password> \
  -n <namespace>

# Update deployment to use secret
kubectl edit deployment <name> -n <namespace>

Image not found

 
# Verify image exists in Harbor
# Check image name and tag
kubectl get deployment <name> -n <ns> -o yaml | grep image

# Common issues:
# - Wrong registry URL
# - Typo in image name
# - Tag doesn't exist

Rate limit exceeded

Wait or use image pull secrets for authenticated pulls

Sealed Secrets Issues

Secret Not Decrypting

Symptoms:

  • SealedSecret exists but Secret not created

  • Pods cannot read secret values

Diagnosis:

# Check SealedSecret
kubectl get sealedsecret -n <namespace>

# Check if Secret was created
kubectl get secret <name> -n <namespace>

# Check sealed-secrets controller logs
kubectl logs -n sealed-secrets -l name=sealed-secrets-controller

Solutions:

# Verify sealed-secrets controller is running
kubectl get pods -n sealed-secrets

# If not running, reinstall
kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.24.0/controller.yaml

# Check if SealedSecret is properly scoped
kubectl get sealedsecret <name> -n <ns> -o yaml | grep scope

Environment Manager Issues

Environment Creation Fails

Symptoms:

  • Error message on form submit

  • Environment not appearing in list

Diagnosis:

# Check backend logs
docker-compose logs backend

# Check if files were created
ls -la helm/ft-services/values-dynamic-*.yaml
ls -la argocd/applications/dynamic/*.yaml

# Check git status
cd /path/to/ft-deployments && git status

Solutions:

Error Solution

"Environment already exists"

Choose a different name or delete existing environment

"Failed to trigger build"

Check GitHub Actions logs, verify GITHUB_TOKEN is valid

"Image not found in Harbor"

Wait for build to complete, check Harbor UI

Git commit fails

Check git credentials, verify repository write access

Duplicate Name Error

Symptoms:

  • "Environment already exists" error for seemingly new name

Diagnosis:

# Check for existing files
find helm/ft-services -name "values-dynamic-*" | grep <name>
find argocd/applications/dynamic -name "*.yaml" | grep <name>

# Check ArgoCD
kubectl get application -n argocd | grep <name>

Solutions:

# Clean up orphaned resources
argocd app delete <name> --cascade -y
rm helm/ft-services/values-dynamic-<name>.yaml
rm argocd/applications/dynamic/<name>.yaml
git add -A && git commit -m "Clean up orphaned environment"
git push

Build Takes Too Long

Symptoms:

  • UI shows "Building…​" for extended period

  • No progress updates

Diagnosis:

# Check GitHub Actions
# Navigate to: https://github.com/{owner}/{repo}/actions

# Check backend logs for build status
docker-compose logs backend | grep -i build

Solutions:

Cause Solution

Branch not found

Verify branch name exists in Git repository

Build error

Check GitHub Actions workflow logs for errors

Rate limit

Wait and retry, check GitHub API limits

See Environment Manager Documentation for more details.

Feature Branch Issues

Feature Environment Not Creating

Symptoms:

  • create-feature-env.sh script fails

  • ArgoCD application not appearing

Diagnosis:

# Check if application was created
kubectl get application -n argocd | grep feature

# Check ArgoCD controller logs
kubectl logs -n argocd -l app.kubernetes.io/name=argocd-application-controller

Solutions:

# Verify branch name is valid
# Must be lowercase, alphanumeric, and hyphens only

# Check quota
kubectl get resourcequota -n <feature-namespace>

# Use Environment Manager Web UI instead
# Navigate to: http://localhost:8080

Feature Branch Not Deploying

Symptoms:

  • Images tagged with branch name don’t exist

  • Pods use wrong image tag

Diagnosis:

# Check if CI/CD built the image
# In Harbor registry, verify image exists:
# hub.friendly-tech.com/ftacs/ftacs:<branch-name>

# Check deployment image
kubectl get deployment ftacs -n feature-<branch> -o yaml | grep image:

Solutions:

  1. Use Environment Manager to trigger builds with "Branch" source option

  2. Verify CI/CD pipeline ran successfully

  3. Check image naming matches branch name

Useful Debug Commands

Pod Inspection

# Execute shell in pod
kubectl exec -it <pod-name> -n <namespace> -- /bin/bash

# Copy files from pod
kubectl cp <namespace>/<pod-name>:/path/to/file ./local-file

# Port forward to pod
kubectl port-forward -n <namespace> <pod-name> 8080:8080

Log Inspection

# Follow logs in real-time
kubectl logs -f <pod-name> -n <namespace>

# Logs from all pods in deployment
kubectl logs -n <namespace> -l app=<app-name> --all-containers=true

# Logs from specific time range
kubectl logs <pod> -n <ns> --since=10m

# Save logs to file
kubectl logs <pod> -n <ns> > logs.txt

Resource Inspection

# Get all resources in namespace
kubectl get all -n <namespace>

# Describe all pods
kubectl describe pods -n <namespace>

# Get events sorted by time
kubectl get events -n <namespace> --sort-by='.lastTimestamp'

# Check resource usage
kubectl top nodes
kubectl top pods -n <namespace>

Getting Help

If you can’t resolve an issue:

  1. Check this troubleshooting guide

  2. Review System Architecture

  3. Consult relevant Runbook

  4. Gather diagnostics and contact DevOps team

Diagnostic Checklist:

  • kubectl get pods output

  • kubectl describe pod output

  • kubectl logs output

  • kubectl get events output

  • kubectl top output (if performance issue)

  • ArgoCD application status (if deployment issue)

Related Documentation

Last updated: 2026-08-08 10:57:10 +0200