In the realm of managing Tanzu Kubernetes clusters (TKCs), we have encountered several challenges that hindered the smooth functioning of our applications. In this blog post, we will discuss three such cases and the workarounds we employed to resolve them.
Case 1: TKC Control Plane Node Connectivity Issues
Symptoms:
- TKC apiserver connection timeouts when attempting to connect using the kubeconfig.
- Traffic was not flowing to two of the control plane nodes.
- NSX-T web UI LB VS stats indicated this issue.
.png)
Case 2: TKC Worker Node Connectivity Issues
Symptoms:
- Workload (example: PostgreSQL cluster) connection timeouts.
- Traffic was not flowing to two of the worker nodes in the TKC.
- NSX-T web UI LB VS stats indicated this issue.
Case 3: Load Balancer Connectivity Issues
Symptoms:
- Connection timeouts when attempting to connect to a PostgreSQL workload through the load balancer VS IP.
- This issue was observed only when creating new services of type LoadBalancer in the TKC.
- We noticed datapath mempool usage for the edge nodes was above the threshold value.
Resolution/ work around
- Find the T1 router that is attached to the TKC which has connectivity issues.
- In an Active - Standby HA configuration, you will see that there will be one Edge node that will be Active and another one in Standby status.
- First place the Standby Edge node in NSX MM, reboot it, and then exit it from NSX MM.
- Now, place the Active Edge node in NSX MM, there will be a slight network disruption during this failover, once it is in NSX MM, reboot it, and then exit NSX MM.
- This should resolved the issue.
In conclusion, these cases illustrate the importance of verifying NSX-T components when managing Tanzu Kubernetes clusters. By identifying the root cause of the issues and employing effective workarounds, we were able to restore functionality and maintain the health of our applications. Stay tuned for more insights and best practices in managing Kubernetes clusters.
Hope it was useful. Cheers!
