Encountering issues with connectivity to your TKC apiserver/ control plane can be frustrating. One common problem we've seen is the kubeconfig failing to connect, often due to missing server pool members in the load balancer's virtual server (LB VS).
The Issue
The LB VS, which operates on port 6443, should have the control plane VMs listed as its member servers. When these members are missing, connectivity problems arise, disrupting your access to the TKC apiserver.
Troubleshooting steps
- Access the TKC: Use the kubeconfig to access the TKC.
❯ KUBECONFIG=tkc.kubeconfig kubectl get node
Unable to connect to the server: dial tcp 10.191.88.4:6443: i/o timeout
❯
- Check the Load Balancer: In NSX-T, verify the status of the corresponding load balancer (LB). It may display a green status indicating success.
- Inspect Virtual Servers: Check the virtual servers in the LB, particularly on port 6443. They might show as down.
- Examine Server Pool Members: Look into the server pool members of the virtual server. You may find it empty.
- SSH to Control Plane Nodes: Attempt to SSH into the TKC control plane nodes.
- Run Diagnostic Commands: Execute diagnostic commands inside the control plane nodes to verify their status. The issue could be that the control plane VMs are in a hung state, and the container runtime is not running.
vmware-system-user@tkc-infra-r68zc-jmq4j [ ~ ]$ sudo su
root [ /home/vmware-system-user ]# crictl ps
FATA[0002] failed to connect: failed to connect, make sure you are running as root and the runtime has been started: context deadline exceeded
root [ /home/vmware-system-user ]#
root [ /home/vmware-system-user ]# systemctl is-active containerd
Failed to retrieve unit state: Failed to activate service 'org.freedesktop.systemd1': timed out (service_start_timeout=25000ms)
root [ /home/vmware-system-user ]#
root [ /home/vmware-system-user ]# systemctl status containerd
WARNING: terminal is not fully functional
- (press RETURN)Failed to get properties: Failed to activate service 'org.freedesktop.systemd1'>
lines 1-1/1 (END)lines 1-1/1 (END)
- Check VM Console: From vCenter, check the console of the control plane VMs. You might see specific errors indicating issues.
EXT4-fs (sda3): Delayed block allocation failed for inode 266704 at logical offset 10515 with max blocks 2 with error 5
EXT4-fs (sda3): This should not happen!! Data will be lost
EXT4-fs error (device sda3) in ext4_writepages:2905: IO failure
EXT4-fs error (device sda3) in ext4_reserve_inode_write:5947: Journal has aborted
EXT4-fs error (device sda3) xxxxxx-xxx-xxxx: unable to read itable block
EXT4-fs error (device sda3) in ext4_journal_check_start:61: Detected aborted journal
systemd[1]: Caught <BUS>, dumped core as pid 24777.
systemd[1]: Freezing execution.
- Restart Control Plane VMs: Restart the control plane VMs. Note that sometimes your admin credentials or administrator@vsphere.local credentials may not allow you to restart the TKC VMs. In such cases, decode the username and password from the relevant secret and use these credentials to connect to vCenter and restart the hung TKC VMs.
❯ kubectx wdc-01-vc17
Switched to context "wdc-01-vc17".
❯
❯ kg secret -A | grep wcp
kube-system wcp-authproxy-client-secret kubernetes.io/tls 3 291d
kube-system wcp-authproxy-root-ca-secret kubernetes.io/tls 3 291d
kube-system wcp-cluster-credentials Opaque 2 291d
vmware-system-nsop wcp-nsop-sa-vc-auth Opaque 2 291d
vmware-system-nsx wcp-cluster-credentials Opaque 2 291d
vmware-system-vmop wcp-vmop-sa-vc-auth Opaque 2 291d
❯
❯ kg secrets -n vmware-system-vmop wcp-vmop-sa-vc-auth
NAME TYPE DATA AGE
wcp-vmop-sa-vc-auth Opaque 2 291d
❯ kg secrets -n vmware-system-vmop wcp-vmop-sa-vc-auth -oyaml
apiVersion: v1
data:
password: aWAmbHUwPCpKe1Uxxxxxxxxxxxx=
username: d2NwLXZtb3AtdXNlci1kb21haW4tYzEwMDYtMxxxxxxxxxxxxxxxxxxxxxxxxQHZzcGhlcmUubG9jYWw=
kind: Secret
metadata:
creationTimestamp: "2022-10-24T08:32:26Z"
name: wcp-vmop-sa-vc-auth
namespace: vmware-system-vmop
resourceVersion: "336557268"
uid: dcbdac1b-18bb-438c-ba11-76ed4d6bef63
type: Opaque
❯
***Decrypt the username and password from the secret and use it to connect to the vCenter.
***Following is an example using PowerCLI:
PS /Users/vineetha> get-vm gc-control-plane-f266h
Name PowerState Num CPUs MemoryGB
---- ---------- -------- --------
gc-control-plane-f2… PoweredOn 2 4.000
PS /Users/vineetha> get-vm gc-control-plane-f266h | Restart-VMGuest
Restart-VMGuest: 08/04/2023 22:20:20 Restart-VMGuest Operation "Restart VM guest" failed for VM "gc-control-plane-f266h" for the following reason: A general system error occurred: Invalid fault
PS /Users/vineetha>
PS /Users/vineetha> get-vm gc-control-plane-f266h | Restart-VM
Confirm
Are you sure you want to perform this action?
Performing the operation "Restart-VM" on target "VM 'gc-control-plane-f266h'".
[Y] Yes [A] Yes to All [N] No [L] No to All [S] Suspend [?] Help (default is "Y"): Y
Name PowerState Num CPUs MemoryGB
---- ---------- -------- --------
gc-control-plane-f2… PoweredOn 2 4.000
PS /Users/vineetha>
- Verify System Pods and Connectivity: Once the control plane VMs are restarted, the system pods inside them will start, and the apiserver will become accessible using the kubeconfig. You should also see the previously missing server pool members reappear in the corresponding LB virtual server, and the virtual server on port 6443 will be up and show a success status.
Following these steps should help you resolve the connectivity issues with your TKC apiserver/control plane effectively.Ensuring that your load balancer's virtual server is correctly configured with the appropriate member servers is crucial for maintaining seamless access. This runbook aims to guide you through the process, helping you get your TKC apiserver back online swiftly.
Note: If required for critical production issues related to TKC accessibility I strongly recommend to raise a product support request.
Hope it was useful. Cheers!
.png)
