Kubernetes 101 - Part12 - Debug pod

When it comes to troubleshooting application connectivity and name resolution issues in Kubernetes, having the right tools at your disposal can make all the difference. One of the most common challenges is accessing essential utilities like ping, nslookup, dig, traceroute, and more. To simplify this process, we've created a container image that packs a range of these utilities, making it easy to quickly identify and resolve connectivity issues.

 

The Container Image: A Swiss Army Knife for Troubleshooting

This container image, designed specifically for Kubernetes troubleshooting, comes pre-installed with the following essential utilities:

1. ping: A classic network diagnostic tool for testing connectivity.
   
2. dig: A DNS lookup tool for resolving domain names to IP addresses.
3. nslookup: A network troubleshooting tool for resolving hostnames to IP addresses.
4. traceroute: A network diagnostic tool for tracing the path of packets across a network.
5. curl: A command-line tool for transferring data to and from a web server using HTTP, HTTPS, SCP, SFTP, TFTP, and more.
6. wget: A command-line tool for downloading files from the web.
7. nc: A command-line tool for reading and writing data to a network socket.
8. netstat: A command-line tool for displaying network connections, routing tables, and interface statistics.
9. ifconfig: A command-line tool for configuring network interfaces.
10. route: A command-line tool for displaying and modifying the IP routing table.
11. host: A command-line tool for performing DNS lookups and resolving hostnames.
12. arp: A command-line tool for displaying and modifying the ARP cache.
13. iostat: A command-line tool for displaying disk I/O statistics.
14. top: A command-line tool for displaying system resource usage.
15. free: A command-line tool for displaying free memory and swap space.
16. vmstat: A command-line tool for displaying virtual memory statistics.
17. pmap: A command-line tool for displaying process memory maps.
18. mpstat: A command-line tool for displaying multiprocessor statistics.
19. python3: A programming language and interpreter.
20. pip: A package installer for Python.

 

Run as a pod on Kubernetes

❯ kubectl run debug --image=vineethac/debug -n default -- sleep infinity

 

Exec into the debug pod

❯ kubectl exec -it debug -n default -- bash 
root@debug:/# ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=46 time=49.3 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=45 time=57.4 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=46 time=49.4 ms
^C
--- 8.8.8.8 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 49.334/52.030/57.404/3.799 ms
root@debug:/#
root@debug:/# nslookup google.com
Server:		10.96.0.10
Address:	10.96.0.10#53

Non-authoritative answer:
Name:	google.com
Address: 142.250.72.206
Name:	google.com
Address: 2607:f8b0:4005:80c::200e

root@debug:/# exit
exit
❯

 

Reference

https://github.com/vineethac/Docker/tree/main/debug-image

By having these essential utilities at your fingertips, you'll be better equipped to quickly identify and resolve connectivity issues in your Kubernetes cluster, saving you time and reducing the complexity of troubleshooting.

Hope it was useful. Cheers!

Benchmarking Kubernetes infrastructure using K-Bench

K-Bench is a framework to benchmark the control and data plane aspects of a Kubernetes cluster. More details are available at https://github.com/vmware-tanzu/k-bench. In my case, I am going to conduct this benchmarking study on a Tanzu Kubernetes cluster which is provisioned using Tanzu Kubernetes Grid service on a vSphere 7 U1 cluster.

Step 1: Clone the K-Bench repo

git clone https://github.com/vmware-tanzu/k-bench.git

Step 2: Install

./install.sh

Once the installation is done it will say, "Completed k-bench installation.".

Step 3: Run the benchmark

./run.sh

If you don't specify any test, then it is going to conduct the default set of tests. All sets of tests are defined under the config directory. If you browse to the config directory and list, there are separate folders specific to each test. You can see folders starting with cp and dp, and it refers to control plane and data plane related tests.

If no specific test is mentioned, then it is going to run all that is defined in the default directory. You can also see details of the test and results in the logs. The directories starting with "results" will have log files corresponding to each test run.

Following is a sample log that shows a summary of pod creation throughput, pod creation average latency, pod startup total latency, list/ update/ delete pod latency, etc.

Now, if you want to run a specific test case, you can do it as follows:

Usage: ./run.sh -r <run-tag> [-t <comma-separated-tests> -o <output-dir>]

DP network internode test

For example, you can run a data plane test to check the network performance between two nodes as shown below.

./run.sh -r "kbench-run-on-tkg-cluster-02"  -t "dp_network_internode" -o "./"

As soon as you run the above command, two pods will be created inside "kbench-pod-namespace" on two worker nodes as you see below.

It will then start "iperf3" process inside those two pods to create a network load following a client-server model as per the actions defined in the config.json file.

Sample logs are given below. It shows details like the amount of data transferred, transfer rate, network latency, etc.

Once the test run is complete, the pods and other resources created will be automatically deleted. Similarly, you can select the other set of tests that are pre-defined in the framework. I believe you have the flexibility to define custom test cases too as per your requirements. I hope it was useful. Cheers!

Related posts

Storage performance benchmarking of Tanzu Kubernetes clusters
Monitoring Tanzu Kubernetes cluster using Prometheus and Grafana

References

Tanzu Kubernetes Grid (TKG) on vSphere 6.7 U3 - Part3

In this blog, I will explain how to deploy an FIO application pod with persistent storage on your Tanzu Kubernetes workload cluster.

Step 1: Deploy a K8s workload cluster

tkg create cluster <cluster name> --plan=dev

Now the workload K8s cluster is deployed with a Master, LB, and Worker node.

Tanzu Kubernetes Grid (TKG) on vSphere 6.7 U3 - Part2

In this post, I will explain how to deploy and manage multiple Kubernetes workload clusters using TKG CLI.

To view the management cluster: tkg get management-cluster

To create a new workload cluster: tkg create cluster <cluster name> --plan=<cluster plan>

Now as per default dev plan one master, one worker, and a load balancer are deployed.

Tanzu Kubernetes Grid (TKG) on vSphere 6.7 U3 - Part1

TKG is an enterprise-ready Kubernetes runtime which provides a consistent, upstream-compatible implementation of Kubernetes, that is tested, signed, and supported by VMware. 

Installation

I am using a 3 node vSAN cluster running vSphere 6.7 U3 to deploy TKG. The first step is to prepare a VM that will be used to kickstart the deployment process. Here I am using a CentOS 7 VM with desktop UI. Download the TKG CLI, TKG Kubernetes OVA, and Load Balancer OVA from the following link:

https://my.vmware.com/en/web/vmware/downloads/info/slug/infrastructure_operations_management/vmware_tanzu_kubernetes_grid/1_x 

I am using the following versions:

• VMware Tanzu Kubernetes Grid CLI 1.1 Linux
• VMware Tanzu Kubernetes Grid 1.1.0 Kubernetes v1.18.2 OVA
• VMware Tanzu Kubernetes Grid 1.1 Load Balancer OVA

Unzip and install TKG CLI on the CentOS VM.

Kubernetes 101 - Part3 - Install kubectl on Windows

This article shows how to install kubectl on a Windows machine and connect to a remote Kubernetes cluster.

Open PowerShell as administrator and run the following:

Install-Script -Name install-kubectl -Scope CurrentUser -Force

This will download the installation files to Windows PowerShell script folder of the current user. In my case it is: C:\Users\vineetha\Documents\WindowsPowerShell\Scripts

Now browse to the above location in Powershell and execute the install-kubectl.ps1 file.

install-kubectl.ps1 [-DownloadLocation <path>]

Note: If you do not specify a DownloadLocation, kubectl will be installed in the user's temp directory.