I have seen many cases where the supervisor namespace gets stuck at Terminating phase waiting on finalization on some of its child resources. This plugin can be used for setting finalizer to null for all objects of a specified api resource under a supervisor namespace. It will be helpful in cleaning up supervisor namespaces stuck terminating phase and can be also used to clean up stale resources under a supervisor namespace.
kubectl-nullfinalizer
#!/bin/bash Help() { # Display Help echo "This plugin sets finalizer to null for specified resource in a namespace." echo "Usage: kubectl nullfinalizer SVNAMESPACE RESOURCENAME" echo "Example: kubectl nullfinalizer vineetha-svns01 pvc" } # Get the options while getopts ":h" option; do case $option in h) # display Help Help exit;; \?) # incorrect option echo "Error: Invalid option" exit;; esac done kubectl get -n $1 $2 --no-headers | awk '{print $1}' | xargs -I{} kubectl patch -n $1 $2 {} -p '{"metadata":{"finalizers": null}}' --type=merge
Usage
- Place the plugin in the system executable path.
- I placed it in $HOME/.krew/bin in my laptop.
- Once you copied the plugin to the proper path, you can make it executable by: chmod 755 kubectl-nullfinalizer .
- After that you should be able to run the plugin as: kubectl nullfinalizer SUPERVISORNAMESPACE RESOURCENAME .
Example
Following is an exmaple of a supervisor namespace stuck at Terminating phase. While describe you can see that it is waiting on finalization.
❯ k config current-context wdc-08-vc07 ❯ kg ns svc-sct-bot-dogfooding NAME STATUS AGE svc-sct-bot-dogfooding Terminating 584d ❯ ❯ kg ns svc-sct-bot-dogfooding -oyaml status: conditions: - lastTransitionTime: "2023-09-26T04:45:21Z" message: All resources successfully discovered reason: ResourcesDiscovered status: "False" type: NamespaceDeletionDiscoveryFailure - lastTransitionTime: "2023-09-26T04:45:21Z" message: All legacy kube types successfully parsed reason: ParsedGroupVersions status: "False" type: NamespaceDeletionGroupVersionParsingFailure - lastTransitionTime: "2023-09-26T04:45:21Z" message: All content successfully deleted, may be waiting on finalization reason: ContentDeleted status: "False" type: NamespaceDeletionContentFailure - lastTransitionTime: "2023-09-26T04:45:21Z" message: 'Some resources are remaining: clusters.cluster.x-k8s.io has 1 resource instances, kubeadmcontrolplanes.controlplane.cluster.x-k8s.io has 1 resource instances, machines.cluster.x-k8s.io has 4 resource instances, persistentvolumeclaims. has 9 resource instances, projects.registryagent.vmware.com has 1 resource instances, tanzukubernetesclusters.run.tanzu.vmware.com has 1 resource instances' reason: SomeResourcesRemain status: "True" type: NamespaceContentRemaining - lastTransitionTime: "2023-09-26T04:45:21Z" message: 'Some content in the namespace has finalizers remaining: cluster.cluster.x-k8s.io in 1 resource instances, cns.vmware.com/pvc-protection in 9 resource instances, controller-finalizer in 1 resource instances, kubeadm.controlplane.cluster.x-k8s.io in 1 resource instances, machine.cluster.x-k8s.io in 4 resource instances, tanzukubernetescluster.run.tanzu.vmware.com in 1 resource instances' reason: SomeFinalizersRemain status: "True" type: NamespaceFinalizersRemaining phase: Terminating ❯ kg pvc -n svc-sct-bot-dogfooding NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE gc1-workers-r9jvb-4sfjc-containerd Terminating pvc-0d9f4a38-86ad-41d8-ab11-08707780fd85 70Gi RWO wdc-08-vc07c01-wcp-mgmt 538d gc1-workers-r9jvb-szg9r-containerd Terminating pvc-ca6b6ec4-85fa-464c-abc6-683358994f3f 70Gi RWO wdc-08-vc07c01-wcp-mgmt 538d gc1-workers-r9jvb-zbdt8-containerd Terminating pvc-8f2b0683-ebba-46cb-a691-f79a0e94d0e2 70Gi RWO wdc-08-vc07c01-wcp-mgmt 538d gc2-workers-vpzl2-ffkgx-containerd Terminating pvc-69e64099-42c8-44b5-bef2-2737eca49c36 70Gi RWO wdc-08-vc07c01-wcp-mgmt 510d gc2-workers-vpzl2-hww5v-containerd Terminating pvc-5a909482-4c95-42c7-b55a-57372f72e75f 70Gi RWO wdc-08-vc07c01-wcp-mgmt 510d gc2-workers-vpzl2-stsnh-containerd Terminating pvc-ed7de540-72f4-4832-8439-da471bf4c892 70Gi RWO wdc-08-vc07c01-wcp-mgmt 510d gc3-workers-2qr4c-64xpz-containerd Terminating pvc-38478f19-8180-4b9b-b5a9-8c06f17d0fbc 70Gi RWO wdc-08-vc07c01-wcp-mgmt 510d gc3-workers-2qr4c-dpng5-containerd Terminating pvc-a8b12657-10bd-4993-b08e-51b7e9b259f9 70Gi RWO wdc-08-vc07c01-wcp-mgmt 538d gc3-workers-2qr4c-wfvvd-containerd Terminating pvc-01c6b224-9dc0-4e03-b87e-641d4a4d0d95 70Gi RWO wdc-08-vc07c01-wcp-mgmt 538d ❯ ❯ k nullfinalizer -h This plugin sets finalizer to null for specified resource in a namespace. Usage: kubectl nullfinalizer SVNAMESPACE RESOURCENAME Example: kubectl nullfinalizer vineetha-svns01 pvc ❯ ❯ ❯ k nullfinalizer svc-sct-bot-dogfooding pvc persistentvolumeclaim/gc1-workers-r9jvb-4sfjc-containerd patched persistentvolumeclaim/gc1-workers-r9jvb-szg9r-containerd patched persistentvolumeclaim/gc1-workers-r9jvb-zbdt8-containerd patched persistentvolumeclaim/gc2-workers-vpzl2-ffkgx-containerd patched persistentvolumeclaim/gc2-workers-vpzl2-hww5v-containerd patched persistentvolumeclaim/gc2-workers-vpzl2-stsnh-containerd patched persistentvolumeclaim/gc3-workers-2qr4c-64xpz-containerd patched persistentvolumeclaim/gc3-workers-2qr4c-dpng5-containerd patched persistentvolumeclaim/gc3-workers-2qr4c-wfvvd-containerd patched ❯ ❯ ❯ kg projects.registryagent.vmware.com -n svc-sct-bot-dogfooding NAME AGE svc-sct-bot-dogfooding 584d ❯ ❯ k nullfinalizer -h This plugin sets finalizer to null for specified resource in a namespace. Usage: kubectl nullfinalizer SVNAMESPACE RESOURCENAME Example: kubectl nullfinalizer vineetha-svns01 pvc ❯ ❯ k nullfinalizer svc-sct-bot-dogfooding projects.registryagent.vmware.com project.registryagent.vmware.com/svc-sct-bot-dogfooding patched ❯ ❯ ❯ kg ns svc-sct-bot-dogfooding Error from server (NotFound): namespaces "svc-sct-bot-dogfooding" not found ❯
Hope it was useful. Cheers!
