I have seen many cases where the supervisor namespace gets stuck at Terminating phase waiting on finalization on some of its child resources. This plugin can be used for setting finalizer to null for all objects of a specified api resource under a supervisor namespace. It will be helpful in cleaning up supervisor namespaces stuck terminating phase and can be also used to clean up stale resources under a supervisor namespace.
kubectl-nullfinalizer
#!/bin/bash
Help()
{
# Display Help
echo "This plugin sets finalizer to null for specified resource in a namespace."
echo "Usage: kubectl nullfinalizer SVNAMESPACE RESOURCENAME"
echo "Example: kubectl nullfinalizer vineetha-svns01 pvc"
}
# Get the options
while getopts ":h" option; do
case $option in
h) # display Help
Help
exit;;
\?) # incorrect option
echo "Error: Invalid option"
exit;;
esac
done
kubectl get -n $1 $2 --no-headers | awk '{print $1}' | xargs -I{} kubectl patch -n $1 $2 {} -p '{"metadata":{"finalizers": null}}' --type=merge
Usage
- Place the plugin in the system executable path.
- I placed it in $HOME/.krew/bin in my laptop.
- Once you copied the plugin to the proper path, you can make it executable by: chmod 755 kubectl-nullfinalizer .
- After that you should be able to run the plugin as: kubectl nullfinalizer SUPERVISORNAMESPACE RESOURCENAME .
Example
Following is an exmaple of a supervisor namespace stuck at Terminating phase. While describe you can see that it is waiting on finalization.
❯ k config current-context
wdc-08-vc07
❯ kg ns svc-sct-bot-dogfooding
NAME STATUS AGE
svc-sct-bot-dogfooding Terminating 584d
❯
❯ kg ns svc-sct-bot-dogfooding -oyaml
status:
conditions:
- lastTransitionTime: "2023-09-26T04:45:21Z"
message: All resources successfully discovered
reason: ResourcesDiscovered
status: "False"
type: NamespaceDeletionDiscoveryFailure
- lastTransitionTime: "2023-09-26T04:45:21Z"
message: All legacy kube types successfully parsed
reason: ParsedGroupVersions
status: "False"
type: NamespaceDeletionGroupVersionParsingFailure
- lastTransitionTime: "2023-09-26T04:45:21Z"
message: All content successfully deleted, may be waiting on finalization
reason: ContentDeleted
status: "False"
type: NamespaceDeletionContentFailure
- lastTransitionTime: "2023-09-26T04:45:21Z"
message: 'Some resources are remaining: clusters.cluster.x-k8s.io has 1 resource
instances, kubeadmcontrolplanes.controlplane.cluster.x-k8s.io has 1 resource
instances, machines.cluster.x-k8s.io has 4 resource instances, persistentvolumeclaims.
has 9 resource instances, projects.registryagent.vmware.com has 1 resource instances,
tanzukubernetesclusters.run.tanzu.vmware.com has 1 resource instances'
reason: SomeResourcesRemain
status: "True"
type: NamespaceContentRemaining
- lastTransitionTime: "2023-09-26T04:45:21Z"
message: 'Some content in the namespace has finalizers remaining: cluster.cluster.x-k8s.io
in 1 resource instances, cns.vmware.com/pvc-protection in 9 resource instances,
controller-finalizer in 1 resource instances, kubeadm.controlplane.cluster.x-k8s.io
in 1 resource instances, machine.cluster.x-k8s.io in 4 resource instances, tanzukubernetescluster.run.tanzu.vmware.com
in 1 resource instances'
reason: SomeFinalizersRemain
status: "True"
type: NamespaceFinalizersRemaining
phase: Terminating
❯ kg pvc -n svc-sct-bot-dogfooding
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
gc1-workers-r9jvb-4sfjc-containerd Terminating pvc-0d9f4a38-86ad-41d8-ab11-08707780fd85 70Gi RWO wdc-08-vc07c01-wcp-mgmt 538d
gc1-workers-r9jvb-szg9r-containerd Terminating pvc-ca6b6ec4-85fa-464c-abc6-683358994f3f 70Gi RWO wdc-08-vc07c01-wcp-mgmt 538d
gc1-workers-r9jvb-zbdt8-containerd Terminating pvc-8f2b0683-ebba-46cb-a691-f79a0e94d0e2 70Gi RWO wdc-08-vc07c01-wcp-mgmt 538d
gc2-workers-vpzl2-ffkgx-containerd Terminating pvc-69e64099-42c8-44b5-bef2-2737eca49c36 70Gi RWO wdc-08-vc07c01-wcp-mgmt 510d
gc2-workers-vpzl2-hww5v-containerd Terminating pvc-5a909482-4c95-42c7-b55a-57372f72e75f 70Gi RWO wdc-08-vc07c01-wcp-mgmt 510d
gc2-workers-vpzl2-stsnh-containerd Terminating pvc-ed7de540-72f4-4832-8439-da471bf4c892 70Gi RWO wdc-08-vc07c01-wcp-mgmt 510d
gc3-workers-2qr4c-64xpz-containerd Terminating pvc-38478f19-8180-4b9b-b5a9-8c06f17d0fbc 70Gi RWO wdc-08-vc07c01-wcp-mgmt 510d
gc3-workers-2qr4c-dpng5-containerd Terminating pvc-a8b12657-10bd-4993-b08e-51b7e9b259f9 70Gi RWO wdc-08-vc07c01-wcp-mgmt 538d
gc3-workers-2qr4c-wfvvd-containerd Terminating pvc-01c6b224-9dc0-4e03-b87e-641d4a4d0d95 70Gi RWO wdc-08-vc07c01-wcp-mgmt 538d
❯
❯ k nullfinalizer -h
This plugin sets finalizer to null for specified resource in a namespace.
Usage: kubectl nullfinalizer SVNAMESPACE RESOURCENAME
Example: kubectl nullfinalizer vineetha-svns01 pvc
❯
❯
❯ k nullfinalizer svc-sct-bot-dogfooding pvc
persistentvolumeclaim/gc1-workers-r9jvb-4sfjc-containerd patched
persistentvolumeclaim/gc1-workers-r9jvb-szg9r-containerd patched
persistentvolumeclaim/gc1-workers-r9jvb-zbdt8-containerd patched
persistentvolumeclaim/gc2-workers-vpzl2-ffkgx-containerd patched
persistentvolumeclaim/gc2-workers-vpzl2-hww5v-containerd patched
persistentvolumeclaim/gc2-workers-vpzl2-stsnh-containerd patched
persistentvolumeclaim/gc3-workers-2qr4c-64xpz-containerd patched
persistentvolumeclaim/gc3-workers-2qr4c-dpng5-containerd patched
persistentvolumeclaim/gc3-workers-2qr4c-wfvvd-containerd patched
❯
❯
❯ kg projects.registryagent.vmware.com -n svc-sct-bot-dogfooding
NAME AGE
svc-sct-bot-dogfooding 584d
❯
❯ k nullfinalizer -h
This plugin sets finalizer to null for specified resource in a namespace.
Usage: kubectl nullfinalizer SVNAMESPACE RESOURCENAME
Example: kubectl nullfinalizer vineetha-svns01 pvc
❯
❯ k nullfinalizer svc-sct-bot-dogfooding projects.registryagent.vmware.com
project.registryagent.vmware.com/svc-sct-bot-dogfooding patched
❯
❯
❯ kg ns svc-sct-bot-dogfooding
Error from server (NotFound): namespaces "svc-sct-bot-dogfooding" not found
❯
Hope it was useful. Cheers!
