How to Fix Kubernetes Objects Stuck in Terminating state — Kubernetes Troubleshooting

When you are working on Kubernetes, sometimes we need to clean the Kubernetes objects. You may experience, Kubernetes object deletion get stuck, we need to perform some actions to delete forcefully. In this will see how we can remove the finalizer and clean the Kubernetes objects.

You may experience or heard about sometimes, Kubernetes objects like namespace get stuck in deletion. Even I have personally experienced when I was learning Kubernetes. When I was trying to troubleshoot the issue, it was completed hit and trail, there was multiple suggestion across and it took time to fix it or more over, understanding the issue.

This article prepared to help all to understand the issues and how to fix it.

Lets discuss following topics today,

• Why the Kubernetes object goes to terminating state infinitely?
• Why the deployment/s were not deleted even after deleting the k8s NameSpace?
• What is Finalizers and Owner reference in k8s?
• Solution for deleting the object which are in the terminating state.

How the delete works?

Kubernetes has its own way of managing memory and resources so does its own Garbage Collection System. It is a systematic way to remove unused/unutilized space. Programming Languages like Java/GO and the Servers built on them all have this process to ensure the memory is managed optimally.

Now, Kubernetes being the modern solution, It does manage its resources and perform Garbage collections when the resources are deleted and in various other contexts too.

Now lets come back to our Kubectl delete and why we are talking about Garbage Collection now.

Kubernetes adds a special tag or annotation to the resource called Finalizers when it creates resources which have multiple dependencies. Here is the definition able Finalizers.

FINALIZERS ARE NAMESPACED KEYS THAT TELL KUBERNETES TO WAIT UNTIL SPECIFIC CONDITIONS ARE MET BEFORE IT FULLY DELETES RESOURCES MARKED FOR DELETION. FINALIZERS ALERT CONTROLLERS TO CLEAN UP RESOURCES THE DELETED OBJECT OWNED.

Finalizers are the keys to tell Kubernetes API that, there are few resources to be deleted or taken care of before this particular resource is deleted.

For example. When you are trying to delete a Persistent Volume the associated persistent volume claims and the pods bound to it would be affected so you must follow the proper order.

Same way, When you are trying to delete an Ingress it might be associated with some infrastructure items like Load Balancers and Target Groups which needs to be deleted first before delete the ingress.

For more understanding, lets create the namespace and deployment with finalizers.

Create the NameSpace and add the Finalizer

Here is the command to create the namespace called, demo-namespace

# kubectl create ns demo-namespace

This command helps to view the namespace manifest.

# kubectl edit ns demo-namespace
apiVersion: v1
kind: Namespace
metadata:
  creationTimestamp: "2022-12-10T02:13:53Z"
  labels:
    kubernetes.io/metadata.name: demo-namespace
  name: demo-namespace
  finalizers:
  - kubernetes

Create a test deployment and add Finalizer

Lets create a deployment called demo in namespace demo-namespace

# kubectl create deployment demo --image=nginx -n demo-namespace

Here is the manifests for deployment.

# kubectl edit deployment demo -n demo-namespace
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "1"
  creationTimestamp: "2022-12-10T03:03:18Z"
  finalizers:
  - kubernetes
  generation: 1
  labels:
    app: demo
  name: demo

Issue 1: The NameSpace was stuck in Terminating state

To understand it, lets delete the NameSpace that we created.

# kubectl delete ns demo-namespace
namespace "demo-namespace" deleted

It will show the deleted message but won’t exit, because it is waiting at the Finalizers.

Continue Reading on: https://foxutech.com/how-to-fix-kubernetes-objects-stuck-in-terminating-state-kubernetes-troubleshooting/