It recently troubles me that my GKE cluster is managing more and more resources and causing difficulties in maintenance, one of which involves how to delete resources properly.

Next, I will explain step by step the difficulties I encountered in my work and how I finally used the combination of bash commands and kubectl to implement the edition of Kubernetes objects through one-line command.

In general

The following are the basic operations that almost every Kubernetes user takes. Check 👉 the cheatsheet if you’re not familiar with the process.

• Delete a single resource through the kubectl command. For example

kubectl delete pod/name-123 -n default

When the action succeeds, the following prompt appears.

pod/name-123 deleted

• But what if hundreds of resources need to be deleted?

It’s not complicated if your resources are with good labels or annotations. You can delete them as long as you find them.

# find the resources and make sure you are doing the right thing
kubectl get pod -l --selector= -n default
# delete
kubectl delete pod -l --selector= -n default

Difficulties

The delete command is not hard, but the problem is that if the deleted resource is stuck and there returns no correct prompt. 🔥

It does not mean that the deletion is successful in the backend when you force the termination with Ctrl+C. While it often takes a long time or will never really succeed.

I don’t know if you have encountered this problem before. I have experienced many times anyway, especially when deleting some GCP resources, such as Bigtable, BigdataInstance, or some IAM resources.

For details, see StackOverflow ticket and Github issue, and many people have the same trouble.

Solutions

With StackOverflow’s answers and Github issue tickets, and after my attempt, I found that the most effective way is to delete metadata.finalizers.

The most likely reason for delete stuck is that Kubernetes APIServer is executing finalizer fucntion.

I can’t explain why since it involves many internal implementations of Operator (Kubernetes CRM). And each resource type may be in a different situation, particularly when many of the resource types I deal with come from Google Cloud. If you know Google 🤷🏻‍♀️

How?

You can use the following method when you only need to delete one or a few resources.

• Edit the object first, and then delete

kubectl edit pod/name-123 -n default 
# delete the finailizers[]
kubectl detele pod/name-123 -n default

• Use jq, call Kube Proxy. Refer 👉 Github issue

kubectl get pod/name-123 -n default  -o json | \
  jq '.spec.finalizers=[]' | \
  curl -X PUT http://localhost:8001/api/v1/pod/name-123/finalize -H "Content-Type: application/json" --data @-
kubectl detele pod/name-123 -n default

• Use yq. For more yqfunctions, 👉 here

kubectl get pod/name-123 -n default | \
yq d -i 'metadata.finalizers' 
kubectl detele pod/name-123 -n default

Batch processing

The above solutions can handle most situations.

However, when facing failures in deleting dozens of resource objects, which I have encountered before, what can we do?

Batch processing is an obvious option, as no programmer would operate one by one like a robot in this situation.

Then the question is how to achieve it?

• bash script

The most straightforward way is to accomplish by executing the above separate operations in the for loop.

One-line command

As a programmer with pursuits and always wanting to try something new, I try to implement it with a one-line command, a relatively cooler method.

Again, How?

I am sure it is feasible. But as a Java programmer, I don’t have a lot of Bash experience myself, and my Bash knowledge is only at the “beginner” level.

So, I need to figure out which commands can help me achieve my goal.

• kubectl get. No need to emphasize. I need to search out the objects.
• yq. As seen before, using the yq should be a must-have skill for operations and reading YAML files if you are a Kubernetes user.

Here I use the yq d -ifunction, where d is to delete a field in the YAML file, and -i option means that this is an in-place operation.

• kubectl edit. We need kubectl edit to help us achieve submitting after modification because yq can only modify files but not submit the changes to Kubernetes.
• grep. A very practical command to help me process the information returned from kubectl get.
• xargs. I have known this command for a long time but rarely used it. I think it is challenging to use grep, xargs, sedand other commands perfectly.

Generally speaking, xargs is always used to coordinate with pipes, read input from stdin, and execute subsequent commands, as well as set parameters. The options I need here include n, P, and so on.

For xargs specific guidelines, 👉 here

One-line command:

kubectl get gcp --all-namespaces |\
 grep -v '^$' | grep -v '^NAMESPACE' | \
xargs -P8 -n3 sh -c  \
'KUBE_EDITOR="yq d -i -- $1 metadata.finalizers" kubectl edit -n $0 $1'

I will write out the result first and then analyze why.

1. kubectl get gcp --all-namespaces gets all the GCP resources. It's easy to understand.

The output is like this 👇

NAMESPACE                 NAME                               AGE
test1         sqluser.sql.cnrm.cloud.google.com/test-admin   4h16m
​
NAMESPACE                 NAME                               AGE
tes2          sqldatabase.sql.cnrm.cloud.google.com/test2     4h17m
test3         sqldatabase.sql.cnrm.cloud.google.com/test3     15d

2. grep -v '^$' | grep -v '^NAMESPACE' are used to remove the empty lines and title lines.

After this, the output is like 👇

test1        sqluser.sql.cnrm.cloud.google.com/test-admin      4h16m
tes2         sqldatabase.sql.cnrm.cloud.google.com/test2       4h17m
test3        sqldatabase.sql.cnrm.cloud.google.com/test3         15d

Now, the interesting part is here.

3. KUBE_EDITOR is a kubectl ENV var, used to change the default kubectl edit editor. The default editor is vi. And here, I use the yq cmd to replace the default editor and remove the metadata.finalizers.

4. xargs is what I take the most time to figure out. P helps the cmd to run in parallel. n takes 3 arguments from the stdin. Because the default output of the kubectl get has three arguments in one line.

Finally, the command will run with kubectl edit ${namespace} ${name} with the editor yq d -i -- $1 metadata.finalizers.

Eventually, I successfully edited all my 59 GCP resources with one-line command and deleted all of them without stuck.

Thanks for reading!

Note: The post is authorized by original author to republish on our site. Original author is Stefanie Lai who is currently a Spotify engineer and lives in Stockholm, orginal post is published here.