The command-line tool kubectl is indispensable when using Kubernetes. You need it to query related Pod and Service information both in developing or performing some maintenance operations, such as events, scale, rolling update, etc.

However, when using kubectl, there are many inconveniences. Though Kubernetes is officially maintaining kubectl¹, and you can submit the problems, improvement, and even PR in its Github Issues, you still have to wait long before its release.

The most common solution is Kubectl-Plugin Pattern²

Plugins extend kubectl with new sub-commands, allowing for new and custom features not included in the main distribution of kubectl --from Kubernete²

Plugins are some codes that can be used as a subcommand of kubectl. Usually, we use plugins to solve complex information queries, monitor log collection, debug Kubernetes resources, and so on.

This article is to record my process and experience in writing kubectl plugins.

Use existing Kubectl Plugins

In the beginning, general users will search for the tools they need in the existing plugin list instead of writing plugins by themselves. Pursue the use, for their own use.

Kubernetes officially provides the tool krew³ as a plugin manager, and it is similar to tools such as brew and apt. The following describes how to install and use krew to find the plugins you want (all operations are based on Mac)

• Install krew. You must install Git first.

(
  set -x; cd "$(mktemp -d)" &&
  curl -fsSLO "https://github.com/kubernetes-sigs/krew/releases/latest/download/krew.tar.gz" &&
  tar zxvf krew.tar.gz &&
  KREW=./krew-"$(uname | tr '[:upper:]' '[:lower:]')_$(uname -m | sed -e 's/x86_64/amd64/' -e 's/arm.*$/arm/')" &&
  "$KREW" install krew
)export PATH="${KREW_ROOT:-$HOME/.krew}/bin:$PATH"

• Search for the plugin.

krew list

• Install the plugin. Kubectl-tree⁴ is a well-known plugin, which has 1.6k+ stars on Github.

• Use the plugin.

• Uninstall the plugin

There are also many kubectl plugins out the scope of the krew management, and you can find them on Github.

DIY Kubectl Plugin — simple bash

When you find that the existing plugins fail to meet your needs, it is necessary to customize one. And it’s not that hard!

The kubectl command can support the plugin mechanism from version v1.8.0, after which we can extend the kubectl command in all the subsequent versions. And the kubectl plugin after v1.12 is directly a binary file with commands starting with kubectl-*, then it enters the GA at v1. 14.

With these improvements, users can extend their kubectl subcommands in the form of binary files. Of course, the kubectl plugin mechanism gives users a free choice of languages, meaning that you can write plugins in any language.

Implement Bash Kubectl Plugin

The simplest kubectl-plugin I can think of is a bash script. For example, the following command, which I make into a kubectl plugin, is the one I use to query whether GCP resources and IAM related resources are ready.

kubectl get all,gcp,iamserviceaccount,iampolicymember,iampolicy --all-namespaces \
          -ocustom-columns='KIND:.kind,NAMESPACE:.metadata.namespace,NAME:.metadata.name,READY:.status.conditions[?(@.type=="Ready")].status,MESSAGE:.status.conditions[?(@.type=="Ready")].message'

• Put the command into a kubectl-getready script and grant execution permissions.

$ cat << EOF >> kubectl-getready
kubectl get all,gcp,iamserviceaccount,iampolicymember,iampolicy --all-namespaces \
          -ocustom-columns='KIND:.kind,NAMESPACE:.metadata.namespace,NAME:.metadata.name,READY:.status.conditions[?(@.type=="Ready")].status,MESSAGE:.status.conditions[?(@.type=="Ready")].message'
EOF$ chmod +x kubectl-getready

• Move the command under to the /usr/local/bin directory.

mv kubectl-getready /usr/local/bin

• Execute kubectl getready to get the same effect as above.

$ kubectl getready 

It is an attempt on kubectl plugin mode, and you can execute it normally without encapsulating this bash script into a subcommand of kubectl.

DIY Kubectl Plugin — Golang

Aside from the above bash attempts, most of the kubectl plugins are written in Go. Using k8s.io/cli-runtime⁵ library is officially recommended and with an example sample-cli-plugin for reference.

Here I continue with my own experiment.

I found myself repeating the commands kubectl get pod, kubectl describe pod, kubectl get events, and kubectl logs in recent work. 🔥

Because Pod crashes and restarts frequently, Pod’s name changes frequently and it is often very long that needs to be obtained through get command before you can debug it further. Then it came to my thought, why not make a simple kubectl plugin that supports Prefix* queries?🧯

I haven’t found a similar plugin yet, or maybe I missed it. Please leave me a comment if you know one.

Design

The purpose of using plugins is straightforward, to query Pod through prefix regex match, then fetch information such as name, namespace, conditions, events, logs, etc.

While writing the plugins, I am also learning how to interact with kubernetes cli-runtime⁵ API and better understand Kubernetes APIServer’s design and implementation logic.

The plugin should be supporting some parameters.

• -A --all-namespaces supports all namespaces queries.
• -n --namespace supports specific namespace queries.
• -s --size supports the number of matches. In terms of performance, it shows only a certain number of matched pods by default.
• -l --logsize supports the number of logs displayed.

Implementation

I also implement it by modifying the sample-cli-plugin above.

To implement such a command-line tool in Go, I always use the cobra⁶ library, by which I can define Command, read Flag, and well encapsulate command-line input.

• Define Command and flag

  r := &QueryDetail{}
	o := config.NewQueryDetaiConfig()
	c := &cobra.Command{
		Use:          "ff [flags]",
		Short:        "View pods logs & status & events",
		Example:      fmt.Sprintf(logExample, "kubectl ff"),
		SilenceUsage: true,
		Args:         cobra.MinimumNArgs(1),
		PreRunE:      r.preRunE,
		RunE:         r.runE,
	}

	//cmdutil.FixDocs("kpt", parent, c)
	c.PersistentFlags().StringVarP(&o.Namespace, "all-namespaces", "A", "","search all the namespaces")
	c.PersistentFlags().StringVarP(&o.Namespace, "namespace", "n", "","Namespace for search. (default: \"default\")")
	c.Flags().IntVarP(&o.ShowSize, "size", "s", 3, "If present, list the number of pods matches the regex (default:3)")
	c.Flags().IntVarP(&o.LogSize, "log-size", "b", 10, "If present, show number of lines(default:10)")

	r.Command = c

• Initialize parameter processing, parse the incoming args, and determine whether to use prefix match. This could be done in either preRunE or runE functions.

func (r *QueryDetail) Complete(cmd *cobra.Command, args []string) error {
  if strings.HasSuffix(args[0], "*") {
    r.IsPrefix = true
    r.NamePrefix = args[0][0 : len(args[1])-1]
  } else {
    r.IsPrefix = false
    r.NamePrefix = args[0]
  }
  return nil
}

• Execute. I skip the part without Prefix and see how I access the Kubernetes API simply through client-go⁷ library, and get the results I want.
  1. Declare a clientset
  2. Get the PodList, and traverse the PodList to get the Pod that matches.
  3. Print the Pod information, including basic information, events, logs.

func run(ns string) {
  clientset := client.InitClient() // step 1
  list, err := clientset.CoreV1().Pods(ns).List(context.TODO(), metav1.ListOptions{}) // step 2
  if err != nil {}
  for _, p := range podList.Items {
    match := strings.HasPrefix(p.Name, o.NamePrefix)
    if match {
      if podNum < o.ShowSize {
        fetchInfo(pod) // Step 3
      }
      podNum += 1
    }
  }
}

func fetchInfo(pod *corev1.Pod) {
	w.Write(utils.LEVEL_0, "Name:\t%s\n", pod.Name)
	w.Write(utils.LEVEL_0, "Namespace:\t%s\n", pod.Namespace)
	if len(pod.Status.Conditions) > 0 {
	w.Write(utils.LEVEL_0, "Conditions:\n  Type\tStatus\n")
	for _, c := range pod.Status.Conditions {
		w.Write(utils.LEVEL_1, "%v \t%v \n",
			c.Type,
			c.Status)
		}
	}
	events, _ := clientset.CoreV1().Events(pod.Namespace).Search(scheme.Scheme, ref)
	w.Write(utils.LEVEL_0, "Events:\n  Type\tReason\tAge\tFrom\tMessage\n")
	w.Write(utils.LEVEL_1, "----\t------\t----\t----\t-------\n")
	for _, e := range events.Items {
		var interval string
		if e.Count > 1 {
			interval = fmt.Sprintf("%s (x%d over %s)", utils.TranslateTimestampSince(e.LastTimestamp), e.Count, utils.TranslateTimestampSince(e.FirstTimestamp))
		} else {
			interval = utils.TranslateTimestampSince(e.FirstTimestamp)
			if e.FirstTimestamp.IsZero() {
				interval = utils.TranslateMicroTimestampSince(e.EventTime)
			}
		}
		source := e.Source.Component
		if source == "" {
			source = e.ReportingController
		}
		w.Write(utils.LEVEL_1, "%v\t%v\t%s\t%v\t%v\n",
			e.Type,
			e.Reason,
			interval,
			source,
			strings.TrimSpace(e.Message))
   }
   // logs
  podLogs := clientset.CoreV1().Pods(pod.Namespace).GetLogs(pod.Name, &logOpt)
  if podLogs == nil {
    return
  }
  readCloser, err := request.Stream(context.TODO())
	if err != nil {
		return err
	}
	defer readCloser.Close()
	r := bufio.NewReader(readCloser)
	cur := 0
	for {
		readBytes, err := r.ReadBytes('\n')
		if err != nil {
			return err
		}
		out.Write(LEVEL_1, string(readBytes))
		cur += 1
		if cur == logSize {
			break
		}
	}
}

Test

After completing the above, it’s time to test the plugin.

• Compile the code first.

go build kubectl-ff.go

• Copy the compiled Go executable file to the /usr/local/bin directory.

cp ./kubectl-ff /usr/local/bin

• Execute and test through kubectl pd command.

$ kubectl ff prometheus* -A

The result would be something like the ff plugin finds the pod match in kube-system namespace and prints useful information.

During the development, I can test either by this method in real-time or by writing a unit test.

Publish to Krew

Upload your Kubectl plugin onto Github or add it to the Krew index so that others can download and install it directly.

• Define the plugin’s YAML manifest first, including an executable tar, sha code, and internal files, such as

uri: https://github.com/xxxyy/kubectl-ff/releases/download/v0.1.0/kubectl-ff-v0.1.0.tar.gz     
sha256: e1aad12fsakfdkxc212k123fd134fb046fa8276042dd1234d872a8812vcb2ad09     bin: kubectl-ff     
files:     
- from: kubectl-ff-*/kubectl-ff       
  to: .     
- from: kubectl-ff-*/LICENSE       
  to: .

• Test it by kubectl krew install --manifest locally.
• Upload it to krew index. Add the manifest to fork krew-index⁸, and then submit a PR. You share your work results with everyone after the administrator merges it.

Summary

This is just a simple kubectl plugin written in Go. I hope it helps you enter the world of kubectl plugin development. I will continue to extend my plugin, such as CRD information reading support.

Thanks for reading!

Reference

1. kubectl
2. kubectl plugin pattern
3. krew
4. kubectl-tail
5. kubernetes cli-runtime
6. cobra
7. client-go
8. krew-index

Note: The post is authorized by original author to republish on our site. Original author is Stefanie Lai who is currently a Spotify engineer and lives in Stockholm, original post is published here.