Making kurt: My first kubectl plugin for krew

Nov 07, 2021 in PROJECTS
kubernetes projects opensource
6 min read

Background

My friend and I wanted to get more involved in the CNCF (Cloud Native Computing Foundation) / Kubernetes ecosystem by making a kubectl plugin. We have both been writing golang at our jobs, but wanted an excuse to play with some of the Kubernetes client-go libraries so we could natively talk to the Kubernetes API server. This blog post will talk about what kurt does, some of the things we are doing around tech/automation, and how we got kurt into the krew package ecosystem.

kurt

We made kurt to automate the troubleshooting task of seeing what is restarting in your cluster by organizing pod restart counts into groupings that will help give you context. Kubernetes can be good at hiding application issues from an operator by restarting pods automatically when there are issues. If you are not looking for pod restarts, you might have a significant problem that Kubernetes gracefully hides from you. This is an awesome feature of Kubernetes, but when you first jump into troubleshooting what is going on in a cluster, you might want to see what is restarting. kurt effectively does a kubectl get pods --all-namespaces with the collect function. This function iterates through all the pods and gathers pod restart counts while also grouping into categories such as node, label and namespace. For example, you could run the following to see pod restart counts grouped by node:

kurt nodes

If there was one particular node that had vastly more restarts than other nodes, this might indicate a hardware issue, or possibly lead you to re-provision the affected machine.

You may also want to get the same sort of information grouped by namespace. To see this, you could run:

kurt namespaces

If you group your applications based on namespace, this might help you hone in on the specific application to troubleshoot.

If you have multiple services in a namespace, and they all have a label with the key app, you could run the following to see the top restarting services grouped this way:

kurt labels -n mynamespace -l app

If you just want to get an overview across all of these groupings, you could run:

kurt all

Which gives an output like this:

kurt: KUbernetes Restart Tracker

==========

 Namespace      Restarts

 default        2
 test           1
 kube-system    0

==========

 Node           Restarts

 minikube-m02   2
 minikube-m03   1
 minikube       0

==========

 Label                                          Restarts

 run:nginx                                      3
 component:etcd                                 0
 k8s-app:kube-proxy                             0
 addonmanager.kubernetes.io/mode:Reconcile      0
 integration-test:storage-provisioner           0

==========

 Pod                            Namespace       Restarts

 nginx                          default         2
 nginx                          test            1
 kube-apiserver-minikube        kube-system     0
 storage-provisioner            kube-system     0
 etcd-minikube                  kube-system     0

You can also limit the scope of your query to specific namespaces, or label selectors.

All options available can be found with kurt help.

What’s a kubectl plugin?

A kubectl plugin is a pattern for extending the functionality of kubectl with other custom tooling. More information can be found in the offical Kubernetes docs, but the gist is that any binary or script in your $PATH named like kubectl-myplugin can be called with kubectl like:

kubectl myplugin

Since we made kurt in golang, it can be run as a standalone binary without the kubectl prefix. However, since it is made to interact with Kubernetes we decided to also focus on the plugin experience. This is where krew comes in.

From krew’s docs: “Krew is the plugin manager for kubectl command-line tool.” krew is essentially the apt/npm/pip of Kubernetes and is a SIG CLI project. krew helps us package kurt as a kubectl plugin by submitting a manifest which gives users the latest version to be installed in the correct location, and re-named as kubectl-kurt. Users can then use it like:

kubectl kurt

Using this plugin ecosystem is valuable to users because it allows any plugin to be installed and upgraded easily with the following:

kubectl krew install kurt

kubectl krew upgrade kurt

We also get the benefit of being discoverable on the krew platform, and can view some high level analytics on number of downloads. In addition, the krew team gave us some valuable tips on best practices which makes kurt more usable as a plugin.

Tech used in this project

We wanted to use kurt as a good use-case to try various technologies and learn more about the kubectl plugin ecosystem. In addition to trying new things, our goal was to automate as many tasks around releasing a golang client application as possible.

Cobra:

To get the awesome CLI experience in kurt, we used a library called cobra. Cobra is used in many projects such as kubectl, terraform and much more. There was a bit of a learning curve to get Cobra working correctly since there are some opinions taken on repo structure, but once it is in place it is extremely easy to add new subcommands and CLI flags.

For example, once the basic project scaffolding was in place, this is all that was needed to add a subcommand like nodes:

var cmdNodes = &cobra.Command{
	Use:     "nodes",
	Short:   "Only print node restart counts",
	Long:    "Only print node restart counts",
	Aliases: []string{"no", "node"},
	Run: func(cmd *cobra.Command, args []string) {
		printNode = true
		printAll = false
		clientset := auth()
		collect(clientset, inamespace, ilabels)
	},
}

In addition to an awesome structure for adding subcommands and CLI flags, it also gave us things like shell compltion for free. For example, the following will give you bash shell completion for kurt:

source <(kurt completion bash)

Once you get over the initial learning curve of cobra, the value that it gives is amazing. I will likely use it for any other golang CLI tool I make in the future.

goreleaser:

We used goreleaser to help us with the automation of cross-compiling and releasing kurt for various platforms. You can do this manually inside any CI platform, but goreleaser gives a simple config file that allows you to specify the platforms/architectures you want, and will automate the archival and checksum file in a github release for you. Setting this up in github actions was very straightforward so any time we make push new version tag, everything is cross-compiled and released:

name: goreleaser

on:
  push:
    tags:
      - "v*.*.*"

jobs:
  goreleaser:
    runs-on: ubuntu-latest
    steps:
      -
        name: Checkout
        uses: actions/checkout@v2
        with:
          fetch-depth: 0
      -
        name: Set up Go
        uses: actions/setup-go@v2
        with:
          go-version: 1.16
      -
        name: Run GoReleaser
        uses: goreleaser/goreleaser-action@v2
        with:
          distribution: goreleaser
          version: latest
          args: release --rm-dist
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

Other automation and testing:

Dependabot to automate 3rd party version bumps.
CodeQL to check for vulnerabilities in our source code.
Custom CI to automate unit tests (with go test) and ensure all golang files are formatted with go fmt.

These all run any time a PR to main is submitted!

Future of kurt

We have a few ideas of how to enhance kurt with:

If you have other ideas of how we can improve, questions, or a feature idea for kurt, feel free to submit an issue or leave a comment in the Disqus chat!