One of the things I love most about my job is diving into the technologies we use and exploring the edges and how components fit together. This helps me form a more complete mental model of the technologies, and sometimes leads to new ideas and unexpected uses for things we may have been using for quite some time. Since adopting Kubernetes at SpiderOak, I've had many such opportunities. Kubernetes can seem daunting and complicated at first, as it's a very powerful system that orchestrates huge numbers of moving parts in production environments, however at its core, Kubernetes itself is surprisingly uncomplicated. This post is an exposition of my exploration of the components comprising a Kubernetes cluster, and seeing how they can fit together in various ways.
Here at SpiderOak, we have an interest in proliferating Kubernetes clusters. This is perhaps slightly unconventional in itself; anecdotally it seems that most users of Kubernetes either maintain one or a small handful of clusters to run their workloads, or are a provider of Kubernetes itself and run many multi-tenant clusters for their customers. The latter group tends to derive its business value from spinning up clusters en-masse, hence the details of such systems are not widely published. SpiderOak maintains a large number of servers running the storage backend for our data backup product, and is in the process of migrating this expansive network to Kubernetes. Rather than maintaining the entire network as a single "pet" Kubernetes cluster, while treating individual Nodes as a commodity, we are of the mind that it is safer to treat Kubernetes itself as a commodity so that we can quickly, easily, and in most cases automatically, recover from cluster-wide failures. To that end, we have been working on a custom Kubernetes controller that runs Kubernetes itself.
Typically, Kubernetes clusters are bootstrapped by a deployment program, eg. kubeadm, bootkube, or similar, however these tools can be clumsy to use in a fully-automated way. Additionally, these tools set up the control plane components (apiserver, scheduler, and controller-manager) to run within bootstrapped clusters in Pods using static manifests on the master nodes. Such self-hosted clusters work well for many purposes, but our goal was to create a simple, automated way to run these components many times using a minimal specification describing each cluster. For these reasons, a Kubernetes controller and a dedicated parent cluster within which to run the control planes of sub-clusters seemed natural.
Development on this controller is well underway, and we can already control the lifecycle of a Kubernetes cluster including secrets and configuration, etcd cluster, and control plane from start to finish using a single Kubernetes API resource in the parent cluster. In order for these managed clusters to actually run workloads, however, we still need a worker node. The Kubernetes component that turns a computer into a node is kubelet, and until today in the development process, we had not actually connected a kubelet to a managed cluster. I decided to take this opportunity to learn more about Docker for Mac.
For some time now, Docker for Mac has included a way to run a single-node Kubernetes cluster out of the box. All of the complexity of running Kubernetes is reduced to a checkbox in Docker for Mac's preferences dialog. This is absoultely wonderful for getting up and running quickly, and is probably the single easiest way to run a Kubernetes cluster locally. I wanted to figure out how this worked, and in particular, I wanted to know if I could hijack the kubelet within Docker for Mac to connect to a cluster managed by the controller described above.
Docker is a containerization platform for Linux-based programs. Mac OS, while loosely sharing ancestry with Linux, is not in fact Linux, so Docker for Mac runs Docker within a lightweight VM. When you enable the Kubernetes cluster in Docker for Mac's preferences, it runs kubeadm within this VM to bootstrap the cluster. After a few minutes, you'll have etcd, apiserver, scheduler, controller-manager, kubelet, and perhaps a few other things all running quietly within the confines of this VM. The Kubernetes API is exposed to the Mac, and you can then use kubectl against it like normal. Just like plain Docker workloads, Kubernetes workloads are run within the VM as well. Normally, users don't need to really care about this VM or what's running inside it. They just consume the Docker or Kubernetes APIs and work with their containers like they would directly on a Linux machine. But in order for me to interact with the Kubernetes components directly, kubelet in this case, I would have to break into this VM somehow.
I found an excellent article describing the Docker for Mac VM by Ajeet Singh Raina. Within the VM, there are a number of service containers running. Docker itself and the Kubernetes components all run in the "docker" container. From there, I was able to see the actual processes running, inspect their command lines, etc. When Kubernetes starts up, the kubelet command is run by a wrapper script at /usr/bin/kubelet.sh. Using the kubelet command within the wrapper script as a base, I was able to construct a kubelet command line that would connect to my managed control plane. Conveniently, the shared directories (specified in Docker for Mac's preferences) are available in the docker container, so it was trivial to provide the kubeconfig file that kubelet requires in order to make the connection.
To my delight, kubelet came right up, connected to the cluster, and created a Node object for itself (with --register-node=true). Next, I created the cluster's first Deployment. This would be the first time the control plane was really tested. I had created objects in the API before, but without a Node, the components of the control plane were not really exercised. Now that I had a Node, the controller-manager would create a Pod for the deployment, and the scheduler would be able to assign it to execute on the Node. Et voilà! Kubelet created a container for the Pod, and my unlikely cluster actually did something.
At this point, the cluster still can't do much, because there is no networking plugin, kube-dns, or kube-proxy yet. Those are the next pieces to be implemented in our Kubernetes controller. But being able to actually run a container was our first big milestone. Exploring the Docker for Mac internals, running the Kubernetes control plane within another Kubernetes cluster, and connecting kubelet running on my Mac laptop to it, has proven to be an enlightening endeavor, and I feel it's pretty safe to say that the cluster I've created here is special and unique. Although our stated goal is to turn Kubernetes clusters into a commodity, whose lifecycles will be unremarkable, there's always something remarkable about the first time it works.
