After setting up lots and lots of virtual machines and container-based workloads on my NUC cluster, I have made many changes to the underlying infrastructure since my last post.
Using the community-maintained Proxmox docker machine driver turned out to be pretty finnicky and I would frequently end up throwing away nodes that never got started properly. Because of this frustration and the fact that most of my workloads are container-based rather than VM-based, I decided to try out Harvester, Rancher’s new virtualization platform based on KubeVirt. The Harvester nodes themselves run k8s on metal, and then VMs are provisioned in KubeVirt within that k8s cluster.
Whereas Proxmox utilized Ceph for multi-node shared storage, Harvester defaults to Rancher’s own Longhorn, a distributed block-storage solution that automatically takes care of placing replicas across my nodes.
Longhorn is much simpler to manage, and comes with a nice dashboard showing scheduled storage vs used storage, as well as a listing of all volumes in the cluster and their status:
Additionally, it is very easy to configure Longhorn to backup volumes to S3-compatible storage. I back up my important PVs on a recurring schedule to Backblaze B2.
Proxmox templates had basic support for
cloud-init, but were totally incomplete compared to the advanced cloud-init configuration that Harvester allows.
For example, I can create a
cloud-init template for setting up Docker in an Ubuntu VM, and it looks like this:
#cloud-config groups: - docker: - ubuntu apt: sources: docker.list: source: deb [arch=amd64] https://download.docker.com/linux/ubuntu $RELEASE stable keyid: 9DC858229FC7DD38854AE2D88D81803C0EBFCD88 package_update: true packages: - docker-ce - docker-ce-cli
cloud-init yaml can be added to a “Cloud Config Template” in Harvester, and then just selected at VM creation to end up with a complete installation of Docker in the VM without any extra steps.
It is also possible to add SSH keys directly to Harvester, and then choose those during VM creation to have them added to the default user. This also works via
cloud-init but is totally transparent – if all you care about is setting SSH keys, you don’t have to worry about learning the
cloud-init syntax whatsoever, as Harvester handles setting up the relevant YAML to inject the keys.
Rancher has built-in support for “cluster management”, as described in the initial NUC cluster article. However, in that article, I had to use a community-maintained docker machine driver for Proxmox in order to provision k8s nodes automatically, and it was a bit flakey. Now that I have switched to Harvester, there’s first-class support for provisioning k8s clusters in Harvester VMs right from my bootstrap node.
The cluster management page makes setting up k8s clusters very simple, and has a nice UI for setting up pools of workers and control nodes:
Harvester is a brand-new product and is definitely a bit rough around the edges. I would not recommend it (yet!) to anyone without a good amount of Kubernetes experience. I have had to debug Harvester internals quite a few times but have not yet hit an issue I was not able to solve on my own or via referencing an issue in their GitHub issue tracker.
The bootstrap cluster as setup with
caddy was alright, but did not feel very stable and was more complicated than I wanted it to be.
In order to reduce complexity here, I abandoned the original VM in favor of a plain Ubuntu 20.04 VM running RKE2 (Rancher’s Kubernetes distribution, which can be set up with a one-liner) and just setting up the SSL certificate using
cert-manager inside of the cluster.
The relevant k8s state for that certificate, along with a LetsEncrypt issuer, can be found in my
nuc-cluster repo, under
The k8s state for my main workload cluster (comprised of VMs running in Harvester, managed by Rancher’s cluster management), also lives in the
nuc-cluster repo, under
helmfile allows me to declaratively state (in YAML files, of course) Helm releases that should be applied to my cluster.
For example, the
helmfile.yaml for my deployment of Miniflux (my feed reader of choice) looks like this:
repositories: - name: k8s-at-home url: https://k8s-at-home.com/charts releases: - name: miniflux namespace: miniflux chart: k8s-at-home/miniflux version: 4.5.0 installed: true values: - values.yaml secrets: - secrets.yaml
This file tells the
helmfile tool which repository to grab the Helm chart from, as well as its name and version, and a path to values files to set the right options in the chart when it is being deployed.
I’m happy with Harvester right now and the direction it is headed. I would like to check out TrueNAS Scale once it’s stable enough to try, but for now I couldn’t even get it to boot in a VM…
Let me know if you have any thoughts or questions by commenting below.