From TrueNAS to Kubernetes (Part 1): why, and picking the stack

From TrueNAS to Kubernetes (Part 1): why, and picking the stack

This is part 1 of a four-part series. The index post is here. Parts on Flux + SOPS, the Tailscale saga, and finally Grafana will follow.

Why bother

I’d been running my homelab on Docker Compose for about a year. It was fine. It was, in fact, more than fine: the stacks were in git, Dockge redeployed them when I pushed, and the box had been up for months at a time. There was no thing I was trying to do that the existing setup couldn’t do.

Which is roughly the worst possible time to add Kubernetes. The honest version is:

• I wanted to learn it properly. Most of my exposure to k8s had been at work, on clusters someone else had built, where my understanding stopped at kubectl apply. I wanted the bit underneath that.
• I wanted dynamic storage. Docker volumes are fine for one host; they’re not great when you start wanting “this service should be schedulable to either node, with persistent state”. For that you need a CSI driver, and CSI lives in Kubernetes-land.
• I wanted somewhere to run things that aren’t a good fit for TrueNAS apps. Home Assistant, monitoring, the odd one-off — they don’t all want to live next to my photos and TV shows.

It is also, embarrassingly, a hobby. I am told this is allowed.

Diagram placeholder: a “before” / “after” sketch. Before: one TrueNAS box with a stack of Docker containers on top. After: same TrueNAS box, plus a separate Proxmox host running three Talos VMs, with arrows showing NFS traffic over Tailscale between them.

Why a separate Proxmox host

The first thing I had to decide was where the cluster would live. There were three options:

Option	Pros	Cons
Apps on TrueNAS Scale (Kubernetes-ish under the hood)	No new hardware, already there	Limited control, weird abstraction, I’d be back to “kubectl is hidden from me”
Bare-metal Talos on a Mini PC	One less abstraction layer, very fast	One physical box = one node, and I want to play with multi-node
Proxmox on a Mini PC, Talos VMs on top	Three VMs on one Mini PC, snapshotting, easy node teardown	Extra hypervisor layer, slightly more to maintain

I went with Proxmox. The deciding factor was that I knew I’d want to nuke and recreate the cluster several times while learning, and being able to snapshot a known-good VM and roll back to it is so much faster than re-imaging hardware. The extra hypervisor layer is real overhead, but on a quad-core Mini PC running mostly quiet services it doesn’t matter.

The hardware itself is an HP ProDesk 400 G5 Mini — second-hand, quiet, sips power. It sits next to the off-site TrueNAS server at a family member’s house, on the same LAN.

Photo placeholder: the Proxmox host, lid off, showing the M.2 SSD it boots from and the SATA SSD that holds the VM disks.

Why Talos

The default thing to do is install Ubuntu / Debian / Rocky and then layer k3s or RKE2 on top. I tried that first. It worked. I then spent an evening figuring out which of the various things on the host were and weren’t required for the cluster, which gradually turned into the realisation that I’d spent the evening doing manual host configuration on a thing where the whole point was GitOps.

Talos Linux sells a different model: there is no host to configure. There’s no SSH. There’s no package manager. There’s no shell. The OS is a single immutable image and everything you can change about it is a YAML patch you apply via talosctl. The first time you read this it sounds insane. The second time you read it, after spending an evening locked out of your own Debian box because you forgot which UFW rule you’d written, it sounds like the best idea anyone’s ever had.

The other thing it gets you is reproducibility. The exact set of system extensions baked into your installer image is determined by a single YAML file (schematic.yaml) that you POST to the Sidero Image Factory, and you get back a deterministic SHA. Mine currently looks like this:

customization:
  systemExtensions:
    officialExtensions:
      - siderolabs/iscsi-tools
      - siderolabs/qemu-guest-agent
      - siderolabs/tailscale
      - siderolabs/util-linux-tools

The same file always produces the same image. If I want to add an extension, I edit the file, regenerate the schematic, and roll out a new installer image to each node. The full procedure is in docs/talos-extensions-rollout.md in the repo. Future-me will need that runbook again.

I’m not (yet) using Omni, the SaaS / self-hostable Talos management plane. For a three-node cluster the raw talosctl workflow is fine, and there’s something to be said for staying close to the underlying primitives while you’re still learning.

The first cluster came up. The second one came up correctly.

I won’t pretend the bootstrap was smooth. The first cluster I brought up worked in the sense that kubectl get nodes returned three Ready nodes; it didn’t work in the sense that I’d made a series of small choices I later regretted:

• I’d given the VMs DHCP addresses rather than static. Talos doesn’t mind, but I do — every reboot reshuffled which IP went where, and my talosconfig would drift out of date.
• I’d used a generated cluster name that included a UUID. Cute, unreadable in kubectl output.
• I’d put the kubelet and the etcd data on the VM’s root disk, which meant snapshotting the VM also snapshotted the cluster state. Not what you want.

I wiped it. The second time round I:

• Reserved static IPs for the three VMs in my router (192.168.1.31 for the control plane, .32 and .33 for the workers).
• Used a sensible cluster name (homelab, revolutionary).
• Gave each VM a separate data disk for /var/lib/etcd and /var/lib/kubelet.
• Saved my controlplane.yaml, worker.yaml, and talosconfig into proxmox/ in the homelab repo — gitignored, because they contain PKI material and shouldn’t ever be committed, but at least kept together with the rest of the config and backed up to my password manager.

The repo’s .gitignore has proxmox/ listed precisely because I’d rather have a deliberate “this is local-only” pattern than hope I remember not to git add -A one day.

Screenshot placeholder: the Proxmox web UI showing the three Talos VMs (talos-cp1, talos-worker1, talos-worker2) up and running, with their resource graphs.

What you have at the end of Part 1

Three Talos VMs, a healthy cluster, and a talosconfig / kubeconfig pair you can use to talk to it. No workloads yet, no GitOps, no storage, no monitoring. Just a cluster.

Which feels like the easy bit, in retrospect. The “real” work starts when you try to actually run something on it — and want that something to be defined in git rather than imperatively kubectl apply-ed. That’s Part 2.