Some members of the TrueNAS community have expressed interest in running their own custom Kubernetes instance, as opposed to the iX-provided installation. Sometimes this is to integrate with their own management system, use a different runtime, or even experiment with a “cluster-in-a-box” style configuration; but whatever the reason, this small tutorial will provide a brief overview of the process of installing Kubernetes within a TrueNAS SCALE “Sandbox” using the popular Jailmaker script.
Prerequisites
Because this tutorial requires the “Sandbox” functionality of the Jailmaker script, we recommend using TrueNAS SCALE 24.04 for the fastest and most efficient setup process. If you’re not already running 24.04, you can download the upgrade file or ISO, or use the System → Upgrade functionality in your current TrueNAS installation.
If you haven’t already done so, you’ll need to enable Sandboxes (systemd-nspawn) by following documentation on the TrueNAS SCALE Documentation Hub. Ensure that you run jlmkr install if you’re using jlmkr v1.5.0 or set up the optional aliases for jlmkr v2.0.0 to allow you to enter the commands without having to specify the full path to the jlmkr script every time, and that your post-init script to launch your jails is in place.
Next, create a new network bridge by following the documentation on the TrueNAS SCALE Documentation Hub. This is to prevent conflicts with the existing k3s installation.
Creating your Kubernetes Sandbox
Using the Jailmaker script, create a new sandbox for Kubernetes. We suggest you use the same Debian “Bookworm” base image as TrueNAS SCALE. There are several additional flags required in order to allow proper functionality during the creation.
The command line below will create and start a sandbox named “k3s” using the network bridge br0, and using the existing DNS servers in your SCALE installation.
jlmkr create --start --distro=debian --release=bookworm k3s --network-bridge=br0 --resolv-conf=bind-host "--system-call-filter='add_key keyctl bpf'" --bind=/dev/kmsg
The --system-call-filter line allows the use of keyrings and cgroups within the container, and the --bind line allows the container access to the circular message buffer device for output. (Note the use of single-quotes around the items in the --system-call-filter line, and the double-quotes around the whole parameter.)
Set a Static IP
Once the new sandbox is created, access its shell with jlmkr shell k3s. Configure a static IP address by editing the /etc/systemd/network/80-container-host0.network file to set DHCP=no, providing an Address and an upstream Gateway for network connectivity.
Restart the container networking with systemctl restart systemd-networkd and check that your IP address has been updated with ifconfig - if it hasn’t, you can exit the shell and restart the sandbox with jlmkr restart k3s before continuing.
Update the Container and Install Other Packages
Run both apt update and apt upgrade commands to confirm that the container is up-to-date, then install the curl package with apt install curl - you can also install other desired packages eg: nano for editing.
Install your Desired Kubernetes Solution
For this example, we’ll install k3s, by using the command:
curl -sfL https://get.k3s.io | sh -
We’ve chosen k3s for this example as it includes CoreDNS, Traefik, a Network Policy controller and the ServiceLB load balancer. Note that the default k3s “internal cluster” network space is 10.42.0.0/16 and 10.43.0.0/16 - if these addresses are already in use on your network, you’ll want to change them using some additional command-line options as described in the k3s documentation.
Check your Kubernetes installation
Use the kubectl cluster-info and kubectl get nodes commands to verify that your k3s installation is working correctly. From here, you can manage your workloads from within the shell via kubectl, or from the static IP configured above and the default TCP port of 6443.
