TrueNAS Scale as a generalized container server (Proxmox replacement)?

TrueNAS General
, , ,
1

For a home server environment (“homelab”), I currently have two main servers: TrueNAS doing strictly storage, and a Proxmox server for apps. This setup has been working great for several years now. Prior to that I had a single Linux server doing everything, managed all from the CLI. I moved to the two-system model so I could have a more “appliance”-type experience, and not have to spend so much time with system care and feeding.

Note that my Proxmox system isn’t running any VMs, only LXC containers.

I’ve recently discovered Jailmaker for TrueNAS Scale, and it feels like a game-changer for me in that I’m thinking I can once again have a single server that does it all. (When I was previously doing it all by hand, I was actually using systemd-nspawn to partition (“containerize”) all the services/apps.)

So my main question: is there any reason I might not want to ditch Proxmox in favor of TrueNAS Scale as a combined storage+container server? I understand there is some ideological argument here in that TrueNAS is meant to be more of an “appliance”, and using nspawn jails arguably breaks away from that model.

I know the next TNS release (Electric Eel) is dropping k8s in favor of Docker. But if I understand correctly, these Docker containers (at least initially) won’t be able to have a virtual interface and DHCP-assigned IP (that is distinct to the TNS host). That’s a requirement for me, so I’ll likely stick with Jailmaker/nspawn containers for the added flexibility (even for Docker-native apps (e.g. Frigate), I’ll likely use an nspawn jail).

I see also that jlmkr.py will soon be in need of a new maintainer. That doesn’t scare me too much, as I think I could self-maintain that script for myself.

Having thought about this a bit, my biggest concern is that iX decides to kill the ability to do nspawn jails entirely (i.e. making jlmkr.py (or its equivalent) useless). IOW, what if they deem Docker to be sufficient, and better aligned to the appliance-model of TNS? Or what if there are too many stability issues from people doing wacky things in an nspawn jail?

(Another option is to virtualize TrueNAS under Proxmox; I really don’t want to go down this road. In my case, I feel storage is the most core/critical application, so it should be closest to the hardware. I view containers/apps essentially as a “function” of the storage, so they can exist one or more layers removed from the hardware.)

Another argument against this is the risk of system stability - if I do something unwise in an nspawn jail, it could in theory compromise the host. This is a risk I’m pretty comfortable with - it was the model I used for years before the current system split model; I have good backups; and this system isn’t critical to my life or livelihood.

Am I missing anything? Any thing else I should consider before consolidating these systems?

2

No :wink:

Jailmaker works fine in Electric Eel.

There is some possibility that TrueNAS may adopt LXC/Incus in the future, and Jailmaker already actually uses Lxc base images.

BTW, I tested the docker functionality in Eel from a Jail users perspective in this video

There is still a lot of reasons to use a Jail if it suits you, and with the ability to run simple VMs as well in TrueNAS, not a lot of reasons to run proxmox + TrueNAS.

3

We have yet to see the UI for EE Docker functionality, but it is on the way.

When it arrives, then there will need to be a comparison video of the usability of EE Docker vs. Jailmaker - and probably about all the different ways you can use the new functionality (because it sounds like there will be the standard EE functionality and then a whole bunch of other ways to run custom apps in EE).

Being a cautious user with simple requirements for running apps, I decided to wait for EE to get to a stable and recommended (for cautious users) release before switching and to just wait and be patient rather than do roll-your-own Jails in Dragonfish.

4

My answer has been (without reference to 24.10, which doesn’t exist yet) that Proxmox has a much better and more comprehensive UI for the virtualization side of things, backups, and the like. The other major difference is that Proxmmox still supports clustering. I’ve considered virtualizing my NAS, but never discarding Proxmox in favor of TrueNAS.

5

One thing I discovered in further researching this: one of the containers I run currently under Proxmox is Frigate NVR, which uses the USB Coral device for object detection. The cursory research I did suggests this would be somewhat kludgy in TrueNAS because (at least currently) TNS currently does not natively support the Coral.AI devices. Similar to using a GPU within a lightweight container (e.g. nspawn, LXC, Docker), the first step is for the host to support the device. And while it seems that’s doable in TNS, it’s not doable in a clean, supported way (i.e. you have to manually install the driver, then remember to re-install after any upgrades, since you’re “coloring outside the lines”).

My goal here is to reduce my overall homelab power consumption, and consolidating two systems into one seems like the obvious way to do that, especially since my hardware is complete overkill.

I also researched virtualizing TNS under Proxmox, and it appears that, to do it right, the entire disk controller should be passed through the host to the guest. While you can virtualize storage on a per-disk basis, the result is TrueNAS will see virtualized disks, rather than the raw hardware. Besides a potential performance penalty, not having direct hardware access means no SMART access. Furthermore, there are added complications when a drive dies (or is on the verge of failing). In short, when disks are virtualized, you give up a lot of what makes TNS great.

However, passing through the entire controller to TNS avoids these complications. Furthermore, in the event of complete host (i.e. Proxmox) failure or corruption, you could e.g. boot TNS from a USB thumb drive, restore config from a backup, and have all your data at hand.

But in my case, my SATA controller hangs off the chipset, meaning it’s in the same IOMMU group as a bunch of other stuff (NVME boot drive, USB, IPMI, etc). So if I passed that through to TNS, those useful things would be invisible to the Proxmox host. Adding a dedicated HBA isn’t an option, because I’m out of PCIe slots!

So I’m sticking with separate systems for now.

Also, I’m building a TrueNAS system for a friend that will also run Plex. I’m using the Electric Eel beta for this, so it gives me a chance to play with 24.10 to see what’s coming down the pipeline.

6

This is/was true for PCIe devices but it is not true for USB Corals. The Frigate app in Scale handles it natively and easily. I have been running it (not the Truecharts version) since June, continuously, and it was idiot-proof to set up using the USB Coral.
I’m holding Frigate at 0.13 even though 0.14 is available, for various unrelated reasons and also because of GPU related matters which I am hoping are different when Electric Eel is released.

(I don’t know anything about other aspects of TN vs ProxMox vs Unraid vs other things: this is simply my experience with USB Coral and Frigate and TrueNAS and I hope it’s helpful or at least interesting).

1 Like