For a NAS newbie, my question:
Why are people talking about installing TrueNAS on a proxmox VM all the time?
I do have a brand new/performant Server I could install TrueNAS directly to an SSD.
What is recommended and when with proxmox, why?
Thank you for any hints
Hey @SamTe
Generally speaking, users install a “Virtual TrueNAS” when they are looking to use their existing hypervisor (ProxMox, XCP-NG, VMware) for the compute portion, and leverage the NAS/SAN functionality of TrueNAS only. They install TrueNAS as a VM so that they can get all of the ZFS functionality and management, and present storage from TrueNAS in a “loopback” style to the host again.
There are some caveats to it, but a properly-configured virtual setup can be very close to the performance of a dedicated unit:
Of course, the other solution is to install TrueNAS bare-metal, and use its built-in virtualization and containerization functions (which will be made even more capable in the future) to run your workloads.
Different strokes for different folks. 
I will speak to my case. I suspect its fairly common.
Proxmox and TrueNAS solve adjacent but not 100% overlapping problems. I have limited room for compute infrastructure and so installing Proxmox and virtualizing TrueNAS (SCALE) allows me to solve a couple of different problems I have with home infra, lab, and offsite work infra.
It’s a hardware-efficient approach to be able to work with a wider set of technologies than either product provides on its own.
That said, with k3s officially on the chopping block I will be scaling back on the App/VM/workload side of TrueNAS. Disappointing because I was really hoping to be able to leverage TrueNAS clustering for availability of both storage and workload.
I thought about writing provisioning code to deploy Nomad onto a TrueNAS cluster but since Nomad is no longer Open Source…
Perhaps its time for a new edge-based scheduler with is neither k3s nor Nomad…
Thank you HoneyBadger. So for me - bare metal and fresh install should be fine for my needs.
A better approach would be to install your own k3s instance in a sandbox on TrueNAS surely?
You effectively get baremetal storage and compute inside the TrueNAS host, but in a self-contained k3s instance.
I’ve been researching this but there’s doesn’t appear to be a way to easily migrate the storage volumes (k8s parlance: PhysicalVolumes + PhysicalVolumeClaims) between SCALE cluster nodes to allow the workloads to move during a SCALE node outage. Whether the k8s is provided by the SCALE platform or self-managed.
That’s the main thing.
But then there are downstream issues with the lack of effective federation between k8s clusters for edge-but-decentralized scenarios as well. Nomad handles that stuff much better but, y, !OSS so…
But we are getting into the weeds relative to the OPs topic so.
TrueCharts doesn’t seem to think so, FWIW. My guess is that they don’t trust that’s going to be a stable platform, and after iX yanking the rug out from under them, I can’t say that I blame them.
Well, their preferred os, Talos, is not systemd based, so I believe is not an nspawn eligible distribution.
But there are other k8s focused distros that are
But, why would the TC guys know about sandboxes? Maybe they should investigate it as an option more thoroughly.
Without it you’re stuck using NFS or block storage, and memory and cpu pre-allocation, and no gpu sharing.
You can tar up the rootfs and push it into a vm if you want.
Fundamentally, the issue is the same as Home Assistant OS.
That could well be another factor–but I expect iX’ habit of making drastic breaking changes (this isn’t the first time, and I’m sure it won’t be the last) is weightier to them at this point, such that they’re quite motivated to come up with a solution that depends as little as possible on anything under iX’ control. I know I would be, were I in their shoes.
Because at this point, what assurance do they (or you, or I) have that sandboxes are going to stick around? iX are growing, IMO[1], less and less trustworthy[2]. We have no idea if, much less when, sandboxes are going to be a first-class feature, nor how many changes they’ll go through along the way. We don’t even have any real assurance that they aren’t going to just go away. So while I know iX’ official recommendation is to install k3s in a sandbox, I don’t know that I’d choose that solution if I wanted a platform that was going to be stable over time.
Why wouldn’t they? There’s certainly been plenty of buzz about them in Dragonfish (in no small part thanks to you), and there was plenty of discussion, even though completely unsupported, about using them in Bluefin and Cobia as well. I’ll leave discussion of the technical pros and cons to others more familiar with them.
I don’t know what their plans are in any detail other than that they mentioned something VM-based. Given that they were blind-sided by this news just like the rest of us, they may change their mind about that, but that’s what I’m understanding at this point. They’re planning to have a beta out 1 July, so we’ll know more then.
