The hypervisor dilemma

TrueNAS General
1

So I thought would get a general opinion to see what others say, I’m currently on the fence because I haven’t had time to test TrueNAS Scale/CE VM’s/Containers/LXC

My current situation is I have a TrueNAS Core server that I’m planning to upgrade to TrueNAS CE later on when I get the time. But right now its a low priority compared to my hypervisor machines I’m running.

Currently I am running 2 Proxmox VE’s with ZFS as its base, one on a old supermicro atom board with 8gig of ecc ram and another on a free old Dell Xeon machine with 16gig of regular ram.

I’ve recently bought some new components to upgrade the old atom machine so I can consolidate it all into one hypervisor machine:

  • ASRock - B650D4U-2L2T/BCM
  • AMD Ryzen 5 9600X
  • 2x Micron 32GB (1x32GB), PC5-38400 (4800MHz) ECC DDR5 UDIMM

Currently amongst the hypervisors, I’m running on them the following VM’s/LXC:

  • VM: pfSense
  • VM: home assistant OS
  • LCX: docker running a few containers.
  • I’m planning on implementing frigate NVR outside of home assistant OS as well and using it for home security, currently just trialing it under HAOS.

These are the ones that will work all the time even under UPS backup if the power goes out. So will be of the highest uptime (obviously not enterprise uptime) but for home use high enough.

The main TrueNAS server with all the spinning rust will automatically shutdown as soon as there is a power outage due to its power consumption instead of letting it draining the UPS.

The plan is to have the data on the high up time hypervisor machine backed up to the main TrueNAS server as well when it can.

The thing I don’t like about Proxmox is the ZFS dataset management or the lack of via the web UI. I get it, its purely a hypervisor but still hypervisors store data and having better data management would be a good thing.

I like the way TrueNAS does it, whilst I’m not opposed to doing dataset management via the command line in proxmox, I just would prefer a web interface due to the lack of time in my life.

But I’m wondering what peoples take would be on using TrueNAS CE to be the hypervisor vs Proxmox ?

For the time being I can stick to Proxmox but with the upcoming TrueNAS CE 25.10 would it be a worthy contender to be a hypervisor over Proxmox with its better NAS/ZFS goodness ?

Does any of IXSystem enterprise clients actually use the VM’s/Containers on TrueNAS I wonder ?

2

Its what I do.

Including running a pfSense VM.

The major issue is that you lose the pfSense VM while you update the TrueNAS, which perhaps happens more often than you would update Proxmox.

Would be nice to have a dual TrueNAS setup with HA pfsense VMs.

3

I haven’t tried Fangtooth at all yet, and my experience is with Electric Eel so take that as a grain of salt before reading the rest of my response. I tried running electric eel bare metal as my main OS on my server. The lack of hypervisor features compared to Proxmox made that a no go for me. I ended up virtualizing TrueNAS on Proxmox using a passed through HBA. Works great. In your case, I would personally run Proxmox on the new hardware, and create a dedicated storage network direct to the TrueNAS box. I keep very little data on Proxmox. I use NFS shares and the NFS driver in Docker, and all of my data is stored directly onto TrueNAS. The only backups I do from Proxmox are to backup my VMs and CTs, one copy to my synology and one copy to my TrueNAS. My VMs are all very small, 32GB or less, so I really wouldn’t get much benefit from PBS or Deduplication. My VMs take up around 1-2 GB each after compression.

In your case you could create a data storage pool on Proxmox in a similar manor and periodically send snapshots to TrueNAS. I don’t shut down my spinning rust in a power outage so that may complicate things. I have my UPS backed up by a Ecoflow Delta 2 powerstation that gives my entire homelab about 4 hours of run time in the event of a power outage. Usually I will have the backup generator running long before the 4 hours is over.

BTW, my main server, even with TrueNAS virtualized and all my workloads running idles at 40 watts and almost never exceeds 75 watts. My entire homelab draws about 180 watts. So that plays a factor as well.

4

Would it make sense to get one of the lower power firewall appliances and run a hypervisor on it? Or get a lower power thin client and put a network card in it to use as a hypervisor?

The lower power draw would allow your UPS to last longer, though storage on these devices is usually pretty limited. Things like HP T740 and T755 (used) come to mind, but they only have an m.2 SATA and an m.2 NVMe socket, you’d need to decide how much networking you want before moving on. Fill the PCIe 3x8 socket with a low profile network card, or just use the wifi socket to house an a+e Intel i226-v card. Intel dual x520 cards work, I have them in mine, Intel quad i350 work, also have one in mine. Looking at stepping up to a quad x710 card in some of my T740 to use the PCIe lanes more effectively. Still working on this last upgrade to decide if it matters for my lab.

5

@Stux - Considering moving from ESXi multi-box solution to a single box solution (3 DL380 G9 boxes draw too much power…). I have considered Proxmox but compared to ESXi it is a PITA (had a lot of issues with LACP for no good reason). So now, I am considering sticking with ESXi and collapsing down to a 1 box solution OR installing TrueNAS on bare metal and then virtualizing as required. My use case is pfSense, a couple windows VMs, and a few Ubuntu VMs - one of which being a docker host which I may migrate to Kubernetes on TrueNAS anyway.

Long story short - looking for some guidance and advice. Sorry to thread jack, I can start another thread but this seemed like a similar line of thought to the OP.

Look forward to hearing back from some high HP experts.

Cheers,

6

There’s no more kubernetes on scale. With 24.10 scale switched to docker

7

Hmm, I thought Fangtooth (25) ran K3s under the hood - dockers will run, but the base engine is Kubernetes. Am I wrong on this?

8

Yep, good old plain docker under the hood

9

Sweet - good to know about the Docker thinger. I used to run a single box ESXi solution for years and it was great. Had access to old work servers so I want full idiot but it is such overkill.

Just making plans. May the 24.x for a test drive on a box - see how it rolls. Then when 25 gets virtualization under control, step forward.

Curious to see how bad this fights me…

10

As i’ve mentioned before, it works using pfSense in a VM in TrueNAS, but it may get old losing your router when you reboot your NAS (say for an update)

1 Like
11

I used to have to deal with that when I ran a 1 box ESXi system with FreeNAS as a VM. I do not update TrueNAS very often - not a huge deal to lose the router from time to time…

Just taking TrueNAS for a test drive on a server now.

12

@Stux - have you ever converted ESXi machines to KVM? Does that process work ok?

13

qemu image convert should work. I’ve used it with HyperV images.

You can write directly to zvols.

14

I think for a home/home lab this isn’t as big a deal, you can schedule to do this of a night when the family is asleep as long as you don’t mind staying up. Good thing about TrueNAS is you have the ability to roll back to the previous version if the update goes pear shape, something that Proxmox doesn’t have if you run its upgrade process…

Proxmox you still need to reboot if there is a kernel update…

Proxmox does allow you to use the main boot zfs pool for everything as well, which explains why there is no rollback process compared to TrueNAS.

Also TrueNAS vs Proxmox for ZFS backend storage, TrueNAS still the superior being…

I’ve found using Proxmox easy to setup and use though… Most likely will migrate to TrueNAS CE later on when I find time to do the R&D.

Ahhh time, such an elusive creature… :joy:

15

Ho-lee-shiz is 25.04.01 TERRIBLE for virtualization. Seriously - no way to attach an ISO without the CLI? What in the…

Coming from a VMware shop, all I have to say about Proxmox, TrueNAS, KVM, XCP-ng etc. is that they pale in comparison. Arguably, I am asking TrueNAS to do more than anyone reasonably should - but 25.04.01 is a step in the wrong direction (right into the woodchipper really). But at least it will do a vTPM… So there is that.

Here is hoping that 25.04.02 fixes this disaster next month and restores some of KVM’s glory. All this is making me reconsider my choices and pushing me to stay with ESXi as the hypervisor and then virtualize TrueNAS.

Grumble grumble.

1 Like
16

25.04.1 is arguably the worst version you could have tested. The new incus backend is experimental. Wait for the .2 release in July which brings back the old vm backend from the older versions (24.10 and older)

1 Like
17

In the wake of this Broadcom mess I’ll bet Red Hat, Oracle, Ubuntu, and everyone else selling a virtualization “stack” wishes they’d spent more time getting the final 10-15% of their features, functionality, and usability ironed-out and fully polished. Companies are positively desperate for a viable alternative to VMware given [publicly traded] Broadcom acting like a private equity wrecking ball.

Instead they’re running into a wall of jank getting kvm hosts connected to shared/pooled storage and using it in the manner of VMFS, correct multipathing, trying to mimic the entire functionality of a distributed vSwitch without suffering brain damage, having live migration of guests operate as smooth and trouble-free as vMotion, fighting with virtio guest tools installations that demand constant oversight and hand-holding, establishing a VirtualCenter-like single pane of glass, dealing with the ramifications of obscure libvirt XML directives so that guests perform in the same league, ad nauseam.

kvm/qemu/libvirt et al are pretty much at feature parity with the core of vSphere but there’s an awful lot of rough edges.

I’m sitting here with a Zen 4 desktop running an Arch derivative and in spite of twisting knobs left and right I can’t even get the kvm/qemu stack to outperform VMware Workstation when running a Windows 11 guest – this wasn’t the case 3-4 years ago with my older hardware.

Nutanix seems to be the only company with a well-oiled kvm machine but then you get to deal with their stubborn insistence on hyperconverged storage. And a price tag not much better than VMware.

I hope I won’t need to un-retire because I don’t like where any of this is going.

2 Likes
18

I am somewhat in the same boat as OP, as I am considering moving the most mission critical proxmox box to truenas and use their new virtualization capabilities.

I can’t find the thread right now but among this Incus-induced fracas, I think one of the TrueNAS staff said that they do not recommend virtualization to their enterprise clients yet. So there’s that.

I think they are in a better spot now with Docker-based app, but for Virtualization, I get the feeling that it’s still experimentation phase.
It’s not just the running of VMs that is a problem, I think they are pretty behind in regards to virtual networking that often comes with it.

While it’s true that in TrueNAS you can run containers and VM, like in Proxmox, I think the way the whole UX is still storage-centric, whereas Proxmox the UX is centered on infrastructure-building. The infrastructure being pooling the resources of a cluster of servers and make those resources (compute, storage, networking) available to run and manage VMs.

This is not necessarily a bad thing. I can see this being more opinionated as advantage if the end goal is to built storage device that also runs a handful of storage services. However, I don’t see an enterprise running their VMs and services infrastructure over a cluster of truenas boxes.

19

IMO, TrueNAS has never been a serious competitor to Proxmox in terms of virtualization–if you want to do any non-trivial virtualization, TrueNAS just isn’t up to the task. It never has been, and I frankly doubt it ever will be. Run a decent hypervisor–Proxmox, xcp-ng, free ESXi if you still trust Broadcom–and virtualize TrueNAS if you also need a NAS (and don’t want to put it on its own hardware).

1 Like
20

I’ve run a Windows 10 vm, Mac OSX VM, ubuntu VM, debian vm, truenas vm, and HAOS all just fine using Scale. Ymmv.

What I will not do until it is all fleshed out in run any of those under Incus! That’s why I pay Truenas the big bucks ($0, lol) so they can flesh it out for me.