So I have been using TrueNAS since FreeNAS V8. Been a journey. Started out thick, went to a 1-box on top of free ESXi, the went back to a 3 box VMware cluster solution (2 hosts, thick CORE).
I have been putting of switching but I knew it was inevitable based on Broadcom being the worst. So I finally went back to a 1-box Proxmox. Started with migrating CORE 13.0u8 and got it stable. Saved the config and imported it into SCALE 24.10.4. Not sure if this was the perfect approach, but I really did not want to have to re-do all my base configs…
SCALE has come a long way since I last tried it out on 22 and 23. Good work TrueNAS team. Went through some shell scripts and made them work on Linux - not too bad. Huge shout out to @joeschmuck for the Multi_Report. What a champion. Need to sort out the Supermicro fan control some day, but I just set the fans to a decent RPM that seems to never really change, so good enough is good enough.
My thoughts: TrueNAS on Proxmox is a great home lab setup. I now store my VMs on the Proxmox ZFS storage and back them all up using Proxmox Backup Server on a VM to a TrueNAS NFS target. Works well and I am confident everything is on good storage (unlike DAS on ESXi…). Starting to leverage some LXCs. Still hosting a lot in a Docker Ubuntu VM. Not sure I will ever leverage the advanced features in TrueNAS with this setup though.
All in all, happy with the solution and really thankful that TrueNAS continues to offer such a ballin free to use NAS+ solution. Great work.
You might already know that I had been running TrueNAS on ESXi since version 9, but I was here when version 8 became TrueNAS. These days I no longer run TrueNAS on ESXi, I run on bare metal, but then again, I’m not running other VMs. I was able to get ESXi 8 (free) with a license, and you are correct about Broadcom. I have never used Proxmox, hopefully I will never need to use another Type 1 Hypervisor.
Thanks for the shout out, it is very appreciated. Things like that get me motivated to continue work on the next major version, because honestly, I’ve been dragging my feet.
I am glad you were able to put SCALE on Proxmox, and yes, SCALE has come a long way. I’m still thinking about updating to the current version very soon. It seems stable and no bugs that might impact my usage. I have outgrown the need to have the most recent version when it is likely to have bugs so I wait for a few minor version changes first, but one time I enjoyed identifying any problems I could find just to feedback that to the developers and make the product better.
Hope that new setup keeps running error free for a long time!
Ahh, V9 days on ESXi - takes me back to this post that I followed to do the same. It has been a journey over the years for sure. First use of ZFS was with OpenIndiana - what a beastie that was.
My ghasts are flabbered as to how much awesomeness is bound up in the multi-report script. And the graphic showing the layout - pretty cool. Any chance it can have 2 chassis? I have my SSDs in the DL380 case and the spinning rust in the SuperMicro case. Not critical but it would be neat to arrange them all.
But again, I tip my cap to you for your effort and for sharing with the community.
Now, we will let SCALE soak for a week or two before I burn the bridge and update the feature flags…
The drive layout is still not a final version of that feature, it works for many but not all. And I do have an update that will fix problems that others have seen. As for multiple cases, as of now, no, but it has been talked about. The current solution is to create a large custom layout where you can place all your drives into it, formatted the way you like it.
Yes, I have seen that ESXi post before, a few times. I personally never used it but I know I have helped several people having troubles and referenced that post. Some thing are easy to read and understand, some things are not especially if you are not familiar with what is going on in the first place.
Back on FreeNAS 8.x and 9.x I was compiling FreeNAS source code with my own modifications, mainly to make MiniDLNA work. I posted a lot back then about how to make it work. Plex was not always the easy thing to do, like it is today. FreeNAS 10, I stopped, but so did everyone else’s computer. What a bad release. FreeNAS 11 eventually came out and I was done compiling code.
I really need to take a hard look at Proxmox. I will when/if I need to run a serious VM.
Thanks - was thinking of just making an 8 column 5 row thinger and using that.
And Coral…. Oh. My. Word. What an experience that was. So happy they recovered to 11 post haste. Yuck.
Proxmox 9.2 seems pretty swell so far - really like that it is ZFS-able. Solid on my HPE DL380 so that is great. So far, it has been reliable and restarted fine after a sudden power failure due to a dusted UPS. Worth giving it a shot if you need a VM hypervisor in your life IMHO. Also really like that you can get under the hood with simple Linux commands and it does not seem to mind too much as the GUI just pulls the configs from text files you can edit. ESXi was less fun, but (and FU, Broadcom) VMware is still so much better than any other hypervisor I have tried. But, all things pass.
So all is still running well. Turns out with a Gen9 HPE DL380 that adding the second riser card in the case causes the fans to run fast. So, I removed 1 HBA and installed the 10GigW LOM card and it runs much quieter. So, the one box solution is going well so far.
One hassle with Proxmox is there seems to be no “easy” way to backup your Proxmox config. Here is hoping my boot M.2 drives stay happy for long time…
At some point, I guess I really should install the new feature flags on the pools - that is the point of no return but I am pretty sure I can make TrueNAS core meet my current needs regardless.
Yes. The main reason that I don’t use Proxmox is that it is not an “appliance”, where you can save 100% of the configuration so you can redeploy and be back up in 15 minutes.
It’s powerful and offers so many extra VM options that TN probably will never get, but without a one file, one click, full config, is a deal breaker for me.
Yeah, they do have the Proxmox Backup Server, but again, that’s an extra machine and I’m sure the recovery process takes longer than the 10 minutes that TN takes.
The alternative or enterprisey way of handling this is ansible/terraform which can stand the system configuration back up in a matter of a minutes, even less so in my experience. Configuration is then stored as “source code” (git/svn/etc), can be templated and then deployed to non-original hosts as well; in-effect restoring to somewhere else in case of hardware failure.
While not out of box button clicky, you can quickly achieve that pretty easily. If you get really adventurous a PXE boot environment that auto restores without any button clicks at all.
For a more crude fashion you can actually backup like, 3 directories I think, can’t remember off the top of my head, which will restore all VMs and host config: VM disk dir, qm configs, and host configs. I’ve done that in the past and it’s pretty easy tbh
I agree it’d be nice to be able to treat a PVE host like TrueNAS in this regard. The problem is that Proxmox don’t lock down their OS the way iX do; you can (and frequently will) install arbitrary other software there. A config backup wouldn’t capture that.
I do run PBS, but it uses TrueNAS storage, and TrueNAS runs on top of Proxmox, so there is a chicken and egg problem…
I am sure a restore would not take that long to stand up Proxmox, TrueNAS and PBS to be able to restore all my VMs (VMs are local to Proxmox, but it is not like VMware where you can just register the .vmx files on the host).
Not ideal, but here is hoping I do not have to rebuild Proxmox any time soon…
I want appliance one-button happiness. But alas, I will cross my fingers and hope that I do not need to re-build Proxmox any time soon…
I may consider trying out TrueNAS as my hypervisor as I now do so little with VMs that cannot be replicated with Applications. But alas, that will be a slow process as I have no cross variable driving throughput…
Really isn’t intended to back up the PVE host, though. It can–the client’s installed in PVE–but you’re completely on your own at the shell to make it work. I think I’d favor ZFS replication as I documented, but that of course depends on your NAS not running under PVE.
Mirroring your PVE boot device, of course, greatly reduces the risk of system loss this way.
Yeah, the nice thing is once you get into PBS, you can restore the VMs with all their settings etc. But chicken and egg to get there.
I run Proxmox from a mirror pair of M.2 drives so here is hoping the last a good, long time. Homelab being down is not the end of the world. In proper production environments, I would do things differently and definitely leverage replication to a repo that I could easily recover from. But my goal was to 1 box this beast to save power - so far look like I am saving about 70w… So it might not have been worth it. May try out the lower powered CPUs in the DL380 as I need 18 core dual sockets like I need a third eye…
Thanks for all the suggestions and feedback - always learn something new on these forums.
You already have answers, here and elsewhere; you just don’t want to listen to them–you want to beat the drum about how TrueNAS is better. And if your only metric is “I can upload a saved config file and restore it to its previous state,” you’re right. It’s a silly and pointless metric, but it’s apparently the only thing that matters to you, so enjoy.
My primary answer is to have the Proxmox system on a mirrored ZFS pool; my secondary is to restore from the replicated backup. The primary has worked well enough that I’ve never needed the secondary.
Zero, because it’s still running on the other drive. And if the system dies completely, the VMs migrate automatically to another host in the cluster and are all up and running in a matter of seconds. Because of that, restoring the host OS isn’t an emergency event the way it is for TrueNAS.
Well, on a home lab 1-box solution, the chicken needs to lay the egg…. It would take a little bit of time to standup Proxmox (the reboot time for my DL380 is terrible) and then stand up TrueNAS and PBS, mount the backup share, then restore systems. But at home, no one is going to fire you over the unplanned downtime.
one thing that I do need to figure out is how to have Proxmox warn me about drive issues like TrueNAS does. But I will figure that out eventually.
