If move pools from TrueNAS (bare metal) to TrueNAS running under Proxmox, do you have to export the pools first?

stk · October 10, 2024, 12:00am

I WAS THINKING of moving TrueNAS to run as a proxmox vm, but decided against it since it adds another point of failure for no huge benefit in my case.

Here’s the original thread…

I’ve added the controllers for my nvme and hard drives to my VM with two add pci device commands. I’ve updated grub to add the intel_iommu=on line to the startup options after “quiet”, and ran grub-update. I rebooted and all is good; lsblk sees the devices just as expected

Now I’m at the final step which is moving the pools to the new instance.

Presumably I:

OLD SYSTEM

export the .cfg info
export the disks

NEW SYSTEM

import the .cfg file
import the disks

I’m guess that the order for 1&2 in each step is irrelevant.

Am I right?

sfatula · October 10, 2024, 12:21am

It would be pass through the HBA not the disks which you seem to have indicated.

This is assuming you are aware of the fair number of lost pools lately coming from Truenas deployed on Proxmox and are willing to assume those risks for whatever benefit you think you are getting. There were at least 3 or 4 here in the past several weeks, and a few on reddit as well. There are some indications that some recent Proxmox has a problem that is causing this.

stk · October 10, 2024, 1:07am

yes, I pass through the two controllers (nvme and sata).

what would happen if I just imported the cfg to the new truenas and ran with it without the export then import? would it work? if not, what would happen.

Thanks for the heads up. I’ll probably put a pause on this project until the stability issues are resolved!

winnielinnie · October 10, 2024, 1:29am

Technically, importing the config file also imports the pools in that config file.

So if the member devices are available, it should be able to import them upon loading the config.

stk · October 10, 2024, 3:04am

cool. so what happens if I just move the config file without exporting the pool first?

Does it just work? Or does it detect a “new owner” and complain?

I’m guessing that it has to work because I seem to recall having to reinstall truenas, then updating the config and everything was happy.

So am I correct that all I have to do is simply bring up the new truenas, import the config file and I’m done… no export/import needed, right?

winnielinnie · October 10, 2024, 3:30am

It might complain, since a ZFS pool “remembers” the last host it was imported on, if it hadn’t yet been exported first. But there’s nothing drastic that happens if you don’t export it first.

Obligatory caveat: Tread carefully with Proxmox + TrueNAS.

stk · October 10, 2024, 6:02pm

Thanks! That’s what I thought re: export/import not needed.

So with Proxmox/Truenas, there is no doubt that if storage is your priority, you never want to have TWO points of failure instead of one!

Therefore, running TrueNAS on bare metal always makes the most sense: fewer points of failure.

But I am very curious… Are there cases where the PCI passthrough is set up properly with full control that proxmox can corrupt your pools? I don’t see how this can happen since proxmox is out of the loop. Are there documented cases of this happening?!?

Also, in your honor, I created a seatbelts script attached
seatbelts.txt (1009 Bytes). Thanks for your post on that.

sfatula · October 10, 2024, 6:15pm

There are several in the last few weeks here, you can find them if you wish. And some on reddit also.

winnielinnie · October 10, 2024, 6:19pm

Hah! Now if only it was “GUI-fied” for SCALE.

Which reminds me, I made a feature request on Jira, which was closed because of the “newer” way to submit feature requests on here…

stk · October 10, 2024, 9:01pm

I think this is the best explanation I’ve seen for how the corruption can happen: Can't import datapool - #4 by HoneyBadger

Summary: even if you use pci passthrough, proxmox can potentially violate it.

sfatula · October 10, 2024, 10:02pm

So, add a little backstory, what is the “pull” to run Scale on Proxmox? I honestly do not understand the why, any background of why this might be a good thing?

It used to seem safe passing through the HBA. But now we see it is not necessary safe. Proxmox adds another level of potential failure, that’s not appealing for a NAS, at least for me. So, it can now fail for a Scale bug, OR a proxmox bug/issue.

argumentum · October 10, 2024, 11:48pm

RGB in the memory sticks, make a gaming PC runs faster.
Just like broken/noisy car exhaust makes it more powerful

I did a Proxmox with TrueNAS because, everybody in YouTube say is the thing to do. The only way to know otherwise, is to find out first hand.
New to the game I guess. I certainly am.

But coming back and participating in this forum is giving me a range of experiences that otherwise I would not have gotten.

In any case, I doubt the OP has a different story.
The premise of a good hypervisor sounds good. And only experience takes you one way or the other. Unfortunately.

sfatula · October 11, 2024, 6:04am

You found out all right! No idea why so many youtubers recommend that. Well…

stk · October 11, 2024, 7:00am

You are correct. I noticed other people did it and I tried it out just to get the technical chops to learn how it was done.

I do have a truenas instance running at ServaRica which runs in a VM, and that does work just fine (so far).

I won’t be deploying it on my main system for the reasons you cite.

sfatula · October 11, 2024, 8:59am

I was actually replying to argumentum as he had been running virtualized via proxmox until he learned recently when he lost his pool.

There is some population of folks out there who’s answer to any question is proxmox. I don’t personally think it’s a good idea for a NAS and data you want to keep. Though, just like pool choices, if one keeps regular backups I suppose one could recover. As long as the bug or issue does not re-occur very often. I still fail to see the point though. Scale does docker, nas, and vms if you want them all on one server.

stk · October 11, 2024, 2:16pm

Yup, and you can also run Proxmox as a VM inside Scale and that works extremely well.

winnielinnie · October 11, 2024, 2:18pm

But… why?

Kevo · October 11, 2024, 5:55pm

I’ve been running TrueNAS in ProxMox for years. Since version 9 something IIRC. I’ve never had an issue. I did have a bit of a ProxMox scare with the 6.8 kernel updates, but I’ve got an old mac mini I run as another ProxMox server that I test all updates on first. Luckily that machine broke and not my main server. So I just waited on updating the main server until the kernel issue was resolved. Also, I am using PCI passthrough on the SATA controller on my motherboard so TrueNAS can use it directly. For my USB backup drive I am actually passing in the drive by-id instead of by USB device because in my testing years ago now, the USB passthrough was not working well. The passthrough by-id has worked so well, I think I could actually drop the sata passthrough and just pass all the array drives by-id as well and it would just work.

In any case, you do need to be careful. I think for a lot of people ProxMox was so stable for so long that they just updated and got bit by some of the recent kernel issues and changes. IOMMU changes were made in 6.8 IIRC that did cause problems. So if you decide to go the ProxMox route just stick to some safety practices no matter what and you should be fine. I’ve never had it break except when updates were involved.

As for why I use ProxMox for TrueNAS. Mainly it’s because I have one decent server and TrueNas was way worse for VMs. So it was a no brainer for me. My router, web server, mail server, NAS, VOIP, media server, NextCloud, directory server, all run on my one server. It’s pretty cool. If Scale ever becomes as good as ProxMox running VMs I’d probably switch over to it, but I don’t think that’s going to happen any time soon.

I should also add that I have only used CORE and not SCALE with ProxMox. There might be some differences based on one being FreeBSD and the other LINUX. Maybe SCALE is riskier than CORE when used with ProxMox.

Last possibly relevant detail is I don’t use TrueNAS in ProxMox while using ZFS in ProxMox as well. ZFS is not very performant and most of the time my ProxMox setups are not funded well enough to have RAID and backup. So usually it’s one fast SSD for VMs and a spinner for VM backups. I have had to restore a whole server before and it’s easy to do with full VM backups on the spinner.

argumentum · October 11, 2024, 7:19pm

I thought that my followers ( in all my vast media platforms ) kept chronological notes of my …adventures

I have 8 physical PC, well 6, …2 are unplugged.

In regards to the Proxmox box with TrueNAS, I chose CORE because SCALE did not behave well under “what are you doing, don’t do that” like unplugging HDDs and re-plugging them live. Loose power ( by pulling the plug ).
I have 2 proxmox boxes but just one with TrueNAS.

I usually run M$ everything and like Hyper-V. I can in Hyper-V copy the folder holding a VM to another PC, import in place and keep running.
The memory needs are quite efficient too. Then again all I run are M$ VMs and is all well integrated.
M$ been … M$, one can never know if a new bug/virus/gofigure, comes up and destroys everything. Also their “RAID” stuff, speed wise sucks. Then again the bug/virus/gofigure aspect makes it unreliable.

Where can I put my files and trust it’ll be there, uncorrupted, until the end of time ?, simple: TrueNAS. I have files since the 90’s ( and yes, am a data hoarder. I’ll have to share that with a shrink )

The question is what can you trust to be boring for ever ? Not a clue, hence “backup, backup, backup”.

In my case, what am looking for is a system that I can “eat my cake and have it too”.
What that setup looks like, I still don’t know.
Ideally would run VMs as if it was on BareMetal without having to ever touch a command line.

Ok, let’s recap. Yes am running Proxmox with a TrueNAS VM. No, I don’t trust it. The TrueNAS that I trust are those running on BareMetal.
If I ever find a way to run all I care, without having to also be the Admin, I’ll make a post. But is unlikely that I’ll ever find it.

PS: I would not virtualize Proxmox inside TrueNAS. Makes little to no sense once you find your way around TrueNAS.

sfatula · October 11, 2024, 7:42pm

Wrong guy then, it was someone else in the post. No, I don’t have a chronological notes, false memory!

Oh, you’re one of those guys, Core.

Ex freeBSD guy here, ran it for many years in a company.