Out of necessity, I am thinking of running TrueNAS virtualized under Proxmox. The idea is to partition two NVME drives, each Drive will be split in half. Half to be used directly by proxmox, and half will be passed to TrueNAS to make a ZFS mirror within TrueNAS.
I have read a few times that passing a partition as disk to TrueNAS is bad idea. I would like to know why exactly.
What are the problems? I want to learn and understand.
Thank you!
Please, avoid answering that is bad idea. It is clear to me that it is a bad idea, I want to understand why.
Proxmox understands zfs - there have been examples where in the past (I’m watering this down to my level of understanding) proxmox tries to do something on the pool/tries to take ownership of the drives & it causes total data loss & pool corruption.
Thanks for the answer.
Assuming this is the problem. This isn’t “passing a partition”-specific problem. You would have the same problem if you passed the whole drive. I wonder if Proxmox can be told to ignore partitions (or whole drives).
You mentioned that these are NVMes - since they are pcie, you should be able to blacklist them from proxmox & pass them through without issue, just like an HBA.
ZFS likes to control the whole disk to be able to guarantee write transactions consistency. This cannot be guaranteed when there is an intermediary in between that can sometimes “lie” to it. A good indicator of this is the inability for you to see SMART data in some sub-optimal setups.
ZFS is not just a file system, it’s both a file system AND a volume manager. And because of this, it is generally recommended for it to have access to the whole disk so it can have full control over the volume management + RAID function process without other things in the way that can potentially intervene.
Not saying it can’t work with esoteric non-recommended setups, but the more you deviate from the “tried-and-true” approach, the higher the risk that your pool can get corrupted in some distant future.
You presumably chose ZFS because it is the most advanced Swiss-army knife file system in the world and trust it whole-heartedly to take care of your data… why would you then hamstring it and set it up for failure?
One issue that @Whattteva mentioned, but the details are that Proxmox’s disk I/O stack does the writes. If there are no problems, then the virtualized ZFS does a good enough job. That might last years without problems.
But, because Proxmox controls the final writes, if it decides not to pass through write cache flushes, (sometimes known as write barriers), OR, writes TrueNAS ZFS data out of order, AND a power loss / crash occurs, then un-correctable corruption can occur.
You might say, hey I don’t plan on any power losses, (I have a UPS). But, OS crashes are not unheard of.
One major point of ZFS, is data integrity. Taking away part of that chain of trust, means permanent data loss and pool corruption can occur.
In essence, ZFS was designed that millions of OS crashes and power losses can occur, and their will be ZERO pool corruption. PERIOD! This was one of the main design goals of ZFS, avoiding boot time file system checks. Now data in flight can be lost, like any other file system, volume manager or RAID scheme. And of course, power losses can cause hardware to fail, (especially consumer / cheaper hardware).
So, careful design of any virtualized ZFS instance is highly recommended to avoid known pitfalls.
Correct: You have to pass entire controllers to ZFS, not virtual disks or partitions.
Proxmox can, and should, be told to ignore entire disks, this is what “blacklist” does. But, again, it applies to a whole PCIe device (controller).
You’d really want four drives for that.
Consider
Using a PCIe switch
Using SATA SSDs rather than NVMe, if you have more available SATA ports than (sets of) PCIe lanes
copies=2 with a single drive
TrueNAS baremetal and VMs/instances/containers under TrueNAS
I’m not educated enough to know this answer, but does this also affect VMWare ESXi?
I have run TrueNAS on ESXi for over a decade, no issues, however I did pass-through the drive controllers. The other method is called RMD which works too, but has more risk and somewhat a pain to setup. I ran RDM for probably a year, until I purchased an extra HBA controller so I could just pass through the HBA. Also, ESXi will pass through NVMe drives that are on the PCIe bus without issue. And, ESXI has a free version, limited functionality, but pretty rock solid stable, providing your hardware is server grade all the way.
But, the smart way to go is TrueNAS on bare metal if you can support it, or passing through the entire HBA (controller).
Thanks, this actually makes sense to me. If this is the problem, passing the whole disk, or r passing-through the whole PCIe device of the disk would be needed.
Now, if I insist on passing a partition, I’ll have to find out if there is a way to tell linux/proxmox to not do caching/reordering of operations in that partition.
Appreciate that quick response. I currently have no plans to use ESXi again for my NAS, but never say never. But I now pass through the entire controller anyway.
You missed the point. Passing the whole SATA or SAS disk does NOT cause the final writes to be with the VM and ZFS. Proxmox still does the final writes because it controls the disk controller. Unlike some Enterprise network controllers, you can’t share a disk controller.
Now passing NVMe whole disks is different because you should be passing the PCIe device through. Thus, that meets the controller requirement.
The one exception that some people do on this pass through of controllers, is passing through a partition for the boot pool. As long as you have a good configuration backup, or are willing to rebuild the configuration if needed, then some people think this is a risk worth taking.
That is somewhat like using USB flash drives for a boot pool. Most USB flash drives are garbage and fail quickly, (within a year, even far less). But, their are some brands and models that do work very well. Since it is just the boot pool for home use, some people are fine with that risk. The data pools should be fine in either case.
PS: The Enterprise network controllers I mention, can allow either separating ports to different VMs. Or, even creating virtual network interface controllers so that several VMs can have direct access to a single network port.
What you said before makes sense, I have to investigate if from the host side there are ways to make the passed partition act as dumb as possible (no caching, reordering). Maybe, some kernel parameters can help. I’ll have to study.
The one exception that some people do on this pass through of controllers, is passing through a partition for the boot pool.
Before, I was specifically referring to passing partitions for making a ZFS raid on the guest side for data. I never considered the problem of the boot partition. I don’t understand why you would wanna pass a partition (or two) to be used as boot. Isn’t it better/simpler to just have the TrueNAS boot partition live in the host storage space (which could be a file within EXT4 partition or a host-managed ZFS pool)?
I was always under the impression that the TrueNAS boot partition was rather simple (and small) and it didn’t itself use ZFS. Am I missing something?
Can confirm without doubt that boot pool is using zfs (this isn’t just because I mirror’d it):
Last login: Sat Jun 6 14:27:02 EDT 2026 on pts/9
root@truenas[~]# zpool status
pool: boot-pool
state: ONLINE
scan: scrub repaired 0B in 00:00:33 with 0 errors on Sat Jun 6 03:45:34 2026
config:
NAME STATE READ WRITE CKSUM
boot-pool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
sdg3 ONLINE 0 0 0
sdj3 ONLINE 0 0 0
I’m sure there is more under the hood going on that I don’t understand. Realistically, as long as you’re comfortable backing up the config & doing the needful yourself without expecting official support, you could keep it running. But the more complicated it is to create, the harder it’ll be to restore - hence (one of the reasons) why it isn’t recommended/supported.
Using a file on a Proxmox host, means double writes, and double the chances of pool corruption. Even if it is just the boot pool. This is because ZFS will write to the boot device file, then Proxmox will write to the file, then the underlying file system will flush its writes to disk. Thus, more chances of write mis-orders and such.
Again, it is all about your risk acceptance.
It has been quite a few years since TrueNAS used a simple boot OS. At least 2019. Using a full blown ZFS instance for the boot device allows certain features to be added. Like boot environments, (OS updates that still allow booting prior OS). Now I have implemented boot environments without ZFS, (or BTRFS), but that was clumsy.
One last thing. If you do not pass the data pool storage device’s controller(s) through and blacklist it / them in Proxmox, then if you get pool corruption, no one may be able to help. Full rebuild and restore from backups, (you do plan on backups?).
Some pool corruption is so complete, even ZFS can not recover. Perhaps a paid recovery service might help. Basically, using ZFS outside its known working configurations means risk. Even some consumer hardware has been known to cause problems, (on bare metal!).
Possibly you missed that this was indeed the case for older FreeNAS releases. They booted from NanoBSD, a special implementation of FreeBSD that still exists and uses the diskless framework to keep the boot partition read-only while permitting upgrades and rollback via two separate complete boot environments that are updated alternatingly.
But that was changed to a ZFS boot pool long ago. I could not find a reference easily - does anyone know from the top of their head? FreeNAS 8? 9? 11? 11 did have a ZFS boot pool IIRC.
Checking, it was 9.3 that started supporting ZFS boot pool. That was released about 2014/12/8.
Another reason the move to ZFS happened, is that adding a Mirror device to the boot pool could be done live. In some other RAID schemes, (Linux LVM Mirroring or Linux MD-RAID), it is absolutely necessary to “encapsulate” the root partition, which involves a reboot, (or 2 if you mess things up).
after years of use of truenas, and i finally moved to Debian OS + Proxmox + ZFS.
You need a small nvme bootdisk to install Debian (my case a 512G ext4) first, and then install the proxmox. I then create the ZFS into the NVME to be used by the OS/Proxmox. OS already enabled SMB / NFS and served home folder. I can create a lots of LXC to mountpath from NVME directly to apps in LXC as mounted point.
So, you may need to sort out what you want to acheive i.e. what is your necessaity?
