I want to have proxmox as host system AND truenas as its guest. My ideal setup (the long term plan) is to have a pair of M.2 SSDs dedicated to proxmox, and pair M.2 SSDs + pair of HDDs for truenas.
My current homelab system doesn’t have enough slots on the motherboard for that, hence the idea of partitioning the M.2s.
I have 1 x 512GB and 3 x 4TB nvme on motherboard. I split the 4TB into 2 partitions as 200GB + remaining space. The 200GB each in part1 partition is become “3-way mirror” metadata vdev for my hdd pool and the remaining ~3.9G x 3 become another Raidz1. They are working complete fine. I do not see problems for split partition on purpose and should conside it impact to 2 or N related zpool at once.
I suggest the “zfs” filesystems focus on storage but flexible conside for OS / apps.
So, what is your target usage of truenas?
smb/nfs? → provided by debian for home files
apps? → provided by proxmox on debia, support LXC and VM
user friendly? → the change is easy now with AI e.g. gemini
i am using esxi + truenas as guest because of the hdd passthrough via HBA for many year without problem, and it is no more a concern to just use debian+proxmox and no longer need the truenas layer.
Gary, now you are debating whether I should use TrueNAS at all when can do everything within Proxmox. That is certainly an interesting proposition, but it’s not what I asked.
Assume, I want to run proxmox and put truenas in vm and pass to it a partition, not whole PCIe device, would there be problem? That’s the question. The other have explained where the problems may be.
In truenas view, it should be no problem, and you may need to look at Proxmox ends.
One reminder, Proxmox can change the disk always, and it is how i lost a truenas pool during testing. A must to be noted.
You may considered the native proxmox zfs vol(s) for simple truenas test.
LLMs are cool, but let’s not use them just to fill space, especially if the context isn’t provided, and it’s throwing in discussions about running on virtual disks instead of partition devices. Whole different set of risks involved there.
@WhiteNoise apologies if I missed it, but what are your system specs, specifically the motherboard? You said:
And my immediate thought is “let’s see if you can run a PCIe bifurcation card to get four M.2 slots” and solve the root cause of those. 
Failing that, if you’re talking about slicing up NVMe devices what you might want to hunt for are M.2 cards that support NVMe namespaces - those will appear as separate individual devices (eg: /dev/nvme0n1 and /dev/nvme0n2) to the host Proxmox OS. At that point, you can quite easily consume the first from the host and pass the second one to a guest as a wholly independent PCIe device - without worrying about anything stepping on each other’s toes.
If they are in different IOMMU groups. Is that the case for two namespaces on the same SSD?
Ooh, good point, it probably won’t.
But at least a namespace can be passed as a “raw disk” which is still not ideal but at least we know how to mitigate the risks.
I’d still want to know if bifurcation and separate physical hardware is possible.
Thanks, my MB does not support bifurcation, however I could buy one of those multi nvme adaptors with a PCIe switch embedded (such as this).
I didn’t think about namespaces. You’ll likely need enterprise drives for that, but it’s clever suggestion, if they are in differente IOMMU groups.
So far, I have used namespaces only once to overprovision a drive.
I think I will likely have this box as TrueNAS only. When my current workstation, which has plenty of slots and features, will be retired and converted to be the next server, I will revisit this.
Thanks everyone in this thread for the feedback.
I read your very helpful thread on passing raw disks. Would you say that the risks are the same as passing a partition as a disk?
I would think it’s a bit better just because it’s unlikely the host OS is able to discern the “sub”-partitions created created by the guest OS and attempt to automount ZFS.
Whether you pass a raw disk or a disk partition, both REQUIRE the host OS to perform the actual writes. Which it may do so out of order. Or not properly perform write cache flushes when requested, (aka write barriers). Remember, by not passing the disk controller to the VM TrueNAS, that means the host OS MUST manage the disk reads and writes.
An overwhelming amount of the time this will work just fine for ZFS. It is when their is a host OS crash or power loss, (A.C. or power supply), that “bad” behavior can cause pool corruption. Thus, we get people here in the forums saying “It worked for months / years, why corruption now?”.
Again, ZFS was DESIGNED for data integrity IF given proper hardware. If you use hardware that is not ideal, you get only limited amount of data integrity. (You get perfect data integrity until something bad happens…) Some people are willing to risk it, only you can decide if it is worth the risk.
Yes, I understood your point. If this is indeed true, you may want to post this comment in the whole disk thread for completeness.
What ‘whole disk thread’? Are you talking about the online documentation for TrueNAS or one of the whitepapers about virtualizing TrueNAS?
I am talking about the thread that honeybadger linked and authored.
Arwen’s argument maps with this paragraph in the blog post, but it is vague on details.
Hypervisor-backed virtual disks or hardware RAID controllers provide a “translation layer” to the disks, and therefore should be avoided for the data disks. TrueNAS boot devices are an exception, and can be stored on a hypervisor virtual disk safely – but it is suggested to create two identical volumes and use the TrueNAS installer to mirror these within the guest OS as well. Ensure that the underlying physical storage backing these hypervisor virtual disks is sufficiently redundant as well.
The details may be more appropriate in the blog post. In fact, the paragraph about the one you quoted describes the issue in general terms:
3. Virtualizing ZFS
The ZFS file system used by TrueNAS combines the roles of RAID controller, volume manager, and file system all into a single software component. ZFS expects direct access to your disks in order to work properly, in order to issue direct SCSI or SATA commands and receive an expected and predictable response. The closer you can get TrueNAS to your storage hardware, the happier it is, and the better it can do its job of keeping your data safe.
Now people may want more details to understand the problem better. However, these words are reasonably clear:
ZFS expects direct access to your disks in order to work properly, in order to issue direct SCSI or SATA commands and receive an expected and predictable response.
Perhaps adding details to @HoneyBadger Resource post might be useful. But, then again, this particular thread has gone on for 32 replies. Meaning not everyone will be satisfied with some wording, and may always have questions.
For example, I wrote this Resource which covers some of the aspects of direct access to disks, (instead of hardware RAID…).
Yet people persisted in thinking a UPS was required for data safety. It is not. However, UPS(es) are a good thing for multiple reasons, some outlined in my Resource or following discussion.
