I virtualized TrueNAS on Proxmox using an old X10SDV and LSI 9300-16i. I used the HBA passthrough design.
High-level steps:
- * Research, * research, * research. The last one was the primary guide.
- More research into how others have ventured into this territory and failed miserably. I tried to avoid doing those things.
- Ensured my hardware was compatible (Vt-d, ACS supported even though I didn’t need it)
- High-level: enable Vt-d in BIOS, modify kernel boot parameters to enable immou, isolate the device with VFIO passthrough, blacklist the HBA drivers, passthrough the HBA (ROMBar=0,PCI Express=1). Raw virtual disk for Truenas boot disk (no ZFS on ZFS). I would hope to be able to snapshot before upgrades, but LVM snaps atop ZFS seem hairy. Storage pools are backed by physical disks (hence the above)
It’s passed all IO testing so far and imported an existing pool. I’m trying to get it break under different scenarios but it’s holding up like a champ. This is just a test but so far it’s gone really well. Networking tests are next. I found a few posts where some had issues with NFS in similar configs.
Any other advice to verify stability? Any unknown gotchas?
Feels nice to have a home lab again.
I’ve decided that TrueNAS is best at being storage, proxmox is best at being a hypervisor, docker is best at containers. I’m separating these functions in the final design. While I’ve always appreciated the builders and maintainers of TrueNAS’s app (or previously jail) ecosystem, this is where my frustration with the product has usually begun.
While frankly, that exposes me to more complexity than I would like, I feel like it will ultimately be more stable.