Not sure if my issue is hardware, XCP-ng, or TrueNAS, so asking this of all 3.
I’m trying to expand a TrueNAS mirrored pool currently on a pair of SiliconPower NVMes mounted in a 4-device sled on my ZimaCube. The plan was to replace one of the SP NVMes with a bigger Kingston NVMe. Rebuild the pool, then replace the 2nd SP NVMe with another matching Kingston.
Except that every time I try and boot TrueNAS with the 2 Kingston ssds in the sled I get what appears to be a “trap or panic” and I can’t go much further with the boot process.
Obviously 2 matched ssds in itself is not a problem, as that’s what I have running now. I’ve tried every combination of the 2 ssds in 4 slots, and they all fail. It also works with 2 x SP drives + 1 Kingston in the sled. The Kingston SSDs do work individually, just not a pair.
All the devices are correctly set up for passthrough to the TrueNAS VM.
Does anyone have any thoughts here.
I’m running TrueNAS SCALE Dragonfish 24.04.2.2 on XCP 8.2.1 on this ZimaCube hardware.. The NVMe’s are the new V3 ones.
A few suggestions since you have provided limited information (no make/model of the actual parts (motherboard/Nvme drives (capacity?), a website showing a complete build without specs is not super helpful).
It appears that you have an NVMe adapter which has a built in controller used to serve up the NVMe drives. What Make/Model is in the card. A few photos work as well, without the NVMe’s installed.
What make (you have that covered) and model and capacity are the NVMe drives? Believe it or not, this can be critical to the card you are plugging them into, but hopefully not the issue.
Run TrueNAS on bare metal to test if the adapter card will work as you desire? I have never used XCP so I don’t know how to configure that. Running on bare metal would either confirm XCP is the problem if it works, or if it will not run on bare metal, XCP is not likely an issue.
Back to the PCIe to M.2 adapter. Do you have a user manual that came with it? What does it say? Are each NVMe M.2 drives accessible as individual drives? Or are they part of an internal RAID or Stripe, or ???. If they are offered up individually, understand that the adapter must switch the NVMe drives online and offline as there is no direct access for the NVMe drives to the PCIe bus. Are you mixing drive speeds? It is allowed on that PCIe card? Does the manual state how the slots are to be occupied? Kind of like RAM on a motherboard, must be done correctly.
Not quite a simple as that, as other Linux based VMs (like Proxmox) work perfectly well with passing the 2 x SSDs through. It appears to be only with TrueNAS under XCP-ng that I’m seeing this issue.
As it’s working bare-metal, I’m not sure you need these, but for completeness:
It’s a proprietary card for the ZimaCube, using an Asmedia ASM2824 chip.
Nope, it’s part of the original hardware, not an add-on and there’s no specific reference for it.
Yes.
admin@truenas[~]$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
… snip …
nvme0n1 259:1 0 931.5G 0 disk
└─nvme0n1p1 259:2 0 474.9G 0 part
nvme2n1 259:3 0 931.5G 0 disk
└─nvme2n1p1 259:4 0 474.9G 0 part
… snip …
nvme1n1 259:10 0 476.9G 0 disk
└─nvme1n1p1 259:12 0 476.9G 0 part
nvme3n1 259:11 0 476.9G 0 disk
└─nvme3n1p1 259:13 0 476.9G 0 part
admin@truenas[~]$
See above.
Following the various customer channels, there have been no specific restrictions mentioned regarding slot placement or/or speeds when support folks have been asked regarding the SSDs in the sled.
Thanks for all that information. When I followed the link you provided to the computer, I saw the PCIe M.2 card, thinking that is what you also had. An assumption which is why Joe’s Rules exist.
I am glad to hear it all works properly on bare metal and I agree that XCP-ng is the suspect. At this point I would make a few recommendations:
If you need a Type 1 Hypervisor, use a different one.
Surf the XCP-ng forums and do some research. Maybe there is a setting that will allow the drives to all pass through properly.
You could run it on bare metal. I know I would not like doing that myself, I like VMWare ESXi (free version, yup, no more free upgrades but I keep hoping).
I’m certain you already know this as you seem computer competent but those Gen 4 and especially the Gen 5 M.2 drives get warm. Okay, the Gen 5 get hot. Make sure there is reasonable airflow across those devices.
Ha, I forgot that they also sold an internal card.
I really don’t class Proxmox as Type 1, but a lot of ZimaCube users (from what I see on Discord are going that route). As this was new hardware, I went with XCP because of Broadcom’s money grab.
Not found anything so far and have posted the same information as here on their forums, but no replies yet.
That’s what I’m going to do while doing more research. I backed up the XCP-ng configuration, so switching back is as simple as it is with TrueNAS: Install and restore config.
The comments I got on the Zima Discord when I posted asking about this amounted to: Why would any sane person run their NAS virtual. LOL
I still have an ESXi server as well where I did some initial playing around with TrueNAS, but see above.
