I would suggest devirtualizing TrueNAS. See if your problems go away
(this can be done quite easily by attaching a second physical boot device as a mirror to the virtual boot device, and then booting off that)
Alternatively, if you must virtualize TrueNAS, try with ProxMox.
end-of-line
TrueNAS has its own virtualization. SCALE has KVM and is generally recommended. Is there a reason that you canāt use that at the bottom of your stack?
Yup, I would totally de-virtualize TN firstā¦ at the very least to troubleshoot the issues.
I think this is clearly showing us the problem is that the HBA (mpr0, in this case) is resetting. Its certianly possible some software problem with PCI-E passthru exists here, but I doubt it.
Most likely this is caused by heat. These LSI cards are designed to be run in servers with large volumes of air moving over them. Sounds like your doing this in a desktop form factor.
Strap a fan on it!
Sorry for a long period of no responses, I normally donāt interrupt troubleshooting for that long but holidays and a project came in between.
I am trying for multiple days now ā¦ without success. Sounds ridiculous but I canāt install TN 13.0 U 6.1. because it gets stuck during boot, more precisely at the āmasksā same as here: Reddit - Dive into anything
I canāt disable UEFI boot (aka enable CSM with legacy option rom) because itās a remote machine and the KVM doesnāt support grabbing dedicated GPU signals ā¦ only the iGPU. That means every time I enable CSM I wonāt get a signal from my ASUS IPMI ā¦ seems actually ridiculous that they donāt have an input on that card to plugin an external source like a PiKVM. So I will try to get the machine prepped with a monitor by a family member in the next view days and try it again.
I mean I can, but I have no experience with Scale whatsoever and as already mentioned I really wanted to use the CPU pinning capabilities of Unraid + other conveniences. Again: not an expert but in terms of refined User interface, functionality and variety of capabilities it seemed the most refined product. Proxmox looked awesome for clustering but was lacking a lot of features compared to unraid. I donāt know about Scale but Core seemed perfect for ZFS, Snapshots, managing storage hardware etc, but not that well for a huge variety of virtualizations. One example: Unraid has awesome support for MacOS virtualization and a good community behind this too. Itās like a OS preference for me. I wanted to try Proxmox, ESX and TN Scale on a separate machine to tinker with. If my troubleshooting leads to nothing I will change the hypervisor ā¦ but for now I am not giving up 
Yes totally agree ā¦ I will write an update here as soon as I have news.
That is genius ! I had a problem with the card getting extremely hot ! It nearly burned a cable isolation. I already strapped a fan on it BUT on the back side and two small ones next to the cable ā¦ I had to get rid of the two smaller ones to rearrange the PCIe cards (all slots are occupied). The heatsink has no fan on it because of space constrains ā¦ that could be the problem. I will open another thread for HBA suggestions. Unfortunately the LSI doesnāt have a temp sensor and I donāt have a IR gun. I will post a picture of the configuration maybe you guys have an idea what an easy fix would be. Thanks anyway @NickF1227 ! I would have never thought about that after I had the card up and running. Can the card be damaged already or should the āresettingā mechanism prevent this ? After all I still got normal net transfer rates expected from a 2x5 RaidZ2 of 200-300Mbytes/s. And the two ssds on the other (8x SATA - reserved for Unraid) half of the controller never had problems from what I saw.
In one of my home servers I have a 120mm fan literally just laying on top of my pcie cards with some bread ties so it doesnāt move around 
Iām sure you can come up with some redneck engineering and get it to work without buying anything
Same here 
My guess is not having any active cooling directly on the heat sink is not good enough. Any ideas here ? Are there small slim fans that can directly attach to the heatsink ? The space is very constrained. I had to move a Quadro K4200 to the x4 slot at the bottom. And because of that there is no more space between HBA and GFX for the 40mms. I want to have the HBA on an x8 by the way to have full bandwidth (only the upper two ones are x8 one occupied by the main GFX 3060TI).
Donāt mind the cable management the case is pretty much full ā¦
Only thing I can imagine is another riser ā¦ but I already have 3 in there 
Hmm. From the picture, it looks like your actually directing airflow AWAY from the card. Try rotating it like this.
They are placed further away right now, because the Quadro was in the way. They are currently attached to the cables, not ideal but the only way to get them zip tied in place (gentle btw the ports should be fine). I donāt think that the current distance from the heat sink is effective. I canāt place the fans like you indicated, at least not without changing the arrangement of the cards.
I just found 3D printed funnels to mount 80mm fans, but it will take some time for me to get the printed parts and figure out how to place it (and the card).
Nah it should be good.
What are you using as boot drive?
USB flash drive. I will try a different slot. I think it is connected to a 3.0 slot instead of 2.0. But using the IPMI mount capability leads to the same problem: the boot process gets stuck at masks. And the IPMI mount is emulating a CD/DVD drive as far as I saw in the boot menu. I also have a spare SATA dongle that I could give a try.
But I will try to get the HBA cooled first, to rule that problem.
How big?
First chose 8gb that one was too small. Then I switched to a 32gb one. Didnāt have a 16Gb on hand at this point, but I will try a 16Gb one in a USB 2.0 slot when I modified the HBA with a 3D printed funnel for two 40mm fans.
Yeah, wouldnāt recommend that. Not anymore.
I had ~30-40 NASās at my last job. Some of them were still running FreeNAS 11 until just before I left.
I cheaped out and used USB drives and a decommed old LeftHand SAN. I figured Iād be fine with 64 gigabyte ones from a reputable brand, usb 3.0.
Years laterā¦My old boss called them nickbombs
They boom.
nice ā¦ when I first read FreeNas 11 tutorials it was best practice together with regular external flash backups.
Keep in mind that my goal is still to virtualize so at first the boot drive will be put as a OVMF virtual disk on an SSD that has parity and will be backed up later. Ideally I want to use two SATA DOMs (one as parity) to directly save the boot environment, at least that was the plan. In a case of a DOM fail I can use the other one. In case of a HOST fail (Unraid) I can boot directly from one of the DOMs to guarantee access to the NAS. Of course still with external flash and data backups. But for now it remains an idea ā¦
These + velcro are a decent external option, obviously unplugging is bad so YMMV.
You can pass thru the USB drive from Unraid directly into TrueNAS. Obviously this is a point of failure, but it would probably work fine.
I have this at home but never thought about it. I will give it a try. But why isnāt it working with a USB and UEFI? That is what I actually donāt understand. I have an older TN machine that still has a USB as a boot media and I never enable CSM intentionally. It just booted. Maybe the reason is an older mainboard. Iām cluelessā¦ but it makes everything more complicated for me with IPMI control.
I also should mention that this is the installation stick. In the past I used 2 sticks. One with the installation ISO and the other one that the boot partition is written to. The first one is then obviously detached after installation.
32GB ones should be big enough for TN to install.
I am currently running a MLC FLASH USB as boot drive, no issues so far.
