Can't boot TrueNAS Scale on Proxmox after HBA passthrough enabled

TrueNAS General
1

I know a number of people are running TrueNAS on Proxmox, so I thought I’d ask for help here. I did post a version of this in the Proxmox forums this morning, but it hasn’t gotten any attention, and I’m hoping someone might have some ideas. Happy to provide more info as necessary. tl;dr: I have a TrueNAS Scale VM in Proxmox, but when I add a passthrough HBA to it, it fails to boot.

I’ve got a new build, that I was planning to use to run Proxmox with TrueNAS in a VM, along with some other virtualized apps. (I’d previously considered running TrueNAS on bare metal (as I currently do), but I think I’d rather run my other items under Proxmox than use Scale apps.) I’ve successfully installed Proxmox, created some VMs, and generally played around, but I’m now having trouble starting the TrueNAS VM after I added a passthrough HBA to its hardware.

I think I’ve set up PCI passthrough correctly, and IOMMU and vfio are all activated and show up in my dmesg output.

I created a new VM for TrueNAS Scale, and installed Scale into it, and this initially booted quickly and correctly. I then shut down the VM, and added three PCIe hardware devices to it: an HBA (LSA 9211-8i flashed to IT mode) and two other PCIe hard drives (U.2 NVMe drives attached to Oculink x4 ports). After this, the Scale VM has trouble booting. When I watch the console, SeaBIOS hangs for some time (showing the SeaBIOS version line, and the Machine UUID), then gives various messages indicating that it’s having trouble booting. (It might show an “Initializing…” spinner, maybe a Proxmox logo, then something about an iPXE command line, then something about no boot device being available, then it’ll go back to the SeaBIOS version/Machine UUID line.) Sometimes it shows all of the passed-through drives as available, then it can’t find the boot disk. (The VM options in Proxmox show “sci0, net0” for the boot options, but apparently sometimes SeaBIOS is attempting to boot from the passed-through drives, which are not selected as boot options.)

I’ve tried rebooting a number of times; once and only once it did go through to the installed TrueNAS, and everything looked fine–all the passed-through disks were visible in the “storage” area. But when I rebooted the VM, it once again hung for a while and then got stuck in the same loop. I’ve also tried deleting the VM entirely and installing it again, with the same result–it’s fast and successful until I add the passed-through devices, then it gets stuck at boot.

Running TrueNAS is in fact my main purpose for this installation, so I’d love to figure out how to get this to work. I can do my best to provide additional information about my setup. Thank you.

2

Followup, after some more experimentation: It’s just the HBA that causes this. If I pass through the two other PCIe drives, it boots right up. If I only pass through the HBA, it hangs. (And to clarify, this is hanging before the VM boots into TrueNAS.)

What should I be looking at, to figure out why the HBA is triggering this?

3

Yoh could try passing the HBA into a vanilla Debian VM to hopefully discount TrueNAS scale as the source of the issue.

4

What steps did you take to pass through the hardware? I run TrueNAS Scale and Core on proxmox, the passthrough process changed with proxmox 7.xx I seem to remember. GRUB or systemd?

P.s I’m not really smart at this stuff but I did tinker my way to success with the help of good forum folk. Just trying to pay it forward.

5

Sounds like the problem I had before. Here are the steps I think I did to resolve it. Mind you, it may not be perfect or in the correct order because this was like a year ago. You also may or may not need all of these steps.

  • rombar=0
  • Make sure all your firmwares are patched correctly and you’re running the correct version.
  • Disable OPROM
1 Like
6

Well, after quite a bit of experimentation, and reading a whole bunch of posts in various forums about this, I figured out that in fact the rombar=0 option is all it took! That is (for anyone else finding this later), in the relevant config file for the VM, add that flag in the line for the device; or simply uncheck the ROM-Bar box in the relevant “PCI Device” section in the “Hardware” tab. As soon as I did that, everything started up right away, I have a lovely TrueNAS Scale VM that sees all of my passed-through disks, and yay.

Apologies for the lack of clarity and lack of hardware description in the original post; the former was because I didn’t know what was going on, and the latter was because I forgot the rules.

1 Like
7

More info from this old thread

8

Thats literally my first bullet point im the post right above!!!

9

Yes, which is why I said so in my reply to you!

I’m sorry if my post implied that I found it on my own.