HBA passthrough

[Maxdiken]

Hello,

I am running a TrueNAS Scale VM on Proxmox. I first passedthrough the SATA controller of the motherboard so I could create a 4-HDD pool in RAIDZ1 on TrueNAS but I then realized I would rather have my HBA passedthrough and keep the motherboard device on Proxmox.
Is it possible to connect all 4 drives on the HBA instead and passthrough the HBA controller to TrueNAS while keeping the pool and its content intact?

Thank you.

[SmallBarky]

Proxmox with TrueNAS as a VM is tricky. Please see linked items. I would back up your pool data elsewhere, right away before trying anything.

Virtualize TrueNAS

Yes, You Can (Still) Virtualize TrueNAS

TrueNAS scale on Proxmox - boot-pool is accessible by host

Hi there, I successfully installed TrueNAS scale on a Proxmox VM (PVE 8.2.7, Kernel 6.8.12-3). My VM storage is located on a ZFS volume I created with Proxmox. Now the issue is that every ZFS volume I create with TrueNAS on this or other ZFS volumes managed by the host is also accessible by the host, zpool import returns all of those pools, including the boot-pool of my TrueNAS VM. I already deactivated zfs-import-scan.service as proposed in this thread. Nevertheless the boot pool is being se…

[Maxdiken]

That doesn’t really answer my question… I am already well aware of the risks of virtualizing TrueNAS on Proxmox.
I am trying to know if using an HBA instead of the motherboard SATA controller as the passedthrough device for an already-existing pool is possible, while keeping the data as is. Would TrueNAS still recognize the pool?

[neofusion]

@Maxdiken

Would TrueNAS still recognize the pool?

If the HBA is set to IT mode, yes.

[Maxdiken]

How do I make sure IT-mode is properly set?

[Fleshmauler]

Output of sas2flash -list or sas3flast -list should give you the firmware version & let you know if the HBA is in IT mode or IR mode.

[Maxdiken]

    Adapter Selected is a LSI SAS: SAS2308_1(D1) 

    Controller Number              : 0
    Controller                     : SAS2308_1(D1) 
    PCI Address                    : 00:06:11:00
    SAS Address                    : 56c92bf-0-000a-4166
    NVDATA Version (Default)       : 14.01.00.06
    NVDATA Version (Persistent)    : 14.01.00.06
    Firmware Product ID            : 0x2214 (IT)
    Firmware Version               : 20.00.06.00
    NVDATA Vendor                  : LSI
    NVDATA Product ID              : SAS9207-8i
    BIOS Version                   : 07.39.02.00
    UEFI BSD Version               : N/A
    FCODE Version                  : N/A
    Board Name                     : SAS9207-8i
    Board Assembly                 : N/A
    Board Tracer Number            : N/A

    Finished Processing Commands Successfully.
    Exiting SAS2Flash.

Is this the expected output to confirm it is set to IT-mode?

[Fleshmauler]

@Maxdiken

Firmware Product ID : 0x2214 (IT)

Yep - (IT) confirms it. Only thing worth noting is that my specific HBA has firmware version 20.00.07.00, while yours is 20.00.06.00. Might be worth checking if any firmware updates.

[Maxdiken]

Could you please explain how would I go about updating the firmware?

[Fleshmauler]

I think this detailed post where a user catalogued their entire experience would be more helpful than anything I quickly type-up:

My experience buying and flashing a used LSI HBA (SAS9207-8i, aka SAS9208-8i, aka 2308) to replace my "dodgy port multiplier"

So, after the recent Dragonfish upgrade, my cheap no-name SATA port multiplier card stopped working. This problem was reported by multiple people and is being tracked as bug NAS-128478 by the TrueNAS engineers, but in the meantime I read through a lot of discussion about SATA port multipliers vs host bus adapters (HBAs), and in particular the “legendary” (according to the docs) LSI HBAs. Although I think using junk hardware in general is a fine and wonderful thing, I found several persuasive ar…

Edit: there are more guides, if this doesn’t work, that walk you through using eufi shell. I’d recommend glancing around the forums since at best I’d be giving a half-remembered process if I were to write it out myself.

[Maxdiken]

Thank you for the link! This looks like everything I need to know for my use case.

As for changing the device the drives are connected to… could you tell me if the following steps is the right way to proceed:

1. Shutdown the TrueNAS VM (I am not sure if I should export or disconnect the pool beforehand)
2. Detach the motherboard SATA controller from the TrueNAS VM.
3. Shutdown the whole system.
4. Disconnect the drives from the motherboard and connect them to the HBA.
5. Boot up Proxmox.
6. Start the TrueNAS VM.

This process implies that I have already passedthrough the HBA to TrueNAS.

If I understand correctly, after that, TrueNAS should recognize the pool exactly the way it was when it was connected to the motherboard. Is that correct?

[axhxrx]

Wow, it’s funny that post came up tonight, because I actually referred back to my own post for guidance when installing an LSI HBA (for only the second time in my life, I feel obligated to add) — and it didn’t go nearly as well as the first time.

I hit several of the annoying or scary-sounding errors in trying to re-flash my AOC-S3008L-L8i (which maps to LSI 9300 series) including:

• Cannot downgrade NVDATA version
• Failed to get valid NVDATA image from File!
• ERROR: Cannot Flash IT Firmware over IR Firmware!

So I was thinking I should maybe write this up in detail, too… but it is now 5AM, so that’s starting to look like “next weekend” which is code for “never”, so will instead just leave these links that were very helpful to me, and some bullet points:

• First, the OP in this thread has an HBA that is already in IT mode (as evidenced by the “(IT)” in the firmware description — so no need to read this, if you are the OP. But for future humans googling this with kagi.com:

• This often-linked post, How to flash a LSI SAS 3008 HBA (e.g. IBM M1215) to IT mode was my starting point, and is useful… but then I got those errors above.

• the “can’t overwrite IR firmware with IT firmware” thing seems to (in my case, and many others I read about while investigating) just mean that you can’t do that without first erasing the flash section of the memory (which you do with sas3flash, the same tool you are already using), and reflashing. I learned that from this Unraid forum post. That post has an interesting suggestion to run sas3flash.efi -l LOGFILEFLASH.txt -o -f SAS9300_8i_IT.bin -b mpt3x64.rom to sanity check first, because even though it will fail with the “cant overwrite IR with IT” error, it will confirm the firmware you have will work with your card.

• I didn’t need to put any jumpers on my card, and many others said the same thing.

• The previous time I did this, as described in my original post, I did it while booted to TrueNAS SCALE. But this time, I did it in a UEFI shell as described in the first link — you just need to copy the 4 files to a USB stick and then figure out how to get to it in UEFI shell. Most examples are just type “fs0:” but in my case to turned out to be “fs1:”.

Good luck!

[neofusion]

@Maxdiken

I am already well aware of the risks of virtualizing TrueNAS on Proxmox.

I know you said you knew of the risks of virtualisation, but in an abundance of caution I want to make sure you actually do know how to handle it.

Have you done the blacklisting (or equivalent) required to keep Proxmox from automatically importing the ZFS pools that will “suddenly” appear on the HBA after you moved them there?

[Maxdiken]

Apparently it seems my awareness is not as complete as I thought it was… Could you please elaborate on that?

[neofusion]

Here’s a story of what can happen if Proxmox and TrueNAS imports a pool at the same time:

Can't import datapool

Hi all, I have a server that is showing the datapool is FAULTED. Luckily all the disks are showing back online. I have this setup in ProxMox as a VM. I’ve tried several commands “zpool import data -FXf, zpool import -f, etc”. I can’t get it to import. Anyone know how to fix this issue? See the screenshots. 16 disks total. Thanks! [image] [TN 2]

Here’s what Proxmox says about blacklisting:
https://pve.proxmox.com/wiki/PCI(e)_Passthrough#_host_device_passthrough

[Maxdiken]

Thanks for the heads-up, this is definitely something that shouldn’t be overlooked.
I just spent 6 hours trying to update the firmware. Fortunately, I ultimately succeeded.
Now onto working on the blacklisting shenanigans… and then hopefully everything will be ready for me to connect the drives to the HBA.

If anyone could confirm if the process I detailed earlier is the correct way to go about things, assuming every preliminary step is taken care of.

[Fleshmauler]

After blacklisting & understanding the risks/precautions to take? Your list of steps is solid.

Only things I’d add: I have no clue if any gotchyas already exist in terms of how TrueNAS identifies the disks vs how proxmox is currently presenting them vs how they’ll be identified when they are fully accessible to TrueNAS after passing through the HBA.

With that being said - if there are no already underlying issues, then you wouldn’t even need to disconnect the pool, TrueNAS will see the disks & just automagically work with them.

If there was some nonsense going on, it might not hurt to back up your files. Maybe TrueNAS won’t see the disks the same way & it just means you’ll have to re-import the pool, or re-import it under a new name. Maybe it won’t be able to import & you’re stuck making a fresh pool & sure wished you had a backup.

I’m saying this out of an abundance of caution instead of actually trying to imply that I’d understand what would possibly go wrong or how to fix it. Recent examples of folks having hardware raid arrays or other unpleasantness make it feel worth noting that non-standard/recommended setups, have non-standard issues/risks.

[richardm]

Moving four drives from pass-thru motherboard SATA to a pass-thru LSI (in IT mode) was totally uneventful for me. I shut down, moved the four SATA connections, and fired it up. Everything started like nothing happened.

I’m not sure I understand the necessity for using an EFI boot environment (or worse) for flashing LSI cards. All required tools are available for Linux and I ran my flashing straight from the TrueNAS console with the LSI as a pass-thru. Are the Linux tools not trustworthy?

[Fleshmauler]

Sometimes they don’t work & then eufi is needed - otherwise that is the preferred way.

[Maxdiken]

Alright so I finally moved the drives to the HBA and TrueNAS recognized them instantly.
Unfortunately, I am getting very limited transfer speeds now. A very steady 10MB/s, while my network bandwidth could (and still can) handle up to 115MB/s when the drives were connected directly to the motherboard. What could be the reason behind that?