Updated BIOS, Disks MIA in TrueNAS

BoulderItalyc · July 20, 2025, 10:00am

I had a working system running Scale 24.10.2.2. It Has an Asus ProArt Creator B650 motherboard and an LSI 9300-16i HBA. There are thirteen 12 TB SATA HDDs connected to the HBA via an internal backplane and SFF-8643 cables. The system was working fine. I updated the motherboard BIOS. After doing that the HBA and disks show up in the BIOS, but do not show up in TrueNAS at all.

The closest post I have found so far is;

but I’m not using a disk shelf or trying to run an HBA over DAC or whatever their config was.

Here is some output for some commands requested in that thread;

lsblk -bo NAME,MODEL,PTTYPE,TYPE,START,SIZE,PARTTYPENAME,PARTUUID
lspci
sas2flash -list
sas3flash -list

admin@Elephant[~]$ lsblk -bo NAME,MODEL,PTTYPE,TYPE,START,SIZE,PARTTYPENAME,PARTUUID
NAME        MODEL        PTTYPE TYPE    START         SIZE PARTTYPENAME             PARTUUID
nvme0n1     Samsung SSD  gpt    disk          250059350016
├─nvme0n1p1              gpt    part     4096      1048576 BIOS boot                35d34cbd-3832-4d9e-96cc-8d76d56cfe42
├─nvme0n1p2              gpt    part     6144    536870912 EFI System               7989cf6c-0049-454c-a287-575643b2fce5
├─nvme0n1p3              gpt    part 34609152 232339447296 Solaris /usr & Apple ZFS 3010fbc4-ceb9-4a3e-a521-1145976514b5
└─nvme0n1p4              gpt    part  1054720  17179869184 Linux swap               9a7127da-573a-4105-ba63-c03d06736d9f
admin@Elephant[~]$ lspci
00:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 14d8
00:00.2 IOMMU: Advanced Micro Devices, Inc. [AMD] Device 14d9
00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 14da
00:01.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 14db
00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 14da
00:02.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 14db
00:02.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 14db
00:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 14da
00:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 14da
00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 14da
00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 14dd
00:08.3 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 14dd
00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 71)
00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 51)
00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 14e0
00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 14e1
00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 14e2
00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 14e3
00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 14e4
00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 14e5
00:18.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 14e6
00:18.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 14e7
06:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Upstream Port (rev 01)
07:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port (rev 01)
07:08.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port (rev 01)
07:09.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port (rev 01)
07:0a.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port (rev 01)
07:0b.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port (rev 01)
07:0c.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port (rev 01)
07:0d.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port (rev 01)
08:00.0 Ethernet controller: Intel Corporation 82599 10 Gigabit Network Connection (rev 01)
0a:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 15)
0b:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller (rev 05)
0d:00.0 USB controller: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset USB 3.2 Controller (rev 01)
0e:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset SATA Controller (rev 01)
0f:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM961/PM961/SM963
10:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Raphael (rev c4)
10:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Rembrandt Radeon High Definition Audio Controller
10:00.2 Encryption controller: Advanced Micro Devices, Inc. [AMD] VanGogh PSP/CCP
10:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Device 15b6
10:00.4 USB controller: Advanced Micro Devices, Inc. [AMD] Device 15b7
10:00.6 Audio device: Advanced Micro Devices, Inc. [AMD] Family 17h/19h HD Audio Controller
11:00.0 USB controller: Advanced Micro Devices, Inc. [AMD] Device 15b8
admin@Elephant[~]$ sudo sas2flash -list
[sudo] password for admin:
LSI Corporation SAS2 Flash Utility
Version 20.00.00.00 (2014.09.18)
Copyright (c) 2008-2014 LSI Corporation. All rights reserved

        No LSI SAS adapters found! Limited Command Set Available!
        ERROR: Command Not allowed without an adapter!
        ERROR: Couldn't Create Command -list
        Exiting Program.
admin@Elephant[~]$ sudo sas3flash -list
Avago Technologies SAS3 Flash Utility
Version 16.00.00.00 (2017.05.02)
Copyright 2008-2017 Avago Technologies. All rights reserved.

        No Avago SAS adapters found! Limited Command Set Available!
        ERROR: Command Not allowed without an adapter!
        ERROR: Couldn't Create Command -list
        Exiting Program.

From the BIOS:

Thoughts?

QonoS · July 20, 2025, 10:14am

Did you make certain BIOS settings for SAS9300-8i HBA? Or was there some notes in BIOS changelog about PCIe or so?

BIOS updates usually result in BIOS defaults being loaded, so all settings that were made have to be made again, if any.

Of course you could just flash back and see if that works.

joeschmuck · July 20, 2025, 11:23am

I agree with @QonoS that it is most likely a BIOS setting that needs to be changed.

I recommend you do the following…

Reset the BIOS to Factory Defaults, see if that works.
Is CSM enabled or disabled? Make it the opposite and reboot to see if that works.
The fact that your drives show up and are listed in AHCI is good I believe.
Are you running TrueNAS on bare metal or in a Hypervisor?
What slot is the HBA plugged into? You have 3 to choose from but only the slot closest to the CPU is capable of PCIe x16. Move the card if you need to. Try again. (The middle slot is x8, farthest away from CPU is x4.
Item 4 also means to look at the BIOS settings for the slot you are using. If it worked before, odds are the BIOS setting for that slot changed.

etorix · July 20, 2025, 12:36pm

The HBA does not shows up in lspci, so it is nomal that sas3flash -list finds nothing.
AHCI woud list drives attached to the B650 SATA controller through motherboard ports.

I’d suspect you have the 10G NIC in the upper slot, the HBA in the middle and the update has reset bifurcation to x16/x0 instead of x8/8. But what are these AHCI drives which do not show up in lsblk?

What are the chassis and backplane?

BoulderItalyc · July 20, 2025, 3:36pm

Thanks @etorix and @joeschmuck. To quickly answer some of your questions;

I did a Clear CMOS, no change.
CSM makes no difference
TrueNAS on bare metal
I had; GPU in x16 slot, HBA in x8 slot, 10G NIC in x4 slot.
There is no explicit “x8x8” setting in the current BIOS. The wording is that the x16 slot is “auto” and the x8 skit is “x8”. If both cards are present, it is supposed to switch the x16 slot to x8.

I don’t believe I had to do anything before to get the HBA and GPU working together. I think it was just plug and play.

As suggested, I did flip the GPU and HBA (So the HBA–which is just a x8 edge connector-- is in the electrical x16 slot, and the GPU is in the electrical x8 slot) and that does get me the drives in TrueNAS. So this seems to be a bug in the updated UEFI.

Unfortunately, it’s a 2.5-slot GPU, so I can’t have it in the x8 slot and the NIC in tthe x4 slot. I’ll need to live with gimped networking, or see if I can use kapton tape to mask 8 of the PCI channels on the GPU.

Rolling back is not an option, I tried, and they do state this in the release notes. It’s probably because of the CVEs the update is supposed to fix.

etorix · July 20, 2025, 4:06pm

Problem solved: Asus botched the PCI bifurcation logic in this BIOS update.
Your best option is to file a bug report with Asus technical support and hope for an update to fix it.
Second best option: Get a smaller GPU (for transcoding?).

joeschmuck · July 20, 2025, 4:52pm

Look very carefully at the BIOS settings. Do a Google search for something like “asus bios pcie x16 slot” and read. I find it hard, not impossible, to believe the BIOS is bad, it is more likely a setting changed.

I would not do that. Work on the fix you need which is the BIOS.

If you have to, can you flash with the older BIOS file? And I dislike a lot of flashing the BIOS, it is one of those easy to break everything situations.

BoulderItalyc · July 20, 2025, 5:12pm

It’s ASUS, not hard to believe. Chatting with ASUS support now and they are filing an escalation, not walking me through bifurcation settings. It’s completely a guess, but my guess is there may be some relationship to the CPU(s). That with AMD having multi generational sockets, as they roll out updates to handle the new CPUs they may break support of the older CPUs. But that’s just my guess on the regression.

I would if I could, but this particular update is one way. They have that warning with the release note (and I tried anyway, no luck).

BoulderItalyc · July 20, 2025, 5:18pm

Light AI. It’s not long–only 2 fans–but 2+ slots.

joeschmuck · July 20, 2025, 5:48pm

I’m familiar with it.

QonoS · July 20, 2025, 8:50pm

Have you tried this setting?

This should result in x8x8.
If PCIEX16_1 is set to “PCIE X16 Mode” the PCIEX16_2 will be disabled.

You can also try experimenting with forcing PCIe Link Speed:

BoulderItalyc · July 20, 2025, 9:09pm

Yes. You can’t force PCI/XEX16_1 explicitly to X8, you can only set it to Auto and rely on the BIOS logic to do X8/X8 based on the setting of PCI/XEX16_2. At least in the current BIOS. Not sure if that was the same before, but it worked before. The default, which is the screen shot you have above, worked before, I didn’t need to change it.

OK, if I get some time.

BoulderItalyc · July 21, 2025, 1:25am

Well this is working now, but net I don’t think I changed anything.

I did do the following;

I took out the HBA and NIC and put the GPU in the X16 slot just to test that basic configuration. It worked, was able to lspci the GPU in TrueNAS
I then left the GPU in the X16 slot and put the HBA in the X8 slot (as it was when I started) and went to the PCIE Link Speed menu and set the HBA slot link speed to PCIE 3

After the link speed change I was able to see both the GPU and HBA in lspci. I then set the link speed for the HBA slot back to auto and rebooted and still lspci showed the HBA and GPU. I added back the NIC and all three appear in lspci and the system seems to be back to the way it was 18 hours ago.

So, I don’t know if it was just gremlins or if putting the cards back in one by one, or toggling the link speed actually forced some sort of setting refresh that a CMOS reset didn’t.

I didn’t open the case or mess with any of the hardware as part of the BIOS updates, just plugged a thumb drive in and out,

QonoS · July 21, 2025, 7:21am

So there is obviously no bug. It was just the usual struggles that happen every now and then. Good that you kept trying!

joeschmuck · July 21, 2025, 1:35pm

One of the great mysteries in the universe.

etorix · July 21, 2025, 6:16pm

That probably did the trick by forcing the BIOS to acknowledge the PCIe 3.0 device vs. the (PCIe 4.0?) GPU. Botched logic…
At least, you know the procedure if BIOS is reset again.