I recently built my first NAS. I decided to get 10Gbit networking right away. Unfortunately the X540-AT2 I got does behave very weirdly.
First I installed TrueNAS Core. It wasn’t detected at all. Neither in BIOS nor in TrueNAS. Thinking the NIC must be damaged I didn’t bother testing it further. But I got curious. So I tested another X540-AT2. Same thing no detection. Then I tried my luck with a Realtek NIC (RTL8125). Still the same. No detection in the BIOS and in TrueNAS.
After that I thought maybe TrueNAS Scale would work as it’s Linux based instead of FreeBSD. But still no success. No detection. Neither in BIOS nor in TrueNAS.
Thinking it may be hardware related I tried another platform. Different Mainboard, different CPU and RAM. Furthermore I tried the hardware with Windows. The RTL 8125 was detected right away and the X540-AT2 NICs just neded some drviers to work properly.
I hooked up the TrueNAS boot drive to my Testbench and look at that, all three NICs worked perfectly. I then reinstalled one of the X540-AT2 NICs in the NAS and it worked flawlessly. I was even able to back up all my data (about 300GB) via the X540-AT2 without any problems.
But after rebooting my NAS it stopped working again. I noticed that when I leave the NAS completely turned off (power switch on the PSU off) for some time it kinda works again. But not every time. I can’t wrap my head around that. Why does it do that? Is there a way to reinitialize the PCIe Bus? Maybe the PCI Bus isn’t correctly initialized or whatever. For me it doesn’t look like a typical hardware problem. Or is the firmware on my NIC faulty?
NAS configuration:
PSU: Silverstone SX500-G
Mainboard: Z390I Aorus Pro Wifi
CPU: i7 9700K
RAM:Teamgroup-UD4-2133 32GB Kit (2x16GB)
M.2 to Sata HBA: ORICO M.2 PCIe M-Key auf 6 x SATA Adapterkarte, SSD Riser Karte bis 6 Gbit/s Unterstützung RAID für Desktop,Linux,Windows- PM2TS6
This is how the NICs show up in Windows Device Manager on my Testbench. They work just fine there. I even confirmed that they negotiate at 10G although it looks like it might not be a genuine Intel NIC. I tried another X540-T2 (not AT2) NIC in my NAS. As it was before, it was detected at start up but wasn’t when TrueNAS was running and after some restarts. The only difference was that it was shown as “Intel Ethernet-Converged Network Adapter X540-T2”. As expected. A Yottamark ia missing. So maybe it’s a replica as well. But every NIC I tried does work under Windows on my Testbench and on my NAS. The weird thing is, it worked at some point. Could my HBA cause problems?
My idea is that the mainboard - targeted for gamers and maybe office - only expects a graphics card in the x16 slot. Also the ACPI tables could be that bad that both FreeBSD and Linux have problems with them. Both are problems i have heard of but not seen for myself.
Maybe a BIOS update may help, if there is a newer version.
Yes that’s my board.
The NIC is in the only PCIe Slot.
I already updated the BIOS as I suspected that already.
If the Mainboard would only expect GPUs in the PCIe Slot the NIC should never work. I would at least expect so. But sometimes it works. The last thing I noticed, when the PSU was fully turned off for long enough, was that the NAS launches directly to the BIOS. The NIC is detected then. When restarting it boots to TrueNAS. Then the NIC won’t be detected.
My boot drive is connected via the HBA card in the M.2 Slot. Maybe there is a problem with it that somehow prevents the PCIe Slot from being correctly initialized or whatever. I will try connecting my boot drive to one of the onboard SATA ports of the motherboard. I’ll report the results.
I tried connecting my boot drive directly to a Sata Port on the mainboard and it works now. The X540 is detected and negotiates at 10G as it should. Bandwidth is also as expected. I don’t know what exactly happened when it was connected via the HBA, maybe something with ACPI but whatever. My conclusion is, always connect your boot drive to an onboard Port not an HBA or anything else.
