For completeness I’ve reinstalled the IR card. I’ll replace it with the IT 16.00.20.00 soon.
Avago Technologies SAS3 Flash Utility
Version 16.00.00.00 (2017.05.02)
Copyright 2008-2017 Avago Technologies. All rights reserved.
Adapter Selected is a Avago SAS: SAS3008(C0)
Controller Number : 0
Controller : SAS3008(C0)
PCI Address : 00:01:00:00
SAS Address : 500605b-b-1218-0145
NVDATA Version (Default) : 0e.01.00.08
NVDATA Version (Persistent) : 0e.01.00.08
Firmware Product ID : 0x2721 (IR)
Firmware Version : 16.00.10.00
NVDATA Vendor : LSI
NVDATA Product ID : SAS9311-8i
BIOS Version : 08.37.00.00
UEFI BSD Version : 18.00.00.00
FCODE Version : N/A
Board Name : SAS9311-8i
Board Assembly : N/A
Board Tracer Number : N/A
I’m researching to see if there is another firmware update, but I know I’ve updated it from the initial. I’ve always, always preferred AMD systems but the prices of the Intel processors are so low, I thought I’d take a chance. I HOPE this doesn’t bite me.
Latest microcode update came out like late nov early dec of last year if I remember correctly… Good idea to check
Thanks everyone for the suggestions. I’ve discovered a couple of items.
- Firstly, this system is a, ASUS ROG Strix Z790-I Gaming WiFi 6E LGA1700 (Intel 14th,13th&12th Gen) mini-ITX gaming motherboard. Sorry for the confusion thinking it was a 760-I.
- Second, there is a BIOS update which I have now updated to Version 2801, 2024/12/06
The only BIOS settings I’m making other than optimized defaults is to disable WiFi/Bluetooth and to disable the ‘enhanced PCIe power management’ setting.
Will watch it to see if anything changes…
The issue is supposed to be confined to overclockable ‘K’ CPUs. Non-K, as here, have lower power limits to begin with.
You might be right, but I even hear that laptop skews were impacted, which are wattage capped, especially compared to desktop cpus.
Part of the issue was high voltage on low utilization, not just wattage cap. You could be right, but I just don’t trust anything 13th/14th gen.
Quick update.
After the firmware update and above mentioned actions, the server has been up for 16 hours without restarting. I’ve been using it normally but will start pressing it harder with file transfers as a test.
But, this is longer than it has stayed up for quite some time.
Thanks for all the eyes.
Another quick update.
The server has been up for over 24 hours and has been heavily used. I think the firmware update was the fix. I will update the remote server as soon as I can. I will also trade out the 3008 with IT at 16.00.20.00.
It would be interesting to understand this more and I’m saddened by the lack of system reporting. BUT… I’m not complaining.
Again. Thanks.
I mean, if I had to guess it was 100% related to microcode issues on the 13th gen if bios update did infact do the needful.
I only thought of it after I saw 13th gen mentioned. While the i9s had it the worst (based on personal experience), I’m guessing voltages were all over the place for even lower cpus…
So, I keep updating just to let everyone know. The server was up for 4 days without problem - it was clearly the firmware update. But as promised, I got an updated LSI3008 and installed it today:
Avago Technologies SAS3 Flash Utility
Version 16.00.00.00 (2017.05.02)
Copyright 2008-2017 Avago Technologies. All rights reserved.
Adapter Selected is a Avago SAS: SAS3008(C0)
Controller Number : 0
Controller : SAS3008(C0)
PCI Address : 00:01:00:00
SAS Address : 5003048-0-1b04-cf01
NVDATA Version (Default) : 0e.01.00.07
NVDATA Version (Persistent) : 0e.01.00.07
Firmware Product ID : 0x2221 (IT)
Firmware Version : 16.00.12.00
NVDATA Vendor : LSI
NVDATA Product ID : SAS9300-8i
BIOS Version : 08.37.00.00
UEFI BSD Version : 18.00.00.00
FCODE Version : N/A
Board Name : SAS9300-8i
Board Assembly : ARTofSERVER
Board Tracer Number : N/A
I’ve got a M.2 10Gb adapter on the way to try and address the 2.5Mb limitations of the B790. Will let you know how it goes. BUT, the server is working very well at this point. Intel or not 
Thanks all.
Another update. Hopefully I’m not bombarding the thread about this…
Because I’d become confident about the stability of the server, I loaded photoprism to hammer on the server and do a full index on several thousands of pictures. Let me tell you my experience:
- First I loaded photoprism as a container on the TrueNAS server and started it on the photos. The container was hammering the local ZFS datasets and I had allocated a far amount of resources. It ran for about 10min and the the server crashed completely twice.
- Second, I thought perhaps the load was just too much on the container engine, deleted photoprism and instead loaded it on a local virtual server and allocated 64 cores, connected it via a SMB3 share to the TrueNAS server. When it indexed this time, the TrueNAS ethernet was pegged pretty hard and after about 30min it crashed again! I de-tuned the virtual server running photoprism and it slowed the load on the TrueNAS server and so far it has not crashed again.
I don’t know if it’s because of TrueNAS being super tender and I’m expecting too much or if it’s the Intel (consumer-grade) system. Unfortunately I’m completely guessing because I don’t know how to see any logs or audits of what it’s reporting. I’m trying to not just assume it’s because it’s a crappy Intel system, but…
As previously mentioned I’m awaiting the arrival of a m.2 10Gb adapter which will move me away from depending on the B790 network interface, but honestly my confidence is pretty low. On a positive note, ZFS has protected all of the data and thus far is working as it should.
All software is fully patched with the latest software including the TrueNAS 24.10.2 update and all applications. If anyone has any suggestions on how to debug the crashes, I would very much welcome it.
If non of this works, I will be on a quest to replace the ITX systemboard with an AMD version - but everything I see is still consumer grade.
In the end, I hope others might benefit from my pain. I just wish I had more insight to what is actually happening here.
Thanks all.
Server. Mini-ITX. Socketed. Pick ANY TWO.
I still suspect it is 13th gen nightmare. Not sure how much fun you have overclocking (ie, undervolting, and possibly underclocking), but that is what “fixed” my issues.
You could do some stress tests on another OS to at least confirm beyond doubt if it is a hardware issue or not.
Thanks for all the suggestions on this thread.
Just wanted to let all know that BOTH of these systems have Intel CPUs that are listed on the Gen 13th & 14th bug list. The 0x125 has made the systems slight more stable, but they are still crashing under load.
As an avid AMD fan for years, the irony of this happening to the first two Intel systems I’ve bought isn’t lost. I had purchased them because the compute cost seemed compelling. I’m paying for it now.
I’m researching on replacing both of the systemboards with MINISFORUM Motherboard BD795i SE Mini ITX NAS Motherboard with an embedded AMD Ryzen 9 7945HX and ECC reg memory. Does anyone have experience with these systemboards?
Also, has anyone moved from Intel to AMD with TrueNAS Scale? I’m guessing it’s safest to reinstall?
Thanks.
It’s possible you can get replacement CPUs on warranty.
Yeah, I’m looking into this - but the ridiculous issue is that you have to report the SN and the FPO number to register the RMA. I can get the SN in software but the FPO is only accessable by removing the CPU and reading it off the back.
If I have to take apart the whole system, I’m not going to be reinstalling the Intel system! Additionally, I’m hearing the ‘fixed’ batches are also problemitic. What an absolute mess. I honesly don’t see how Intel is going to survive their endless [self created] crisises.
Anyway - don’t mean to rant.
12th gen. is not affected by the issue.
So a quick, and hopefully cheap, way to get out of this would be to buy a pair of 12th gen Core, install these, send the 13th gen to RMA and put the replacement parts on small ads to recover the cost of the Alder Lake CPUs.
Good idea. I ordered the AMD system for the main server and 12th Gen for replication server. What a mess. Thanks all.
Wanted to close this topic out with the following:
As a recap I had two independent TrueNAS Scale servers at different locations that started unexpected rebooting. After some research and all software updates, I discovered that both of servers had 13th Generation Intel CPUs with a design flaw that causes the problem.
I’ve replaced one of the CPUs with a generation 12th Generation CPU since it supposably was outside of the CPU bug. I left everything the same with a fully updated system board etc. Unfortunately, the system is STILL rebooting every few hours.
On the primary server I replaced the system board and CPU with a MINISFORUM Motherboard BD795i SE Mini ITX NAS. I also replaced the memory with ECC Reg memory. The BIOS is rather primitive, but I was able to move the M.2. storage from the Intel system to the AMD system with no issues.
I did notice that even with a fresh TrueNAS install on the MINISFORM system it had problems with secure boot. I’ll research this at a later date. For the moment I’ve disabled secure boot to support the existing install.
The performance of the system is significantly and noticeably better. Weirdly I’m achieving much faster Ethernet performance (which was further improved by a M.2 10Gb).
I absolutely recommend the BD795i SE system. The embedded system board is cheaper than the single Intel 12 CPU I bought. I’ve always been an AMD fan, but this situation has really reenforced this position.
Just wanted to close out the thread. Now I’m going to try and some sort of replacement/refund from Intel (right…)
Thanks