So I have the HW listed in my signature running the latest TN. I originally had this running on 1G for 4+ years with the same HW with pretty much ZERO drops due to networking or anything similar. Now I recently upgraded to 10G using these on-board NIC’s (2 RJ45 (10GbE) by Intel® X550-AT2), and my 10G NetGear switch. I also have my Linux PC connected to this same switch with no issue. All cabling is brand new Cat 8 dedicated 6 feet patch cable directly to the switch so cant get much better. I first started with this 10G networking with the latest TN v25 4-5 months ago and continued through the latest 26 Beta.2.
The issue:
If I have this plugged into the 1G, same Cat 8 cable but connected to my 1G Cisco switch, it will stay on indefinitely. My last 1G uptime test was 30 days!. However when moving to 10G, the first test dropped within 2-3 hours, then second test in 24 hours. This has happened over 5 times while testing on 10G. The only load is simply testing immich site with only a few pages of photos not even getting close to the 10G limit. I tried disabling Tx Rx flow control but that did not really help. When it hangs, the entire host is hung, local console does not reply but still shows the TN cli display, so the only option is rebooting.
Any idea on this one? I could buy a dedicated 10G NIC but not sure why these on-board NIC’s would not work.
I would update your NAS sig to include all your info. Take Samuel_Tai as an example of the details.
Just to clairify, you are using only ONE 10Gbps port off your NIC? Have you tried the second port or swapping out network cables? All cables could be bad no matter how they are rated.
So its a new CORSAIR RM850x PS which should be plenty I would think. Its in the closet but the room is very cool, so would think its cool enough. I have not looked at those tweaks so will do that next thanks!
As far as the NIC ports, that was my plan maybe try the other port next, and sure I can try another cable. Will see what I find.
A server power supply prioritizes the 5V rails for disks. Most gaming power supplies prioritize the 12V rail for CPU/GPU. The power distribution table for this power supply confirms it.
Model
RPS0124
Part No.
CP-9020200/75-003898
AC Input Rating
AC Input
100–240V
Input Current
10A-5A
Frequency
47~63Hz
Max Load
Max Output
+3.3V
20A
150W
+5V
20A
+12V
70.8A
849.6W
-12V
0.3A
3.6W
+5Vsb
3A
15W
Total Power
850W
Only 150W total is supplied for your 9x SSDs. With the additional load from your 10G interfaces, your power supply is probably marginal for your setup.
Don’t guess. Look at the SMART temperature readings for your SSDs and the CPU temperature readings in the UI under Reports.
Thanks, so looking at each SSD they are all the same, this was about when it was last online with the 10G:
Key
max
mean
min
Temperature
32.3
28.5
25
CPU: All cores are within 1 deg of this:
Key
max
mean
min
cpu0
43
29.1
27
I am a little familar with the 5v vs 12v but not 100%. If I remember correctly this started to happen a few months back and back then I had a very old but good PC @ Cooling 500w PS. Since I was having this problem and added the GPU, I figured I should update the PSU to this model. I figured the SSD’s have very low power req’s, but maybe they do need more?
Nvidia 5060Ti, but since this is run off the 12v I should have plenty of power for it right. Its only used for occasional Immich ML workload, so prob not used at all during these 10G tests.
Unfortunately, I suspect this power supply doesn’t have an independent 5V rail, but steps down from the 12V rail. Your GPU is limiting the 5V rail’s below the rated 150W.
Does the IPMI still respond? I have the non-10gig version & it should show the voltage of the power rails coming in. Though sadly the logging sucks & only shows you the voltages while you have the page open, though I guess the alarms would show if anything went critical while you weren’t looking.
I haven’t experienced a PSU causing a hang, though I guess anything is possible. I think the power options Sam mentioned previously are more realistic. Since the hang happens pretty quickly, anything of note in top or dmesg if you check every hour or two?
The only other thing I can think of is the chipset. Mine used to get REALLY toasty, easy 75c on idle, & that was without additional load of 10gig. I slapped a 40mm fan on it & now it never goes above 45c.
Pretty sure IPMI is alive but just no TN console. Looking at live stats yea seems like 5v is about right. I dont see anything in logs showing other power related issues. Coincidentally after changing the BIOS settings such as disabling c-state seems like the GUI ssh shell is no longer working, its just the white cursor. I do have ssh from my PC, so not a huge deal but used to work every time until this latest BIOS change. I have tried different browsers and incognito windows on each, same results.
spoiler; it won’t be fun if you do see entries that aren’t boot related & then a wild @winnielinnie might appear to remind us the importance of memtests even on ecc.
Sadly our ipmi wouldn’t log ecc events & we’d be stuck using the OS
Also I noticed you have the same MB. I updated to the latest BMC and BIOS versions (notice a trend here, lol). Wondering if you considered it or have any thoughts. I figured this should only help esp since the original vers were much older.
