If you have a spare monitor, attach it to the VGA output of the server itself (if there is any) and check out the screen. TrueNAS puts out system messages time after time, maybe those might help you too.
2 Likes
Thanks again to everyone for the advice!
Following up: after trying all of the combinations of hardware I could think of: swapping RAM sticks from mobo and 2nd CPU riser card, swapping the CPUs themselves, removing all 8 hard drives to reduce PSU load, adding shims to push the CPU riser card further into the mobo sockets… The only thing that has worked long term is just removing the 2nd CPU riser card entirely.
I’m still totally perplexed that the system was able to do full diagnostics and over a day of memtest86+ (which I believe does jump around to use all available cpu cores) but as soon as a Linux OS is loaded it would keel over. At least the system is back up and running again with the single cpu.
Try a bottle of contact cleaning spray, and sray it plentyful on the connectors between the boards.
It might be a nasty, corrded pin.
Also, check if all pins are straight and inline.
Also, free version of memtest is quite a bastard…
For example, it does not see or handle any CPUs correctly that is more than 16 cores. (The stuff freezes, during the “get SMP data” step at booting.)
The other thing I recognised it that independently from my settings of using multiple cores, it actually only uses 1,
Does not use HT at all.
I had to go to BIOS and allow only 4 cores/8 threads in the CPU. (But, since it is a E5-2650L, it only clocks to 1,69GHz, so, imagine, that a single test with 128GB of RAM is running more than a week now…)
However, I am not convinced that this is not because of the janky BIOS of my chinese motherboards…
Also, I dont know, it it used dual CPU at all.
So, I just bought a shitty CPU with only 4/8 cores, but clocks up to 3,5GHz. (it was like 5 EUR 
)