TrueNAS system no longer booting at all: Questions!

TrueNAS General
1

Hi all,

I’ve been a long time user of the previous forums and just made an account here. I have a bit of a unique situation and, thus, a few questions.

A few days ago, it seemed like there was some kind of power outage or my USP is going bad and my TrueNAS system lost power and shut down along with my main computer.

Upon rebooting my TrueNAS system, everything turns on, fans, lights, everything sounds normal except it no longer goes into POST. It just stays at the very first screen that comes up when you first turn the system on or sometimes the video wont even turn on. However, every time my server turned on it would do a series of beeps and then go into POST. It no longer does this. And of course, as a result, I cannot access any of my files or my plex server or any of that. After some time of the machine being on but not posting, it will sometimes beeps 3 times but unlike any other beep I’ve ever heard it do. Unfortunately, I don’t have recording of it but I’m sure I can get it if you guys need to see it or hear it.

I am wondering what could be wrong? I’ve been looking online and have tried a series of things. I tried to diagnose with the diagnostic LEDs on the mobo and the code I’m getting is:

“4Fh - Dxe IPL started”

It won’t let me post links here but if you want to look up my motherboard you can search “Intel S2600CP Manual” and that diagnostic page is page 210. I would link it but it won’t let me make this post with a link here. Happy to send it to anyone interested in looking into it.

I can attach a picture of those lights being on and off if that would be helpful. System Status LED blinks Green. I DO have access to my BMC Web Console but everything checks out as healthy and normal.

From reading online I have read that it could the the CMOS battery that went out. I tried switching it and it made no difference. I also read that it those 3 beeps is RAM error. Despite my BMC Web Console not showing any errors on my RAM, I took out each RAM stick one by one and tried to boot to no avail. Could my Power supply be faulty? I am feeling that my UPS is faulty, because this is the second time that everything connected to the UPS has just been shut off and there was no power outage or anything.

Should I purchase a replacement motherboard & try that? Would my system be able to boot right up if the only thing changed was the motherboard? Will I have to reinstall TrueNAS and if I install a newer version, will my pool import just fine?

Hoping someone can help.

Thank you for your time! Please let me know if you need any further information from me

Lastly, here are the specs of my machine:
TrueNAS-13.0-U5.3 Build:

CPU: 2 x Xeon e5-2670
CPU Cooler: 2 x Noctua NH-U12A
Mbo: Intel S2600CP2J
RAM: Hynix 128 GB Kit
Boot Drive: 2 x Kingston 120GB A400, mirrored.
HDDs: 8 x Western Digital 14TB Ultrastar DC HC530
PCIe: Intel x540-T2 10Gbe Dual port NIC
PCIe: LSI 9211-8i SAS-Sata 8 port PCI-e card
PSU: EVGA Supernova 750W 80+ Bronze
Case: Fractal Design Meshify 2 XL Black
UPS: CyberPower OR1500LCDRM1U 900W

2

bypass the UPS by plugging directly to regular power. That should tell you if it is a UPS supply problem.

1 Like
3

After CMOS battery change did you try a clear CMOS too?
You should try to remove all that Is not necessary for boot the system, keep only CPU - mainboard - RAM, plug directly without UPS as already suggested, and see if something change… As you already tried the RAM rotation you can even brutally try without RAM or CPUs for see if motherboard detect that or BIOS fail before.
Because, in my little experience, i would not understimate that can be the BIOS itself that Is corrupted, and if you are lucky you can try to reflash it (if you have access or with the right programmer) before think about a replacement. If the chip Is still good there are good chance to fix, but Is not and the chip Is soldered will be hard.

If the mainboard Is same model, It will work without issue. But if not, probably you will need to just reconfigure network interface.
Having a config backup Is a must for situation like yours, if you don’t backup already It regulary after all this situation Is solved get a look to joe’s multi report script from resource section

4

Thank you for responding. I already did this I plugged straight into a wall, into a power strip, same result. The UPS is becoming faulty because there’s no power outage but its losing power then beeping and won’t turn on for a while, it’s weird. This never happened before. Also If I cut the power it won’t keep my systems on like it’s supposed to even if just for 20 mins.

5

After CMOS battery, I did try to clear my CMOS too. I tried removing everything and only kept CPUs to boot and nothing.

I’m starting to wonder if maybe my PSU got damaged or something. I am so clueless, I’m just shooting in the dark.

A corrupted BIOS would suck, not sure how to reflash but I know I can figure it out.

I have a config backup, but I also have all my configurations written out so it wouldn’t be a problem to reconfigure TrueNAS. I just want to make sure I can get my data. I have a backup, but it wouldn’t be ideal to go that route.

Can you link me to “Joe’s multi report script”? I’ll look myself too, this new forum is just my first time really using it.

Thank you for taking the time!

6

Question: Do you guys think TrueNAS scale is Superior to Core? If I have a Core created pool would I be able to import it to Scale just fine?

Thinking about making the switch…

7

Unless any disk got damaged, data and pools should be fine.
If you have a spare PSU try it, but from your desc at least to me not seems a fail PSU

Btw this Is the script mentioned above.

IMHO what superior mean for you :wink:
I sidegraded very recently, there are pro and cons… Personally i faced some minor problem and besides one i can’t still resolve (TLS on ssh), all seems run smoothly. But im a simple home user, can be different for everyone.
In your place i would think about Scale migration after have fixed that PC :laughing:

8

The folks here have provided excellent advice. Replacing the power supply is likely your last hope.

The power outage could have caused it regardless of the UPS being there to protect it. Power spikes happen. I hope replacing the power supply gets the system back up and running. Of course if you suspect the UPS is going bad, I’d replace it when you can afford to.

I have to agree, the Initial Program Load seems to be the correct error code for your motherboard.

When you have tried everything you can think of, you could start to remove parts. Remove all add-on cards in the system, bring it back to factory stock.
Remove the second CPU and RAM that goes with it (assuming it will run with a single CPU). If that fails, then swap the RAM out from the first CPU and second CPU group. There are a lot of things you can try however it will be time consuming.

I’m not sure what your use case is however if it is mainly a video library and some personal information and backups, not a heavy duty workhorse then maybe this is a blessing. It looks like this system eats up electricity and I’m certain you could purchase something significantly less power hungry. I’m just trying to find a silver lining, if there is one.

9

Thank you for chimming in. You’re one of the GOATS here I see!

I replaced the PSU yesterday and, unfortunately, no better luck.

Would you or anyone else think the mobo may need replacement at this point? Just having a hard time getting to the bottom of things.

I use my case for all that you mentioned, plex server, open vpn, I have all the radarr, sonarr, etc apps, transmission, next cloud, etc. I like to play with things and implement things into my workflow to make things easier and its just a hobby. I also video edit off of it directly through my 10 GBe connection but I haven’t figured out how to have my machine export the edit. Lastly, I usually have at least 1 virtual machine running. Nothing too fancy. Would your recommendation of purchasing something significantly less power hungry still stand?

I’ve considered it and it would possibly simplify my life. I’ve thought of purchasing a truenas mini many times in the past. But they no longer have the tower with the 8 bays. I need at least 8 bays and I’m not sure I want a server rack in my office. I’ve done that in the past and it was fun while it lasted but I definitely appreciate a smaller footprint now.

10

Mini R would get you the 10GBase-T but tops out at 64GB of RAM for order. I don’t know how much it can take reliably. Some Mini users have put in more. 167 watts with all drives and add in 10G card, per the specifications page. It also gets you four extra drive bays.

I guess just look at what options are on the market. There are even odd items Vertical Wall Mount Rack 1U - 4U at RackSolutions
I just noticed they show an example of under desk mounting too
https://www.racksolutions.com/server-racks/wall-racks.html

https://www.truenas.com/truenas-mini/

11

Just out of curiosity, most server motherboards have a physical jumper to reset the bios (unlike consumer boards where you can pull the power & the battery & have it reset)…

Did you try all the following stuff that I found on Intel's website & am merely assuming applies to you? 
8.1.1 Set BIOS to default (Clearing the CMOS)

To clear the CMOS, perform the following steps:

1. Power down the server. Do not unplug the power cord.

2. Open the server chassis. For instructions, see your server chassis documentation.

3. Move jumper from the default operating position (covering pins 1 and 2) to the

reset/clear position (covering pins 2 and 3).

4. Wait five seconds.

5. Remove AC power.

6. Move the jumper back to the default position (covering pins 1 and 2).

7. Close the server chassis.

8. Power up the server.

The CMOS is now cleared and can be reset by going into the BIOS setup.

Note: Removing AC power before performing the CMOS clear operation causes the system to

automatically power up and immediately power down, after the procedure is followed and AC

power is re-applied. If this happens, remove the AC power cord again, wait 30 seconds, and reinstall

the AC power cord. Power up the system and proceed to the <F2> BIOS Setup utility to

reset the preferred settings.

 

and in the latest bios package : BIOS_R02.06.0007_package_Romley_Platforms

there is a : Update_Instructions.txt

that has :

 

 

*******************************************************************************

BIOS Recovery Process

*******************************************************************************

In the unlikely event the BIOS is corrupted, leaving the system in an 

unbootable state, it may be necessary to perform the following BIOS Recovery 

procedure.  

 

1. Copy the following BIOS update files to the root directory of a USB media 

device (Recovery Media):

Note: These specific files will be located in the "BIOS Recovery Files" folder 

included in the System Update Package.

 

- IPMI.EFI 

- IFlash32.EFI 

- RML.ROM 

- R##.##.####REC.CAP (where ##.##.#### = BIOS revision number) 

- STARTUP.NSH (modified to support BIOS recovery)

 

 

2. Power OFF the system

 

3. Locate the BIOS Recovery Jumper on the server board and move the jumper 

block from pins 1-2 (default) to pins 2-3 (recovery setting)

 

4. Insert the recovery media into any available USB port on the back of 

the server

Note: The front panel USB ports may not work for this procedure on some server 

systems.

 

5. Power ON the system

 

6. The system will automatically boot into the embedded EFI Shell

 

7. The STARTUP.NSH file automatically executes and initiates the flash update.

When complete, the IFlash utility will display a message

 

8. Power OFF the system and return the BIOS Recovery jumper to its default 

position

 

9. Remove recovery media from the USB port

 

10. Power ON the system 

 

11. Do *NOT* interrupt the BIOS POST during the first boot.
12

Thank you for this suggestion and for linking that desk mount! I’m not sure it’s for me still but that is way better than having a server rack in the room. haha.

13

I have tried resetting the CMOS. I should go through the entire recovery process though in the case that the problem is my BIOS is corrupted. I did not look too in depth but I just didn’t find the BIOS files for my mobo. I need to look again and with more time.

I am going to take everything apart and test it outside of the case to really dive in, just haven’t made the time. I appreciate you sending this though I will give this another crack. If those instructions had a link for the actual bios package please link it so I can try it with a USB.

I have done this once in the past but I can’t remember where I got the BIOS files from.

14

It doesnt help that the mobo support has reached end of life, that is what made me feel like I couldn’t find the BIOS files, but I’m going to look better now.

15

Don’t know if it will help. System Event Log viewer. Maybe iBMC log will show something
https://www.intel.com/content/www/us/en/download/17948/system-event-log-sel-viewer-utility.html

16

https://www.intel.com/content/www/us/en/download/19271/firmware-update-package-update-for-efi-intel-server-boards-and-intel-server-systems-based-on-intel-60x-chipset.html

Newest version says it doesn’t have recovery files (which is odd because I swear it has a .cap file in the recovery folder). However it advises to use the recovery files from the older version which you can get by click the drop down & selecting E006

image
image691×369 21.1 KB

Edit: Mind you intel doesn’t list CP2J as an applicable varient, but the page for CP2J bios download straight links to that same page. I guess they can’t always get it right, but seems like your best bet.

17

I have access to my BMC log in and can see this. Did you want me to share anything? I’d be happy to.

18

Thanks for linking this!

I just found this before you sent this, I downloaded all 3. However, I could NOT find recovery files for the E006 versions. Let me show you with screenshots

image
image1333×984 59.2 KB

image
image1338×982 59.8 KB

image
image1341×985 34.2 KB

I am currently on Bios: SE5C600.86B.02.06.E006

I would love to use the recovery files in the 007 version but could that brick my mobo?

Essentially, I’m seeing the opposite of you, the only file that has the recovery file is the new one, and the 006 versions don’t have it in the files.

19

I thought the log view may help in determining a problem. Just something to try.

1 Like
20

Yeah saw the same with only 007 version having recovery. IMO if you’re on version 006, I’d just flat out update to 007 instead of trying with recovery. Would in theory over-write possibly corrupter bios.

I’m guessing recovery versions are only in place because there are checks prior to installing bios for current version vs targeted install version (where if same version or older it’d reject the installation, while recovery has an overwrite).

That being said, who knows, maybe that was never the issue.

Side note - the documentation on Intel’s website is all over the place, so best effort is in effect.

2 Likes