New server builder here - slowly approaching my wit's end - any help greatly appreciated

TrueNAS General
,
1

Hello everyone,

Let me first say that I’ve researched and scoured this site too for help, and it did lead me closer to an understanding of my issues. I have two issues that I’d like to discuss.

Specs:

MSI MPG B650I EDGE WIFI Mini ITX AM5

AMD Ryzen 7 9700

Kingston FURY Beast 32 GB (2 x 16 GB) DDR5-6000

MSI MAG A750GL PCIE5 750 W

Lenovo / IBM ServeRAID M1215 (LSI SAS9340-8i) - refurb.

3 x Seagate IronWolf Harddisk ST10000VN000 10TB (Storage)

2 x Intenso SSD Top Performance 256GB 2.5 SATA-60 (TrueNAS OS, mirror)

2 x SILICON POWER SSD UD90 2TB M.2 PCI Express 4. (Fast storage)

Noctua NH-D15S

My first issue - PCIe slot and missing HDD:

My PC cant recognize my last drive (my third HDD) which is connected via my ServeRAID card and a SAS-to-SATA (first i bought a EMRICK03G card but that was clearly not suited). The other 4 - directly attached to mobo - are fine. I have seen a third card option presented on this site (and I’ve read the long standard post regarding this issue) that suggets (LSI 6Gbps SAS HBA 9200-8i IT Mode ZFS FreeNAS unRAID + 2*SFF-8087 SATA #A6-31). My question is. Will this exchange simply solve my issue - or can I hopefully still use my current card to make BIOS and TrueNAS recognize the last HDD-drive?

I know there’s always to option to decouple my 2. SSD and set up mirroring later, but, words cannot describe how close-ass that whole setup is in my rack (fractal case node 304) - really don’t want to do that again, but if neccessary, I’ll have no choice…

My second issue - loss of connection to NAS(?):

After successfully installing TrueNAS on both SSD’s, and accessing the Web GUI via the provided IP from the command prompt - I had to leave my PC for a while. When I came back, it has lost its connection. And on my server screen (the command window with 10 available commands) the IP did not show up. And I therefore couldn’t access the site from my PC. Which I just did some hours earlier. After some research - I’m still clueless :smiley: … Is it just the internet cable? Some hardware issue? Some configuration that I need to address? Probably not an issue with the motherboard (please no)? And how do I re-access my Web GUI and make sure it doesn’t happen again? The whole point is it’s running 24/7, so if it just randomly and often shuts off - that wont work.

Thank you in advance for any help leads me to my server-dream :slight_smile: Have a great day!

2

Check on the screen with the 10 options the options about networking.
Maybe a reboot/ BIOS reset will bring back the network connection. The MoBo uses a 2.5G Realtek LAN. This can be instable. Good old Intel 1G or better 10G is far more stable (as are Mellanox or Chelsio).

On the screen with the 10 options (choose linux shell) - or if you gain network access again over SSH or the built in shell:

sudo sas2flash -listall

or

sudo sas3flash -listall

will show you if the HBA is running in IT mode and the firmware version.
IR or RAID mode is not good, and it should be running on the latest firmware version.

5 Likes
3

Thanks man! Today, I managed to regain access (I didn’t do anything just turned it on today) so I prompted from the built-in shell with the following results, below. I guess that means my current card isn’t compatible with my mobo

LSI Corporation SAS2 Flash Utility
Version 20.00.00.00 (2014.09.18)
Copyright (c) 2008-2014 LSI Corporation. All rights reserved

    No LSI SAS adapters found! Limited Command Set Available!
    ERROR: Command Not allowed without an adapter!
    ERROR: Couldn't Create Command -listall
    Exiting Program.

Avago Technologies SAS3 Flash Utility
Version 16.00.00.00 (2017.05.02)
Copyright 2008-2017 Avago Technologies. All rights reserved.

    No Avago SAS adapters found! Limited Command Set Available!
    ERROR: Command Not allowed without an adapter!
    ERROR: Couldn't Create Command -listall
    Exiting Program.

The version is also quite old it seems (from 2014 and 2017).

Regarding my task about a stable connection, I guess I’ll just have to see how my server acts in the next period of time, and if it keeps being unstable, my guess is I’ll have no choice but to switch my mobo with another that has the correct specs, as mentioned in your post.

If you have some additional comments, please let me know and in any way, thanks for your time!

4

Is the card seen with
lspci -nvv ?

5

Hello again. The prompt gives me a lot of different “addresses” with a lot of assorted data andinformation, but I couldn’t find as of now any specific mention of the model I’ve plugged in. Only “DeviceName” I could see explicitly mentioned was completely unrelated to the issue (Realtek ALC1220). To be honest, I’m not sure what to look after but I’m fairly certain there are no direct mentions of my card’s name.

A “typical” data looks like these two below, though there are some others that are either shorter or mentions “nvme”, “xhci”, “ahci”, “vga controller” instead of the most abundant “[Normal decode]”:

04:09.0 0604: 1022:43f5 (rev 01) (prog-if 00 [Normal decode])
Subsystem: 1b21:3328
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 44
IOMMU group: 21
Bus: primary=04, secondary=0b, subordinate=0b, sec-latency=0
I/O behind bridge: [disabled] [32-bit]
Memory behind bridge: [disabled] [32-bit]
Prefetchable memory behind bridge: [disabled] [64-bit]
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16+ MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities:
Kernel driver in use: pcieport

or

00:01.1 0604: 1022:14db (prog-if 00 [Normal decode])
Subsystem: 1462:7d73
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin ? routed to IRQ 30
IOMMU group: 1
Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
I/O behind bridge: f000-ffff [size=4K] [16-bit]
Memory behind bridge: f6b00000-f6dfffff [size=3M] [32-bit]
Prefetchable memory behind bridge: [disabled] [64-bit]
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16+ MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities:
Kernel driver in use: pcieport

6 
lspci

without the -nvv option should give you a less crowded output.

If you cant see something like “LSI/Avago Host Bus adapter”, and the sas2flash/sas3flash utility also cant see any cards, then indeed the card is not detected by linux at all.

Can you check in the BIOS if the PCI-link of the corresponding slot is “trained” ?

Some Motherboards, deactivate slots when another one is used.

7

Oh, hold on, I have one that may be it:

01:00.0 RAID bus controller: Broadcom / LSI MegaRAID SAS-3 3008 [Fury] (rev 02)

also (but probably not the card but mobo-controller)

0f:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset SATA Controller (rev 01)

So maybe any of these, if any, probably the first one.

This is my cards adv. specs:

I also checked BIOS again (also did this when first powering on my server).
Firstly, under “PCIe Subsystem Settings”, I have PCI_E1 Gen Switch to Gen3 (as my card), Chipset Gen Switch on “Auto”, and PCI_E1 Lanes configuration on “Auto”, ASPM Control for CPU PCIe “Auto” as well. Re-Size BAR Support is Enabled.

Secondly, under “Integrated peripherals” I have Onboard LAN Controller on “Enabled”, Network stack “Disabled”. Under SATA Configuration SATA Mode is AHCI Mode (which can be set to RAID Mode instead), and then i have 4 x SATA_1(2, 3 and 4) Hot Plug at disabled. But 4 in total, so not the final and fifth HDD.

I don’t know if any of this is useful for you, but I can confirm that I can’t directly see the name of my card anywhere in the BIOS. As a side-note I can say i updated my BIOS as the first thing before anything else at first boot-up with my server.

I also found this link, can this be a “last way out” for making my card combatible? I think it is the exact card I have: GitHub - EverLand1/9300-8i_IT-Mode: A guide on how to flash IT mode firmware to your LSI Avago 9300-8i RAID Controller, making it an HBA, or Host Bus Adapter.

8

The card must operate in IT mode, not because it would otherwise be unrecognized by the system, but because any system that obscures the disks from ZFS can cause data loss.
The card is not detected by sas2flash, but it does appear to be recognized by Linux.
Check again the entry for your card with lspci -nvv. This should tell you what driver/kernel module (if any) is assigned to your card.

9

Okay, so from what I can gather then, that means my current driver is raid “megaraid_sas”. So this should, if correctly understood, named something along “IT Mode” instead.

And so, is what you’re saying, that it won’t help to “flash” it and I should just get a new proper card?

01:00.0 0104: 1000:005f (rev 02)
Subsystem: 1000:9340
Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- SERR- <PERR- INTx-
Interrupt: pin A routed to IRQ 50
IOMMU group: 12
Region 0: I/O ports at f000 [size=256]
Region 1: Memory at f6d00000 (64-bit, non-prefetchable) [size=64K]
Region 3: Memory at f6c00000 (64-bit, non-prefetchable) [size=1M]
Expansion ROM at f6b00000 [disabled] [size=1M]
Capabilities: >access denied>
Kernel modules: megaraid_sas

PS My connection hasn’t been lost all day, so that’d good :slight_smile:

10

The driver in use points to the fact that your card is operating in RAID mode. Flashing it to IT mode would hopefully bind the card to the MPTSAS module.
There are multiple ways to flash the card (DOS, UEFI Shell, etc), but i am not familiar with them.

Buying a 9300-8i already in IT mode is of course an option.

2 Likes
11

An even better option, IMHO, would be to drop the HBA.
Take this opportunity to also drop the SATA SSDs and use a small NVMe as single boot drive. Less clutter, more SATA ports. (Why do newcomers think they need to mirror the boot device???)

5 Likes
12

Likely because most other typical server platforms (ie. Proxmox, Ubuntu server, etc.) aren’t built as a firmware so they usually can put actual user data (typically VM’s and containers) on the boot drives. Moreover, their communities/forums would often also recommend mirroring the boot drives.

People used to seeing these advices would then carry that knowledge over to TrueNAS without fully understanding the nuance around it. Heck, I’ve seen people recommending that even within this community (Reddit r/truenas).

13

I would trade that card for a card that’s already in IT mode. The fact that the kernel sees it and the flash utility doesn’t bothers me, because that utility will literally let you flash IT and non IT firmwares to the card. No matter that it’s old, there are stacks of these from ancient servers that still work fine, all over the place.

One thing I noticed is sellers usually call out Truenas and Unraid in their descriptions. It’s no mystery what people use them for.

Keep in mind these are server cards and you gotta be on point with your cooling. Servers are built to blow cold air over everything and make a lot of noise doing it, so hardware designers expect that environment and give you a heat sink with no fan. Generally, a cool room is good enough, but definitely consider the possibility that you may need to give it a fan someday.

Finally, disable that network card in the bios. You don’t need it showing up.

14

The 2 flash utilities are for Non-Hardware RAID cards. MegaRAID is a different animal, real hardware RAID which generally includes write cache and RAID-5/6. The MPT ones with IR, (Integrated RAID), don’t include either of those features. Thus, can have 2 sets of firmware, IR or IT.

There are some MegaRAID cards that can be flashed to IT mode. However, that is not trivial or certain. Easier to use one of the MPT varieties.

3 Likes
15

Hi again all. I can’t thank you enough for all the insight - it’s really helpful to read all this. I’ll give some individual responses underneath.

However, I can say that for my own sake, now I’ll either

  1. just buy a correct card in IT mode, or
  2. convert to a single boot drive via a small NVMe

I think, just because I’ve come “this far”, I’ll swap my current with a correct one. At some point if I need to rearrange my setup, I’ll probably go with the single boot option.

So maybe just as a follow-up question - is there anything particular to be aware of when converting away from the mirror setup to a single boot? I’m thinking just in terms of keeping the SSD’s as intact as possible.

Internet still going strong to my server connection, so I’m thrilled about that :slight_smile: !

@Farout Got it, and thank you for all the replies, very helpful!

@etorix This seems very optimal, and I’m kicking myself a bit over my current setup. I can say the reason I’ve done this is because I got it suggested by my best mate, who’ve had his own NAS systems since forever and you know, I trust him. I’m faily certain he’s more advanced in other areas than TrueNASbuilding, so as @Whattteva said the reason he suggests it could be what he replied there.

@afrosheen That’s great info, I’ll make sure to bookmark all of that, great insight. I’ll definitely keep a look-out for a card with a fan. My office where it’s placed is actually pretty cool luckily, so I think i’ll be alright no matter what, but if I get the chance, I’ll just take one with a fan for sure.

@Arwen this is where I don’t really get a lot, if any, of what you’re saying - haha :smiley: But, with the other kind blokes replies, I think I have what I need to take my next steps :slight_smile:

Let me just say, this is a fantastic first impression. Received way more than expected! Havea great one!

1 Like
16

A single boot drive conversion would be trivial. Frankly, although you must have a boot drive, bringing truenas back to life if a single dies is super easy with a current backup file and the install media. You just reinstall on the boot drive and load the backup config file after you’re done. That’s it. I like to move my system data to a pool so the boot drive can be extra lazy without being pecked every 5 seconds for little writes, but it’s not a big deal. Just spend a few minutes after installation to setup the automatic configuration backups and send them offsite (Google drive is convenient for that). With that config backup on hand, recovering from disasters is usually straightforward. Although there’s nothing wrong with your mirrored ssd boot setup, @etorix’s solution frees up two onboard sata ports. Personally, my system is mirrored like yours with an LSI 9211 card handling the array. Same problem, ran out of onboard ports, 8 drives plus the mirrored boot equals 10 sata, not easy to find on ITX boards.

Good luck, you have choices to make and coins to toss next. We do have a fair number of experienced people here who are helpful and we’ve tried different configurations over the years, so you’re in good hands.

1 Like
17

Supermicro A2SDi-H-TF and -TP4F boards… but I get your point :wink:

Mini-ITX and/or embedded boards require picking a model with exactly the required feature set. Here the Node 304 case should have been matched with a motherboard having at least 6 SATA ports: AMD AM4 (B550), Intel (most socketed CPUs but the latest ones), or the long time favourite Xeon D-1500 (X10SDV). Unfortunately it is too late to re-do the build.

2 Likes
18

Basically, ZFS does not do well with hardware RAID cards. There have been multiple cases of ZFS pool corruption when using hardware RAID cards. MegaRAID is hardware RAID and generally not suitable for use with ZFS.

The 2 programs previously listed are for chips that can have either lightweight RAID firmware, or simple disk pass-through firmware. Naturally we recommend the simple disk pass-through firmware for use with ZFS.

2 Likes
19

Alright, so I think I shall note this is a solution post - to hopefully help future noobs like myself. Some must-do’s, pitfalls and such (I can’t wait to look back at this post later an realize my question were all "so you’re saying 2+2=4? I DONT UNDERSTAND "). Some replies to the kind people in the end as well.

As of right now I can’t wait to get into setting it all up :slight_smile: !

  1. Think hardware and software must be compatible. It is easy to say after the fact, but when you approach server-building as newbie think it through from the first piece of hardware to the very final OS, software you’d like. It is not enough to assume (like I did) “Oh, just make sure hardware fits and boot it up, simply install TrueNAS and it’ll work”. Especially focus

1.1) No, if you - like me - don’t have the required SATA ports on mobo and needs an HBA card: BUY IT IN IT-MODE. Very often the sellers (there’s A LOT on eBay) will explicitly mention “TrueNAS” to eliminate confusion. Just search for “HBA IT-mode TrueNAS”. I bought this one, and it worked (LSI Inspur 9300-8i SATA / SAS HBA Controller IT-Mode 12Gb PCIe x8 unRaid TrueNAS - LSI Inspur 9300-8i SATA / SAS HBA Controller IT-Mode 12Gb PCIe x8 unRaid TrueNAS | eBay.de).
NOTE: Buy if you don’t have SAS-to-SATA cable(s) with it.
NOTE: Don’t just buy the simplest/cheapest HBA card, the controller/firmware will not be combatible - and if possible - buy one with fan on it.

  1. See to buy a mobo that has as many SATA ports as possible. The likes of AMD AM4 (B550), Intel (most socketed CPUs but the latest ones), or Xeon D-1500 (X10SDV) have been suggested.

  2. See that your mobo is not Realtek LAN. As mentioned above it can be unstable - though how much we should put into can is uncertain. I had a mishap at my first boot-up after TrueNAS OS install, but since then, no interruptions. Instead, just to be sure, seek out Intel, Mellanox, Chelsio.

  3. Optimally you’d like to install your TrueNAS OS on a simple NVMe and use it as “stripe” to boot from. I chose purely based by suggestion by a friend to install mirrored OS on 2 x SSD, which is non-optimal. This with (2) will mean you free up SATA-ports which results in less clutter.

One question
1] This is a purely practical weird one, but is it best to have the final chassis on my case (to ensure directed airflows over hardware and minimize dust) or is it “okay” to have it open and better release generated heat?

@afrosheen . Cool that’s all noted, it’s good that it at least isn’t that big of a deal to change. I’m still iffy about my setup, but fuck it for now, I’ll keep it like this - it finally works :slight_smile: .

@Arwen . Thanks for spelling that out for the newcomer :wink:

20

I don’t think any cases besides that garbage can apple made years ago are designed for passive cooling, so definitely keep the doors and walls on it so the fans can pull cold air in and push hot air out without the air just going anywhere. Drives in a certain spot getting too hot, add fans and/or a speed controller. Maybe change the case. But definitely keep it closed. Those open frame cases, for example, the asic mining rigs, expect to live in a frigid environment like a data center.

If you’ve never been to a data center to see how servers get treated, imagine a hot row and a cold row, holes in floor panels beneath the racks blasting cold air straight into the rack, and heavy plastic sheets from the ceiling that separate the hot air in the hot row (behind the racks) from the surrounding air. The hot row air gets pumped out and recooled.

So, a server’s natural habitat is a pretty cold room. Then any air they’re sucking in to cool their components is always cold. As homelabbers, we have to replicate that in some way.