Beginner - TrueNAS Setup on used SuperMicro X10SDV-6C+-TLN4F Board

TrueNAS General
, ,
1

I’m a “first time” NAS/TrueNas user, that have bought a used NAS DIY System.
It has not arrived yet.

Specs:

Mobo: Super Micro X10SDV-6C+-TLN4F 
https://www.supermicro.com/en/products/motherboard/x10sdv-6c-tln4f
- Xeon D-1528 - 6C/12T
- 2x 10GBaseT (X552/X557-AT), 2x 1GBaseT (I350)
- IPMI (HTML5 KVM)
- 32GB ECC mem (2x16GB DDR4/ECC 2100)
- 6 x Onboard Sata connected to CPU

Boot disk
128GB MSATA (Transcend MTS830S)

NAS Disks
5x4TB WD RED (WD40EFRX-68) 

Chassis
Fractal Design Node 304

My Background:
I have been working w. linux for 20+ years, and networking for 30+.
But i have never had a NAS (Raid) system before.

Current storage:
Linux server w. a 4TB Rotating SATA.
Disk offered as NFS/SMB share(s).
Disk used for nightly linux sys-disk backup (tar, selected items)
Backup - Nightly rsync SATA to 4TB USB3 Drive.
Network: 1Gb Lan
Clients (Wife & I) : Mostly WiFi AC6 - I get 3…400 Mb, and use Cable if i need more.

New NAS Goal
Raid for better redundancy.
Larger storage
Better performance - (Not a current focus point)
Use NAS for NFS/SMB shares., instead of for Linux 4TB.
Use NAS as Proxmox PBS “backup destination” … Or maybe run PBS on NAS ??
Stability … Stability … Stability
10Gbit will not be added right away, this is currently my only 10Gb capable machine.

Questions:
Q1:
What TrueNAS type should i go for ?

  • I’m leaning towards SCALE, as i’m used to maintain deb servers.
  • I do use pfSense, so i have a “little bit” of FreeBSD experience.
  • Focus would be stability

Q2:
This is IMHO a “Powerfull” Machine (for a home user).
Should i run TrueNAS on the “iron” or install Proxmox VE , and run TrueNAS as a VM ?
Rephrasing the above … I’m a Proxmox beginner too.
Would it be feasible for a Proxmox & TrueNAS beginner to run TrueNAS as a Proxmox VM.

Q3:
If i install SCALE, can i run the Proxmox PBS (backup server) together w. TrueNAS. I read that PBS could be added to a “debian”
Own Reasoning : Kernel issues …

Q4:
Will TrueNAS suggest some sensible defaults during install ?
What would you suggest for my 5 x 4TB Spinning disks ?
Raid5 or ZFS or ??

Q6:
I have a 6’th SATA Port - Designated SATADOM
Can that be used as a “normal” SATA port ?
Would a 6’th disk add any benefits, besides more storage ?

Q7:
Any Supermicro “Gurus here” ??
I’d love some Bios setting hints, for my type of MoBo.

Q8:
If there’s some kind of “Bios Raid” , it should be disabled correct ?

Q9:
What have i forgotten ? :slight_smile:
What is going to “bite my …” ?

Any pointers to a recent Webbased how2, would be appreciated.
Please not “loooongg” YT videos

TIA
Bingo

2

I’m using a similar Xeon-D board (though the 4C/8T variety) for my OPNsense router at home. It’s pretty nice.

SCALE. iX are killing off CORE.

I wouldn’t.

Yes, but not in the way you’re thinking. You can’t just install software into TrueNAS; you’d need to either set up a sandbox (I expect @Stux will be along shortly to share his videos on the subject), run it in a VM, or install it as an app if it’s available from iX (it was available from TrueCharts, but they’re no longer supporting TrueNAS installations). I’m running it inside a VM on my smaller NAS.

There is no “RAID5” with TrueNAS; everything is ZFS. RAIDZ1 (which is similar to RAID 5) is borderline IMO with 4 TB disks; we generally recommend RAIDZ2, but those are small enough that Z1 wouldn’t be too risky.

Yes.

“More storage” seems reasonably significant, but it wouldn’t really do anything else.

Supermicro’s defaults are generally pretty sensible.

Correct.

2 Likes
3

I documented the build of a Node304/X10SDV system in this thread:

Originally it was an ESXi box, but I now just run Scale on bare metal and use VMs and Sandboxes as necessary.

I’m not sure if your MSATA uses one of the 6 sata ports, but I’d use the m.2 slot to ensure you can run 6 HDs.

3 Likes
4

I have the same system, with six 12 TB drives. Just perfect for the job and the case. 16 GB Optane M10 as boot drive (small and cheap).

Q1 I hate to say that but in your situation you have no real reason to go for CORE.

Q2/Q3 What do you want to do with Proxmox? TrueNAS can run full VMs (KVM) as well as llighter “system containers”, known as “sandboxes” (currently systemd-nspawn, LXC planned for the 25.04 release); 24.10 Electric Eel will have native Docker with docker-compose. If this is enough for your needs, running TrueNAS bare metal will make it easier and much safer.
If you do want Proxmox as a better hypervisor than KVM, you have some more reading to do about virtuallising TrueNAS. This involves passing the drive controller through, so you’d have to use NVMe for boot and everything Proxmox.

Q4/Q5 A sixth drive makes mot use of your ports and the bays, and 6-wide raidz2 is a fair balance of resiliency and space efficiency for most home uses.
The SATADOM port is also a perfectly nomal SATA port. Using it for data implies booting from a M.2 NVMe.

Q7 Defaults should be good, but I would disable anything anything CSM/Legacy.
Q8 SATA set to AHCI.

Nothing, especially if you read the build post by @Stux for all neat tricks.

The 10 GbE ports can do 1 and 2,5 GbE. You’re future proof.
The x16 port can bifurcate all the way to x4x4x4x4 to hold more NVMe drives in riser cards (Asus Hyper M2, or the plain Shenzhen equivalent) for a “fast pool”, application pool.
And second-hand DDR4 RDIMM is cheap, to add more RAM for your apps/VMs.

3 Likes
5

@dan
Thank you for answering so many of my questions, I really appreciate it.
And for setting the “No Raid5” in TrueNAS straight.
Re: RAIDZ1 vs RAIDZ2 … I’ll try to read up on what it is :blush:
What would you recommend ? - I Might go with bigger disks in the future.

@Stux
Thank you for your link to your setup :+1:
I’m looking forward to see what “magic” you have done with a "similar MoBo.

Re: Boot disk
That MTS830S seems (from pict) to be a NVME

image
image1193×342 92.5 KB

Gents Thank you for helping out a beginner.

I’m sure i’ll be back w. more questions, when i have looked at @Stux install guide , and have gotten my new “toy”.

/Bingo

1 Like
6

I’d lean toward RAIDZ2, especially if you think you may upgrade by replacing with larger disks in the future.

3 Likes
7

@etorix
I am switching from VMware to Proxmox for my “Home VM’s”
I run ie. PiHole , Zabbix and some other “light weight” stuff.

And i run some Debian based “VPN tunnel linuxes”, each on their own Vlan (protected by pfSense), since i have 8 SSID capable AccesPoints I have just assigned a SSID per “VPN tunnel linux” , so switching is as easy as connecting to another SSID.

Especially the “Linux VPN’s” have to run in a “True VM” not a container, as i do some kernel route trickery, and an adapted iptbles …

So you’d have to pry Proxmox from my “cold dead hands” :slight_smile:

That said …
Energy cost is quite high here in DK , and i’d expect the box to draw … Mobo 30W , Disks another 30W … That will sum up to around $240/yr in electricity.

So if i can use the TrueNAS for some other stuff, that would be a bonus.

Do you have any pointers (ebay or …) for “cheap” suitable DDR4 RDIMM’s that would fit my box ?
Right now i have 2 x 16 installed, so i have 2 free slots.

/Bingo

8

@dan
What would RAID2Z “cost” in terms of usable diskspace vs RAID1Z on my 5 x 4TB ?
Would i need to add another 4TB ?

erorix mentioned this wrt adding a 6’th disk:

6-wide raidz2 is a fair balance of resiliency and space efficiency for most home uses.

I’ll be “off” to read up on the Stux setup now :blush:

/Bingo

9

RAIDZ2 gives you another disk’s worth of parity vs. RAIDZ1. Adding another disk would even it out.

1 Like
10

The caption plainly says “SATA”, so it would eat a SATA port. Just get any cheap NVMe drive.

Single vs. double parity (RAID5 vs. RAID6). The thing is you cannot change raidz level after creation, and Z2 is increasingly desirable as drives get larger…
That said, with larger drives you could use less drives, and thus less power. But that would strengthen the case for raidz2, hence a minimum of 4 drives.

Containers can have their own network stack. But OK. You have to decide between Proxmox and TrueNAS as VM and TrueNAS baremetal with KVM.

I wanted to link JGreco’s resource on “not loosing your data” but the old forum is unreachable right now… :scream:

eBay, any professional refurbisher you may know of in your area, the small ads on the ServeTheHome forum.

11

My problem is that I don’t even know what to look for …
My first ECC machine.
Seller says he’ll put this in now:

2x16GB DDR4/ECC/2400

Supermicro’s webpage only mentions “Archived EOL Mem”

image
image1685×539 51.8 KB

Well … I’ll have a look on the ones installed, or at STH.

/Bingo

12

THANX :+1:

I would have been bitten there …
So i can just fit a NVME in the same slot ??

Re: Proxmox again - clarification
I have two other physical Proxmox machines, that hopefully shall backup to the TrueNAS.
I’ll not run TrueNAS as a VM , that will be on the “bare metal” as recommended here.

13

DDR4 RDIMM or Registered DIMM. Simple, but you have to avoid LRDIMM (Load-Reduced, typically comes in 64 GB and larger) with this board. ECC UDIMM will work as well, but it is usually more expensive.

Your D-1528 runs it at 2133 MHz. 2400 MHz modules will work (I don’t even own 2133, only 2400 and higher), as well as higher speed modules (but I’ve never tried 2933 or 3200 MHz RDIMM with a D-1500 because I keep the more expensive modules for the Xeon Scalable/W-2000 which do use these at full speed). And I wouldn’t worry much about the QVL.

Yes.

2 Likes
14

Some WD Reds are CMR and OK for ZFS, some are SMR and absolutely unusable in a redundant ZFS vDev.

Good news - WD40EFRX are actually WD Red Plus and are CMR. Phew.

1 Like
15

I don’t think that will fit without losing a drive bay. You need a half length PCIe card. Can be full height though.

Also 8x4x4 riser card adapters are a good option. Can add a half height (dual width possible) GPU and dual m.2.

1 Like
16

TrueNAS works fine as a guest on this board. Make sure you update the bios.

You can pass through the entire sata controller and boot off NVMe.

1 Like
17

My existing 5 x WD-RED 4TB, that someone mentioned were CMR Red-Plus by the sellers model mention (WD40EFRX-68), needs a 6’th for RAIDZ2.

Now that i have to shop for a 6’th drive - Parity2, can i get a larger drive ie. 8TB ?
Or does the size have to match ??

Reason for 8TB , would be : That’s prob going to be my next drivesize, when replacing.

Would 6 x 4TB , mean i have approx 16TB usable , as the two parity “doesn’t count” ??

Edit:
My PSU is a : Corsair SF450 PSU (80 Plus Gold)
That should be enough … Correct ?

/Bingo

18

Or, better yet, a sVDEV that will speed up small files, provide AWESOME metdata speeds, and can still be used exclusively for hosting datasets that benefit from SSDs (by adjusting record size and small file cutoff of the particular dataset) Just sayin’ :slight_smile:

19

FWIW, I run a 8-He10 HDD + 4 SSD system here with a pretty similar processor / motherboard (see signature). The power draw is about 110W or so. I haven’t checked it in a while. So it’s not super-low power.

Lower-power systems can certainly be made, i.e. reduce # of drives and choose drives with a higher capacity than I did. Choose a lower-power CPU like N100 but likely make some big compromises re: performance for anything more than a 1GBit/s file server. Or go all-flash and pay big $$$ / Euros for lower power.

1 Like
20

You can use an 8TB, TrueNAS/ZFS will treat it as a 4TB drive until all the drives in the VDev are upgraded.

Yes, rough numbers are 4x4TB from 6way RaidZ2 with 4TB drives.

Plan to plan your upgrade when you hit 80% utilization and finish your upgrade by 90%.

It’s much safer to do in-place drive replacements with RaidZ2 too :slight_smile:

2 Likes