I am looking to spec out a PRODUCTION 100G networking Dell R730xd storage server for TrueNAS SCALE
I have the following configured and wanted to make sure these are proper configuration
My usecase is to have a single giant storage and backup server to backup VMs and docker containers from multiple hosts of all kinds.
WIll also have some NFS shares and some iSCSI block volumes that will be mounted to VMs.
This will be a production build.
1x Dell PowerEdge R730xd 2.5 Chassis with up to 26 Hard Drives (24 Front Hot-Swap + 2 Rear Hot-Swap Hard Drives)
2x 250GB Samsung 870 Evo SSD mirror ZFS RAIDZ for TrueNAS SCALE OS
10x 4TB Samsung 870 Evo SSD stripe + mirror ZFS RAIDZ for data storage
1x Intel E5-2699v4 2.2GHz/55M/2400MHz 22-Core 145W
4x Dell 64GB DDR4 ECC LRDIMM 2666Mhz
1x No Raid Card option installed in server
1x Dell Intel X710 Quad Port 10GbE SFP+ rNDC onboard NIC
1x Intel E810-CAM2 2 x 100G QSFP28 PCIe 4.0 x 16 NIC
1x 6 PCIe Slots, 4 x8 and 2 x16
1x iDRAC8 Enterprise
2x Dell 1100W 80 Plus Platinum Efficiency Power Supply Unit
as you can see i will add 2 x 100G qsfp28 network card that will connect to Mikrotik switch with 2 x 40G qsfp+ ports
So here are my questions
Is is ok to have no hardware raid card from config above or do i need the HBA330 as i read elsewhere?
I plan to run some VMs on this so am thinking TrueNAS SCALE is the only option for me? I do not plan to cluster this as of yet, so will be a single cluster and will be like that for a very long time, lets just say.
Anything thing else i should be aware of in the config? Like are my choice of SSD drives good? I am thinking going with the 2.5 chassis so i can get high IOPS and throughput especially if i am also using as NFS and iSCSI for servers also
Is going for SATA III SSD ok or i should opt for SAS SSD? I will like to upgrade to 100G switching so will like to remove avoidable bottle necks
Hardware RAID is strongly advised AGAINST. But you may well need a HBA to connect (some of) your 12 SATA drives. I doubt that drive bays are individually plugged into SATA ports on the motherboard.
You can correct me if something makes no sense, i may be using wrong terminology for ZFS
mirror RAIDZ i meant RAID1
mirror and stripe RAIDZ i meant RAID50
well are you familiar with Dell R730xd 2+26 SFF server?
are you saying i need a HBA controller and without it i can used all the drive bays? I would think this should be default on the motherboard then and not optional
So based on that i need to make sure i have the Dell HBA330 controller then?
want to make sure i have correct configuration is why i created the thread
“raidz” is ZFS terminology for “parity array”, with raidz1, raidz2 and raidz3 being the functional (but not technological) equivalents of RAID5, RAID6 and… there’s no “RAID7”
if there are multiple vdevs in a pool, they are striped (RAID 0 equivalent)
ZFS terminology for (the equivalent of) RAID50 is “striped raidz1”, with 10 drives I suppose you have a stripe of two 5-wide raidz1 (but it could be three 3-wide raidz1 with one spare, or an imbalanced stripe of 4- and 6-wide…)
“stripe of mirrors” is the equivalent of RAID10
mirrors and raidz do not go together, which is why I asked
Actually not, but I would expect that the 24 front bays are on a SAS backplane with expander(s) and need to be connected to a SAS HBA. You’re the one whith physical access to the server to see how it’s wired.
Not one of your questions but you’re aware that this configuration, with a LAGG bond, with achieve a bundle of eight 10 Gb/s links, not anything near “100 Gb/s”, and that it will take MANY clients simultaneously accessing the NAS to make the most of it, right? No single client will see 40 Gb/s.
The question I have is why spec out a high end server and connect it to a lower end mirotik switch. For a high end production server I would go with a Cisco or HP/Aruba
I am sure that microtik will work and provide adequate speed etc. What I speak to is support ability and reliability. Microtik is definitely not known as a “production switch” and more of an cheaper alternative switch. What does the cloud run on? It aint microtik…You are specifying a Cadillac server environment and tying it to a Chevy in my opinion. Again that is my opinion and I will shut up from here…
Agree the cloud does not run on Mikrotik but am not running a cloud. I am running a private infra and the Mikrotik has been working and will continue to work just as expected.
I am fine with no support because i know my way around and there is community forums just like TrueNAS is. I got spare switch if anything happens, and am familiar with the OS now that am good enough to confidently run it on my own private “cloud”
I mean so what are your takes on running opensource in production then?
I mean TrueNAS SCALE community? Pfsense community edition?
Hell Ubuntu OS with no support? unlike Redhat?
Thoughts on that maybe all non-enterprise software with current and active paying license and hardware should be only for nothing critical?
This is just computerised configuration gobbledegook.
A couple more points (to add to what @etorix said):
Despite data being on SSDs, if the motherboard supports NVMe/Optane, you might want to think about using this for a small SLOG to speed up the synchronous writes from iSCSI and NFS.
With 256GB of memory, you might want to think about either a special vDev and / or L2ARC also on NVMe/Optane.
I gotcha. I did not read this as a “home” environment. Heck I would never spec a High end/High dollar Dell server for a home environment. There are a lot less expensive alternatives. Heck at home I run on D-Link switches but our data center at work would have a cow if I put a D-link in there.
Mirrors are also better for IOPS. Of course there will be some loss of capacity.
SATA 3 is probably good enough, adding more vdevs will increase performance more than going for 12G SAS SSDs instead of 6G SATA. If you really want more, go full NVMe.
What choice of drives for OS and for the data and what will be a good starting point for lets say 10 drives or if more then how many more that can then be scaled by adding more drives. The server takes up to 24 drives excluding the 2 drives for the OS at the rear of the server
full NVMe will empty my bank account faster that i can blink, so i prefer that NOT to happen
5 2-way mirrors. Add more, two at a time, if you need more capacity. The pool being SSD, you might get away with >50% occupancy and block storage because SSDs fare better than HDDs at allocating small blocks… but you’re essentially on your own to find out at which point fragmentation will eventually hurt.
@Protopia has a point for the SLOG with sync writes on VMs and iSCSI, especially as the EVO do not even have PLP. But then a-no-so-small Optane drive (to be future-proof, 2 txg @ 100 Gb/s = up to 1 Tbit to store), and some benchmarking may be in order to see how a single Optane would fare against a stripe of 10+ SATA SSDs in the default ZIL (I expect that Optane would still win on latency, but at some point it is bound to lose on throughput).
If you had data centre SSDs with PLP I would do without a SLOG.
Mind that 64 GB RAM or more is recommended for iSCSI when allocating the VMs.
As for L2ARC, I would wait and see whether it would be useful (arc_summary on the actual production setup). Or throw in more RAM.
I do not see the point for a special vdev with a SSD pool.
SLOG only holds data for a short period - but if you want absolute data integrity it should be mirrored (because non-mirrored SLOG failing will result in some committed writes being lost).
It depends on how much memory will be left over for ARC after you have used some for VMs. The established rule of thumb is that you need at least 60GB of memory for ARC before even thinking about L2ARC.
I am trying to understand what you mean here.
So you are saying i should do 5 2-way mirrors? so essentially have half capacity?
I mean using SSD already provides IOPS and throughput compared to HDD. The iSCSI blocks will only be used by couple of VMs. This will not be a huge setup with hundreds of VMs, just like 10 VMs lets say that really need fast storage. Other VMs i can put on slower iSCSI blocks on slower vdevs
I do want the capacity since using with VMs but i think mirrored stripe vdevs should do just fine. As long as i have some kind of redundancy and fast enough am good with what i want
So are you saying DC SSD with PLP will provide performance that SLOG would have otherwise provided?
When you say Optane, are you referring to tike the Intel Optane DC SSD?
Yes. Please read the resource above block storage I linked to.
The issue is that zvols/iSCSI are all about lots of small block transactions. SSDs are good at IOPS (the 'lots" part) but raidz is intrinsically inefficient at storing small blocks due to parity and padding. And “like 10” performance-critical VMs" plus an unspecified number of “other VMs” is a lot of VMs. Maybe not the hundreds of a big enterprise setup, but quite a lot.
Another terminology slippage. You mean “striped raidz vdev” here (and you said I could correct ).
Yes. PLP is all about speeding up sync writes. If you have an option to change the build from consumer SSDs to data centre go for that and skip the SLOG.
Ideally yes, as you’re setting up a professional server for professional use, but the consumer 900p/905p would do in a home lab.
).