I am about to pull the trigger on a server upgrade and wanted to make sure I was on the right track. This server will be used as a primary file server for Plex, general file server/backup, etc. Less than 5 users at any given time accessing the system.
I currently have a 180TB unraid server, 2x parity and 20+ data drives, about 130TB full, that is storing all of my data. All data will be moved off this to the new system below.
I have purchased a sueprmicro 847 and and looking to fill it with the following:
Epyc 7532
256 GB DDR 4 3200MHZ ram
h12ssl-i
40GB mellanox card
several lsi9500-8i cards for the backplanes
2x Raidz2 vdevs of 8x 20TB HDD (could expand to 4x total in the future, total of 32 HDD)
No cache, doesn’t seem to help my use case
no ZLOG, doesn’t seem to help my use case
2x 20TB hot spares
Metadata: confusion here, but seemingly a 2x or 3x mirror of 1tb optane NVE is my though
No DeDupe
2x 128 GB SSD as boot drives
Thoughts? Also should I use scale or core, since I wil only be using this as a filesystem, I have another machine for docker/vms/etc.
You can make that 9300-8i (cheaper, extensively field tested). Why “several”? Aren’t there expanders in the 847?
Optane is not required in this role. Mind however that a special vdev is critical, so a 3-way mirror is the best match to raidz2, and that you cannot remove it, contrary to a persistent L2ARC. (But is L2ARC even useful with 256 GB RAM?)
Your call. Choosing CORE will certainly reduce the administration work to upgrade…
My rear backplane BPN-SAS3-826EL1-N4 has 4 nvme u.2 drives that required 4 connectors, then there are 2 connectors for the rear backplane and 2 connectors for the front backplane, which could be cascaded from one to the other for 6 total connections.
So you’re suggesting 3 drives for that pool. Is l2arc with a metadata only going to be just effective here without the risk? It will be mostly large files but there will also be about 5-10% of data that is small files, photos, metadata files for plex etc.
What do you mean by that? It seems like core is better for people who want just a NAS, but Scale is more for people who wants dockers/VMs/data all in one location right?
Something like this?
This looks like a SAS backplane with four mixed slots which can alternatively take U.2 drives, with separate entries. This is not a U.3 Tri-Mode backplane. So you can keep it simple with a regular SAS HBA (not Tri-Mode) for the SAS part.
If you intend to use some slots for NVMe, feed these from a PCIe slot (may need a re-timer for PCIe 4.0 integrity). Otherwise, use the backplane as SAS and put NVMe drives directly in PCIe slots—no hot-plug obviously, but does PCIe hot-plug ever works?
For reads, yes, but not for writes.
Sorry. It was an ironic way to point that CORE is effectively EoL and you won’t be bothered by decisions over upgrading to 13.4 or 14.x if you go this route because there likely won’t be anything but security upgrades for 13.0 and 13.3.
But if you only want storage, CORE is certainly a very mature, stable and capable solution.
Yes just like that. I do intend to use them for kioxia drives or some other u.2 later so I’d prefer to get out ahead of having to take the motherboard out.
You’re suggesting purchasing something like that retimer, and then putting some cables in between this card and my backplane for those nvme drives?
What about for the other 2/4 connectors for the front and rear backplane HDDs, what would you suggest for that part of it?
So as long as im comfortable with the risk of raidz2 metadata drives, that should be no problem. Thanks for confirming
No problem, if that’s the case ill stick with scale it sounds like. Theres not much performance difference between the two?
Yes. There have to be dedicated cables for the NVMe lanes anyway.
For the SAS part, back and front, you may use any HBA, including the long-trusted 9200/9300 series. This could be a 9500, in SAS mode only, but the driver stack was rewritten for Tri-Mode and there’s less confidence in the drivers for 9500+ cards than for 9200-9400.
I suppose there are expanders in the front backplane as well, and you could use a single HBA for all.
Since im limited to half height cards, do they make a 16i in the 93/400, or will I need to get 2, one for the front and one for that back? Or would it be just as safe to get one 93/400-8i, pass the rear back plane into the front, then cascade all of those drives to an 8i?
The 9305-16i is half-height, and you could indeed connect the backplanes through their expanders and feed everything from a -8i. Each 12G SAS lane has the bandwidth for 5-6 HDDs.
The 9305-16i is half-height, and you could indeed connect the backplanes through their expanders and feed everything from a -8i. Each 12G SAS lane has the bandwidth for 5-6 HDDs.
So this would not be an issue for my use base of 32 HDD’s on this one 9305-16i or 8i? Im not familiar with the math
I hear the advantages of having metadata for speeding up writes/reads on smaller files, which will be sparsed in a lot of my plex metadata folders as well as make up the majority of rom backups and etc. I see threads where people mention loading large folders is much faster which sounds like a nice plus as well.
But I’m inherently nervous on a drive/set of drives that if they fail will invalidate all my storage. Obviously raidzX on those drives, as well as using high quality drives is fine, but that’s not perfect. And I don’t have the capacity to really try both.
Was hoping for more of a concrete answer on whether the metadata vdev was work it over just using metadata l2arc only.
From etorix above, it sounds like if I have lots of small file WRITES, then yes, but if its WRITE small files once, read them many times maybe not so much?
An 8i has 8 lanes of 12gbps SAS. So supports 96gbps of SAS bandwidth, but 8 bits per byte… so 12GB/s of BW
12000/36 drives is 333MB/s per drive.
BUT sometimes it’s the PCIe BW that is the bottleneck in these cards.
Your Epyc supports PCIe5, but the card is PCIe4, 8 lanes
PCIe4 is good for 2GB/s per lane. Or 16GB/s.
So no bottleneck. But if you were using PCIe3, there would be a bottleneck of 8GB/s which would reduce the bw of all the drives to 222MB/s. Which is probably fine. If you’re not using SSDs.
Interestingly, the 16i may have twice the total SAS bandwidth, but it only has the same PCIe BW so is bottlenecked.
The metadata vdev (mirror, not raidz) takes up all metadata writes, plus small files. It speeds up reads and writes but you cannot lose this vdev. L2ARC is for reads only, writes go straight to the slow pool but it is an optional vdev which can be removed or lost with no adverse effect other than loss of performance.
For a mostly static pool hosting Plex media, a persistent L2ARC should do.
Or maybe no L2ARC at all because your ARC is large. If you don’t reboot often all metadata will end up there and there will be no opportunity to drop to L2ARC.
