Hot spare in the head unit instead of the JBOD – 5×9-wide RAIDZ2 with 28 TB disks?

Hi all

I note that Claude helped me with phrasing this

I’m planning a new pool and would like a sanity check on where to put the hot spare.

**Hardware / software**

- TrueNAS Community Edition (SCALE) 25.10.7

- Head unit: ASRock Rack B650D4U, AMD EPYC 4245P (6C/12T), ConnectX-4 Lx

- RAM: 96 GB DDR5-5600 ECC (2× 48 GB Kingston UDIMM, 2 of 4 slots used, so room to go to 192 GB if it would help)

- HBA: Broadcom/LSI 9500-8e (SAS3808, external ports only, mpt3sas)

- JBOD: Supermicro SC847E26-RJBOD1, 45 bays (24 front + 21 rear), dual SAS2X36 expanders per backplane (6 Gb/s). Because the drives are SATA (single-ported), only the primary expanders are usable. Currently front and rear are cascaded on one x4 link; I plan to recable so each backplane’s primary expander gets its own HBA port (2× x4 6 Gb/s).

- Disks: Seagate Exos 28 TB ST28000NM000C (SATA, factory-recertified), building out in batches rather than all at once (starting with ~16)

- L2ARC: Samsung 990 PRO 2 TB NVMe (persistent L2ARC on). ARC hit rate is ~98% and the L2ARC hit rate only ~5%, so I may drop it or switch it to metadata-only on the new pool.

- Boot pool and app pool are on SATA SSDs on the onboard AMD chipset SATA controller

**Use case**

Mostly large media files (TV shows and movies) for Plex/Emby, plus torrent seeding (read-heavy, low write load). Planning recordsize=1M, lz4, atime=off on the media datasets. Not much small-file or block storage on this pool.

**Layouts I’m considering (all RAIDZ2)**

1. 5× 9-wide = 45 disks, JBOD completely full, ~980 TB nominal. Hot spare would sit in the head unit, attached to the onboard chipset SATA controller (the HBA has no internal ports).

2. 3× 9-wide + 2× 8-wide = 43 disks, ~924 TB nominal, leaving 2 JBOD bays for a hot spare and a free bay for in-place replacements. I know this needs a forced add (mismatched replication level) and the GUI may complain about mixed widths.

3. 5× 8-wide = 40 disks, ~840 TB, leaving 5 bays for spares/free slots. Cleanest, but gives up the most capacity.

I ruled out dRAID because I’m growing in steps and dRAID vdevs can’t be widened later.

**Questions**

1. Is there any downside to having the hot spare in the head unit while all data vdevs are in the JBOD? In my case it would be on a completely different controller (onboard AMD AHCI rather than the 9500-8e + SAS expander). Are there issues with ZED behaviour, enclosure/slot mapping, or anything the middleware doesn’t like?

2. With option 1 the JBOD is full, so there’s no free bay for a `zpool replace` with the failing disk still attached. Is relying on “spare kicks in → pull dead disk → insert new disk in the same bay → replace” acceptable practice?

3. Would you go with the mixed 9/8 layout (option 2) at all, or is the mixed width more hassle than it’s worth?

Thanks!

Put the hot spare wherever you like and move it wherever you like.

What parity are you planning to run? Why stop at 9 wide arrays?

Why are you planning around filling your available space? What capacity do you really need for the lifespan of this space heater?

Which array are you attaching L2ARC to and why?

Why not just post your prompt? I think if you thought about this by writing it you’d have changed the content.

I could go three vdevs 15 wide with raidz2.

What parity would you run at 8 wide or 15 wide?

I currently set up l2arc just because it barely gets any hits, but I don’t need that 2tb samsung 990 anyway. I think the hit rate might be 5pct

Some seem to prefer narrower vdevs though. What is your view?

I dont “need” 90 % of my homelab.

Today I have about 309 tb of storage.

So I dont quite know why I plan to fill the space heater. Maybe because I like homelabbing or because I have too much disposable income.

Personally if I was going this wide (which I do) I’d go Z3 but that also depends on how upset you would be if you lost this data.

Often a waste IMO. There are some use cases but they are quite niche.

Have you considered a cold spare? If you are actively monitoring your system and therefore will be alerted when you have an issue just having an available disk in the system is often good enough.

Do you plan to have a backup of this data?

What about 4*11-wide raidz3 with one spare (or free slot)?
Capacity comes very close to your mixed width layout with better resiliency.

No.

PTP, BTN and the likes are my distributed backup.

For a few months I have been running a 3 wide stripe (raid0?). Works just fine.

I have a synology backup solution (c2?) for more critical data. Much of that critical data is also mirrored on onedrive.

Raidz3 seems a bit overkill to me. But 11 wide might be an option (though I do like IOPS too).

Also, right now I have 16 drives. So 2x11 would mean spending some more money (6*650 chf approximately).

I have nothing useful to contribute, but hot damn that costs more than my car is worth without including the storage.

Oct 2025 - 410 USD for 96GB if I read my order correctly.

But yeah, today it might be 4x.

My 12 bay ssd array (7.6 TB kingston dc600) could also buy a car or two. But when I bought those a disk was 600 chf or such. Crazy times as we all know

Actually, you don’t need much IOPS for media storage. Let’s say you have 10 active torrents at a time, and 5 Plex streams. That comes to ~15 IOPS (but don’t quote me on that). AIUI, any pool layout would give you single-drive IOPS or better. Modern enterprise HDDs give ~100-500 IOPS.

AFAIK, there aren’t any. ZFS relies on the drive’s UUID. So you would even be able to move your drive to JBOD after resilvering.

IIRC, it’s generally recommended to not remove the dead/unhealthy drive before resilvering the new drive.

Mixed width can be ok, but your pool would get the worst performance/reliability of each vdev.


Regarding the pool layout, this white paper can be helpful. At least for understanding trade-offs.

Also, raid(z) is not a backup (z). Strictly speaking, raids/mirrors are the tools for HA, not the tools for preventing data loss.

Thanks. I might have some torrent clients reading and writing data though.

These will need much more IOPS than Plex.

I probably have 3 plex streams at most.

Is it that important to keep all these torrents fully alert? I do no think that you have a hard need for IOPS.
That leaves a media pool that is way too large for a full backup—and probably does not warrant it. So you’ll backup only the very most important files, accept to lose the rest if the building burns down, and try to have some resiliency in the array for lesser catastrophes. Where you set the cursor for the latter is entirely up to you.

Usually you can adjust the number of simultaneously active torrents (or global number of active peers). So, even if you’re gonna contribute 1Gbps upload, something around 20-30 will eat that bandwidth.

Would the SSD be better as a write cache? I don’t mean as a SLOG, but point your torrent client at it for incomplete torrents. This will reduce array write IOPS significantly, and may reduce fragmentation.

Width depends on why you need it. It’s hard to follow what you’re planning to use this for. But for torrents with a robust ARC you’re probably wanting to prioritise space. The x15 z3 (or z2) makes a lot of sense in that context. A small number of big arrays is easier to administer/maintain too.

Agree with others re:IOPS. A sane torrent client config with 1M record size shouldn’t stress the array. You probably don’t need to optimise for IOPS.

Thanks.

I have a 12 bay ssd raid F1 with 7.6 TB drives. The vast majority of my torrents go there while incomplete. I think this might be also help with fragmentation (maybe that is not the right term) as full consecutive writes will to onto TrueNas most of the time.

Well, seeding torrents seems like random IOPS of relatively small blocks. But I will survive.

4x 11 wide, 5x 9 wide are my preference right now.

Thanks for all the feedback.