Hello all, hoping to get a little help with simplifying my setup. Currently I have three pools on my main server. tank, dozer, and backups.
tank: raidz2 11 wide - total capacity of 137.04 TiB
dozer: mirror 2 wide - total capacity of 1.76 TiB
backups: 2 x raidz1 6 wide - total capacity of 69.62 TiB
I’m planning on moving backups to a less powerful server and hopefully remove it to another physical location at some point, it exists in a rack mount disk shelf and connected with two SAS cables. But it is only use for backups of critical or hard to replace files.
dozer is currently my app pool and where I put configuration files and such for applications hosted in docker. I’m really not using much of the space there at this point and was wondering if I could improve the performance of my tank pool if I moved the data from dozer to tank and then setup the dozer ssds as a special vdev for smaller files, etc.
Other than the potential downfalls with moving the data and reconfiguring everything to point to the new location, etc are there problems with this idea that I should consider?
This server is more or less my all-in-one solution for self hosting things. Nextcloud, Plex, Ollama, Open web UI, etc as well as backup destination for my wife and I’s computers.
AIUI, you won’t be able to store your apps’ data in an “all-SSD” dataset. Unless you use custom apps (aka vanilla docker compose). And even in that case, images themselves would be in the “ix-apps” dataset. And you won’t be able to create “all-SSD” zvol (so no VMs on SSDs).
I am personally a fan of sVDEV, as it provides some form of tiered storage. But this is a very opinionated topic. This thread provides other opinions.
Also, I’m almost 100% sure the benefit isn’t worth the migration headache. Well, strictly speaking, it depends on your use case. Generally, you would get a faster directory listing.
Is the Dozer pool SSD or NVME? I would leave it as is for APPs.
Do you have performance problems with Tank?
A three wide mirror sVDEV would be the match for a Raid-Z2 pool to give the same drive reduntancy of two drives that can fail. You wouldn’t be able to remove sVDEV without recreating the pools, in your case.
iX Systems pool layout whitepaper
Special VDEV (sVDEV) Planning, Sizing, and Considerations
NVME for dozer. As much as I’d like to do a three wide mirror, I’m completely out of M.2 slots on this MB and don’t have the PCIe slots to add an additional card.
Once I move backups to a separate box, I’d have two spare SAS connectors inside the box that I could use for ssds.
As for performance, I’m unsure if I’m having a problem currently. There are various tasks that seem very slow, but unsure if moving the metadata to ssd would help with it. Mostly around backup backup tasks. Doesn’t seem to spent much time actually sending data back and forth as much as preparing and analyzing.
Edit: fixed where I said NVME for tank, it is actually dozer.
Makes sense. I’m using a combination of Time Machine and Arq for local backups. I already have a lot of ram, so many trying to tune that is a good first step.
Key to L2ARC and rsync is the benefit is best for WORM Pools or pools where the contents don’t change much.
That makes rsync fly since it goes down every directory tree looking for changes. Hence, if L2ARC can confirm nothing has changed (and does so at SSD speed, because that’s where L2ARC is typically stored) then traversals for most files can get cut down dramatically.
On the other hand, L2ARC has to miss something before it can be added to the L2ARC. So for data sets where there are a lot of changes, sVDEV will be more performant by virtue of always being “hot” but sVDEV is no free lunch either.
My dataset is mostly WORM but Time Machine corrupts itself occasionally and then all those blocks can get re-written. For my use case, rsync makes sense on a cross-file system basis (ie backing up ZFS files to NTFS or APFS). For zfs-zfs backups, don’t even bother with L2ARC or sVDEV, just use a replication step, nothing is faster.
TM probably won’t benefit from sVDEV. I didn’t make a direct comparison, but my TM backup to TrueNAS (with sVDEV) is as slow as it was to my old (non-ZFS) NAS. Looking through the history is perhaps a bit faster (than on my old NAS), but again it’s not a fair comparison by any means.
Happened to me once. Or mb twice. The last time I just reverted the dataset to the last “healthy” weekly snapshot.
Yeah, for my server backups, I’m using Truenas’s built in functionality to send to the internal backups pool. Once I move that to a separate box, I’ll do the same exact over the network.
I’d like to run a Linux box booting off ZFS for my desktop so I could just send snapshots for backup, but every time I’ve looked into doing it I got bogged down trying to figure out the right combination of things to do to make it actually boot. Maybe I just need to setup a machine with non-zfs boot and use zfs for the actual data storage.
Look, none of this is as easy as it ought to be, hence the need for TrueNAS in the first place. The demi-gods here can set up a functional equivalent to TrueNAS in most respects using nothing more than Debian or FreeBSD and a bit of time. Where TrueNAS shines is making all that far more accessible to the likes of you and me.
The dual FS system is nothing new - see Pis who will have a FAT and a Ex3 or whatever FS on each boot disk. The trick is in figuring out how to reliably back stuff up, especially the config files that are worth their weight in gold. Downloads, etc. are peanuts, config files where stuff just works as expected represent hours, if not days of my time.
NB that you would lose your history from the last week. I’m personally okay with that, as I store backups just for emergencies and rarely use file history.
Well, corrupted backup doesn’t work for history, but files can be restored manually.
There is that but there is also the much smaller load that your approach imposes on the backup mechanism. Your approach likely preserves 99% of the TimeMachine corpus, which means that past snapshots are still relevant and that they can be used if my boss has an epiphany about a file she changed two weeks ago, or whatever.
My approach ends up with files that are accurate as of this week, a huge backup has to be transferred to the backup server, ditto all the offline repositories, and a giant set of snapshots that currently bloat my NAS file system to the tune of 3% before getting shipped into the TrueNAS bitbucket.
