I’ve acquired a free Cisco UCS server and a load of SAS SSDs, the problem is the HBA in the system cannot operate in IT mode (it’s Cisco 12G UCS SAS Modular RAID controller).
I want to use the system as backup target. So, given that neither option is great for TrueNAS and ZFS, what’s the best option:
RAID all the disks together at the hardware, present it as single virtual disk to TrueNAS and add it to pool on its own with no ZFS redundancy
or
expose all the disks to TrueNAS as JBOD and build VDEVs and a pool as usual
Please don’t suggest I put an IT mode HBA in it, that’s not an option at this time. I just want to know what’s the least worst option with the hardware I have at the moment.
Either option could be bad. It may not behave well with ZFS and it makes any pool recovery or migration harder. See number 5 in the original posting. You could try JBOD and running it for a backup target for a while but you may get stuck having to recreate the pools if you get a plain HBA later.
See if you can disable any write cache on the RAID card. Plus, check out any other options in it’s configuration that may help make it more reliable.
One of the key problems using ZFS with hardware RAID cards is the write cache and it’s ability to re-order writes.
Next, adding a UPS can help reduce the chances of a power loss corrupting your pool. (ZFS was specifically designed to PREVENT any power loss or OS crash from corrupting the pool. However, hardware RAID cards are not part of that design!)
ZFS purposefully does writes in a specific order, part of the COW, Copy On Write. But the other half of the ZFS write order, is to make sure that directory entries for any data file updates is written AFTER the data is actually written. (In reality, it is more complex than that, but that gives you the idea.) Then if a OS crash or power loss occurs, between the data writes and the directory writes, the data writes are lost. (Just like any other file system.) BUT, the pool is not corrupt.
However, if the opposite were to happen, directory writes before data writes, and the OS crash or power loss occurs, then you have directory entries pointing to garbage and potentially have corrupted the pool beyond repair. Remember, directory entries can point to further directory entries, so a directory entry that is supposed to point to functional, usable data structures, (aka another directory entry), may not do so and all that data could be gone.
Backup target of what?
ZFS replication would need ZFS, and this hardware is not suitable.
But as a target for rsync under whatever (non-TrueNAS) OS you fancy, a hardware RAID should be fine.
Thanks for the detailed, reasoned response @Arwen. That all makes a lot of sense, and it’s good to understand just why hardware RAID is not really compatible with ZFS.
The system has dual PSUs and is on a room UPS (with generator backup), so there’s more chance of me accidentally unplugging it than there is a power outage
I’ll have at the RAID controller when I’m back on site on Monday and see if I can disable the cache on it.
