Apps are about small transactions, for which raidz# is highly inefficient and mirrors are recommended. iXsystems further recommends to put apps on SSDs rather than HDD.
TrueNAS does automatically moves the system dataset from the boot pool to the first created pool, which might not be optimal if one begins with the “bulk storage” pool but it is always possible to move the system dataset through the GUI. The app/instances/VM pool, as far as I know, one has to declare it, TrueNAS does not choose—and in any case it can be moved.
Sh!t happens. That’s what resiliency aims to address.
As soon as expansion was initiated, the vdev had an extra member taking part in the redundancy scheme. Losing this disk was a second drive incident and a second lost drive.
Long expansion might be a consequence of excessive width.
Had you burnt the drive in before adding it?
This was not normal and should have been investigated…
Possibly, when used according to recommended practices.
ZFS was designed for enterprise use, and as most professional tools it can be a very efficient way to shoot oneself in the foot when used without proper care.
Yes, through the command line. Note that dRAID cannot be extended and relies on having suffcient spares. And “beyond an effective width of 40, dRAID3 gets pretty wacky” (read all posts by @jro in the thread).
If you haven’t shut down the NAS yet and can still access the pool, I’d strongly suggest looking for extra storage and backing up anything critical, as there’s no guarantee this pool will ever mount back. (Yes, it’s not my money here, but it’s also not my data…)