I’m on a Dell R720 with an h710 raid controller
I have a 2 disk mirror that I’m replacing a disk. 2 12TB disks. It gets to about 75% then hangs for days until read/writes on the pool are completely broken. All I can do is a reboot.
When I do a zpool status it shows:
errors: 5 data errors, use ‘-v’ for a list
But with a -v
errors: List of errors unavailable: no such pool or dataset
In searching the forums, I found recommendations for scrubbing it twice then the resilver will work, but I’m not sure how to cancel the resilver and it also won’t let me export/disconnect the pool. Any tips?
Another odd thing is that the resilver status scan shows this -
scan: resilver in progress since Thu Aug 22 12:27:19 2024
4.14T / 4.20T scanned, 267G / 4.20T issued at 271M/s
119G resilvered, 6.19% done, 04:14:27 to go
The scan portion never goes above about 98% - 4.14T/4.20T
Possibly due to a dataset that was removed?
If I’m unable to export the pool after a reboot, is my only option to shutdown, remove the resilver disk and it will cancel? Then I can run the scrub a few times to remove the missing errored files. Then retry the resilver?
Hardware RAID controllers are a bad thing with ZFS.
I don’t know the Dell models, but have you loaded IT firmware on it?
If you have not already loaded IT firmware, don’t because that may cause loss of your existing data. (Though with a HW RAID Mirror that may not be a problem.)
Next, if zpool status no longer shows the pool, it may be exported. Use the following command to check if the pool is exported and it’s rough status; zpool import
Removing a dataset won’t cause a ZFS scrub to fail. ZFS was designed to have all actions taken “live”.
Please describe your hardware more completely. Like if you have additional pools. Also list your TrueNAS SCALE version.
I have not loaded IT firmware. zpool status still shows the pool. I have the r720 serving the OS all drives individually and Truenas handles the raid setup. I do have a 2 other pools. A software pool and a boot pool.
I’m on the latest version 24.04.2
Someone with more knowledge of SAS controllers will have to help. We need to make sure your “h710 raid controller” is suitably setup. If not, that is known to cause problems.
I ended up backing up the pool to a new disk, then recreating the pool. My h710p controller served up the disks individually to truenas. Shouldn’t have been a factor I think.
There is a SERIOUS difference between a hardware RAID controller serving up disks as JBOD, and a plain disk controller, (aka HBA), supplying access to the disks.
Now to be fair, such a hardware RAID controller supplying disks in JBOD might work perfectly for years. And you may never have problems. However, if a problem does arise, like your resilver hanging, we prefer to eliminate known causes of problems like hardware RAID controller cards.
In case a more detailed technical explanation helps:
HBA in IT mode - the disks are presented to the O/S in raw format - absolutely zero changes are made to how the disk is presented, and zero changes are made to commands issued by the O/S to the disk.
HBA in RAID mode - the whole concept of RAID mode is that the RAID card presents pseudo disks (i.e. made up of RAIDx configurations of several disks) and translates the I/Os presented by the O/S into different I/Os to the disks themselves. If you set the RAID card to present single disks (i.e. JBOD) you would hope that it does the same as in IT mode, but IMO (personal hunch) it is actually more likely that it presents disks like a one-disk stripe i.e. it is presented as a pseudo-disk with the same size as the original disk, but the I/O commands still go through some translation functionality.
Well, if we are getting into details, (and possibly for other future readers), one issue with hardware RAID JBOD is optimized elevator seeking and writing.
Most of the time this does not impact ZFS. However, during a failure or crash, ZFS counts on it’s write transactions to be written in a specific order. Specifically;
Data
Metadata describing where the new Data is
Critical Metadata describing where the new Metadata is, AND activating the new Metadata, (which activates the new Data)
All this is designed to overcome long initial file system checks, (aka FSCK or CHKDSK). ZFS is intended to be always consistent on disk. The new writes go into un-used space, (aka Copy on Write), so the old Data & Metdata still exists until the last moment when the Critical Metadata is written, and activates the new Data & Metadata.
Where this can and does go wrong with hardware RAID controllers, is when a hardware RAID controller writes data out of order because it thinks it can “optimize” write speed by writing Metadata or Critical Metadata first, before Data. And if their is a crash, the ZFS pool now has some corruption. Potentially un-fixable and possible more than the original write intended.
Their is no FSCK for ZFS. If there is corruption at a certain level, it is impossible to fix. Sometimes the fix is to remove a directory tree and restore it from backups, EVEN if only 1 file was impacted. Other times you might need to destroy an entire Dataset or zVolume to remove the corruption. Again restoring what you can from backups.
To be clear, none of the data loss is unique to ZFS. All file systems can be corrupted. It is just you may not know about it. And if you do, you may not know exactly what is corrupt.
If you value your data, listen to the experienced users here. They are giving you outstanding advice.
I would do the following myself:
Check the firmware on the HBA, find out if it is IT or not. If it is IT mode then you are in a good place. If it is not IT firmware then go to step 3, otherwise go to step 7.
Backup all my data.
Destroy the pool.
Flash the HBA to IT mode.
Create a new pool, do you need redundancy? How important is your data? those two questions are a good starting place for determining the pool structure.
Run the command zpool -v status and list it in the forum (here) using code “</>” on the bar above. Also list how important your data is, basically answer the questions above. This will show us exactly what you have for a pool.
If the experts here agree or disagree, they will tell you. If things are looking good, restore your data, if they are not looking good, it’s your choice to take the advice or not.
But first, find out what firmware is running that HBA. Without knowing that, we are all making assumptions and speculating. Give us some hard facts and you will get a more specific answer and much faster.
