Persistent ZFS Pool Errors and Data Corruption Issues – Assistance Needed

[stainzor]

I am experiencing issues with a ZFS pool (named “DISK”) running on a Proxmox VE server with TrueNAS as a VM. The pool consists of two 20 TB disks in a mirrored configuration (RAID 1). The main problem is that the pool reports data errors and shows permanent data loss for certain files. I have conducted multiple zpool scrub operations, and while they report no repaired errors, data errors persist. Attempts to import the pool with various flags (-f, -o readonly=on) often result in I/O errors and segmentation faults.

SMART tests have been run on the disks, showing no critical errors, yet TrueNAS encounters access issues with specific blocks, indicating underlying read problems. Additionally, zdb commands reveal block errors and leaked space.

I am seeking assistance with:

• Understanding the root causes of these data errors.
• Recommendations on potential repair options to recover data, given that I do not have a separate backup for this data.

Any insights or advice on how to proceed to minimize further data loss and potentially recover the data would be greatly appreciated.

[Stux]

@stainzor

I am seeking assistance with:

• Understanding the root causes of these data errors.

@stainzor

running on a Proxmox VE server with TrueNAS as a VM

In all seriousness, unless you’re using PCIe pass through to pass through a disk controller into the truenas, you’re doing it wrong

ZFS Pool offline (and possibly missing)

So, couple of things, The only reliable way of passing through HDs is to pass through the drive controller. Otherwise you are liable to corrupt the pool. And this is exacerbated by a bug/issue in Proxmox. More info here:

[Arwen]

@stainzor

The pool consists of two 20 TB disks in a mirrored configuration (RAID 1).

In addition to what @Stux wrote, it is always helpful to use the ZFS terminology. ZFS does not support RAID-1, but does support something similar, “Mirroring”. One difference is that a ZFS Mirror only mirrors active data. So during disk replacement, ZFS can bring it’s mirror back faster than RAID-1.

Using RAID-1 wording implies either hardware RAID, (highly discouraged), or external software RAID, (like at the Proxmox level). If using either, the question is “Why use TrueNAS?”

Here is a link to useful terminology:

Terminology and Abbreviations Primer

[etorix]

@Stux

unless you’re using PCIe pass through to pass through a disk controller into the truenas, you’re doing it wrong

And on top of that it seems that you should also blacklist the controller and/or disable Proxmox scanning for ZFS pools.

Can't import datapool

Hey @Gilfoyle - I remember trying to help out in Discord on this one. Guessing the pool is still unmountable? Proxmox isn’t supposed to grab pools without the cachefile set to its own /etc/zfs/zpool.cache path (TrueNAS uses /data/zfs/zpool.cache) but Proxmox also appears to have a system service that will import pools on disk scan. So if there’s something that triggers a disk scan on a host and the member disks are visible this is likely to cause the simultaneous access/force-import bug. Isolat…

[HoneyBadger]

@stainzor Sorry to hear that you’re having problems here. It sounds like you’re familiar with some of the deeper poking about with ZFS commands.

When you show the pool listing with zpool import does it show the previous system for importation as your Proxmox host?

If you have block errors and leaked space I have a suspicion that the pool was configured with individual drive passthrough in Proxmox, and your host picked up and mounted the pool. You may be able to rescue the situation by attempting to import with the -fF flag (force-rewind) - I would recommend also combining this with the -o readonly=on and -R /mnt flags.

If that fails, you may need to force-rewind harder with -fFX but that may require disabling pool scrub/scan on import as well.