Corrupted files. Need help

Tha_reaper · October 5, 2025, 10:59am

Because of bad RAM on my previous build, some errors slipped into my pools. On my main spinning rust storage pool i was able to redownload all affected files, but on my SSD apps pool some errors persist and i’m encountering a couple of problems mainly in backing up my data. My apps pool is a mirrored pool.

Let me start by saying that all my apps appear to be running fine. I use a couple of apps from the ix catalog, and i run dockge with around 15 containers.

my output of zpool status -v apps

  pool: apps
 state: ONLINE
status: One or more devices has experienced an error resulting in data
        corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
        entire pool from backup.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-8A
  scan: scrub repaired 0B in 00:39:14 with 4 errors on Fri Sep 26 11:14:20 2025
config:

        NAME                                      STATE     READ WRITE CKSUM
        apps                                      ONLINE       0     0     0
          mirror-0                                ONLINE       0     0     0
            1acc454d-fce4-4ed1-842b-b792957ff6e1  ONLINE       0     0    22
            cef98fd0-12ef-4c21-8942-a717cdc593e5  ONLINE       0     0    22

errors: Permanent errors have been detected in the following files:

        apps/ix-apps/docker@auto-2025-10-05_00-00:/overlay2/6a7bb2534e25a766065e7db4aa5926dfff02c25e77df992c1cf079af2c795cac/diff/usr/lib/libreoffice/program/libodbclo.so
        apps/ix-apps/docker@auto-2025-10-05_00-00:/overlay2/6a7bb2534e25a766065e7db4aa5926dfff02c25e77df992c1cf079af2c795cac/diff/usr/lib/libreoffice/share/palette/standard.sob
        /mnt/.ix-apps/docker/overlay2/6a7bb2534e25a766065e7db4aa5926dfff02c25e77df992c1cf079af2c795cac/diff/usr/lib/libreoffice/program/libodbclo.so
        /mnt/.ix-apps/docker/overlay2/6a7bb2534e25a766065e7db4aa5926dfff02c25e77df992c1cf079af2c795cac/diff/usr/lib/libreoffice/share/palette/standard.sob
        apps/ix-apps/docker@auto-2025-10-04_00-00:/overlay2/6a7bb2534e25a766065e7db4aa5926dfff02c25e77df992c1cf079af2c795cac/diff/usr/lib/libreoffice/program/libodbclo.so
        apps/ix-apps/docker@auto-2025-10-04_00-00:/overlay2/6a7bb2534e25a766065e7db4aa5926dfff02c25e77df992c1cf079af2c795cac/diff/usr/lib/libreoffice/share/palette/standard.sob

my main method of backing up my apps pool is rsyncing daily to my spinning rust pool (and syncing that to backblaze b2).
when using my cron job rsync -ar /mnt/.ix-apps/ /mnt/Klemmers/backupapps/ix-apps --delete
i see the following errors which match the zfs errors:

rsync: [sender] read errors mapping "/mnt/.ix-apps/docker/overlay2/248777748c7028bf5e6635732ca15643f8316553314924eb93d3310f11a421eb/merged/usr/lib/libreoffice/program/libodbclo.so": Input/output error (5)
rsync: [sender] read errors mapping "/mnt/.ix-apps/docker/overlay2/248777748c7028bf5e6635732ca15643f8316553314924eb93d3310f11a421eb/merged/usr/lib/libreoffice/program/libodbclo.so": Input/output error (5)
ERROR: docker/overlay2/248777748c7028bf5e6635732ca15643f8316553314924eb93d3310f11a421eb/merged/usr/lib/libreoffice/program/libodbclo.so failed verification -- update discarded.
rsync: [sender] read errors mapping "/mnt/.ix-apps/docker/overlay2/248777748c7028bf5e6635732ca15643f8316553314924eb93d3310f11a421eb/merged/usr/lib/libreoffice/share/palette/standard.sob": Input/output error (5)
rsync: [sender] read errors mapping "/mnt/.ix-apps/docker/overlay2/248777748c7028bf5e6635732ca15643f8316553314924eb93d3310f11a421eb/merged/usr/lib/libreoffice/share/palette/standard.sob": Input/output error (5)
ERROR: docker/overlay2/248777748c7028bf5e6635732ca15643f8316553314924eb93d3310f11a421eb/merged/usr/lib/libreoffice/share/palette/standard.sob failed verification -- update discarded.
rsync: [sender] read errors mapping "/mnt/.ix-apps/docker/overlay2/6a7bb2534e25a766065e7db4aa5926dfff02c25e77df992c1cf079af2c795cac/diff/usr/lib/libreoffice/program/libodbclo.so": Input/output error (5)
rsync: [sender] read errors mapping "/mnt/.ix-apps/docker/overlay2/6a7bb2534e25a766065e7db4aa5926dfff02c25e77df992c1cf079af2c795cac/diff/usr/lib/libreoffice/program/libodbclo.so": Input/output error (5)
ERROR: docker/overlay2/6a7bb2534e25a766065e7db4aa5926dfff02c25e77df992c1cf079af2c795cac/diff/usr/lib/libreoffice/program/libodbclo.so failed verification -- update discarded.
rsync: [sender] read errors mapping "/mnt/.ix-apps/docker/overlay2/6a7bb2534e25a766065e7db4aa5926dfff02c25e77df992c1cf079af2c795cac/diff/usr/lib/libreoffice/share/palette/standard.sob": Input/output error (5)
rsync: [sender] read errors mapping "/mnt/.ix-apps/docker/overlay2/6a7bb2534e25a766065e7db4aa5926dfff02c25e77df992c1cf079af2c795cac/diff/usr/lib/libreoffice/share/palette/standard.sob": Input/output error (5)
ERROR: docker/overlay2/6a7bb2534e25a766065e7db4aa5926dfff02c25e77df992c1cf079af2c795cac/diff/usr/lib/libreoffice/share/palette/standard.sob failed verification -- update discarded.
file has vanished: "/mnt/.ix-apps/docker/overlay2/828208f51df2b8fa7fedab67ca932d585bcb99d3441972e751fcc66da7e69abe/diff/tmp/.org.chromium.Chro-ium.PcSRUk"
IO error encountered -- skipping file deletion
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1338) [sender=3.2.7]

in dockge i also have a container that is impossible to delete. that gives me the following error:

 ✘ Container flaresolverr  Error while Removing                                                     0.0s 
Error response from daemon: container 84291cbb12e239debc408d06497b0ad1811b1967caaf4a28a6329b996b97999a: driver "overlay2" failed to remove root filesystem: unlinkat /mnt/.ix-apps/docker/overlay2/828208f51df2b8fa7fedab67ca932d585bcb99d3441972e751fcc66da7e69abe/diff/tmp: directory not empty

I hope someone can help me solving these errors.

oxyde · October 5, 2025, 12:05pm

Do you have snapshot enabled?
With rsync you have also corrupted your backup, i don’t see any option to recover those files… Hope i’m wrong

Tha_reaper · October 5, 2025, 12:22pm

Snapshots have been pruned because those files have been corrupted for some time already. So I will not be able to recover them from snapshots.
I would be fine with just deleting those files if that’s possible.

The rsync task was set up after the corruption took place, so no damage done there… Well at least not more than there was before.

NugentS · October 5, 2025, 4:31pm

of those 6 files - the first and last two are in snapshots - so just delete the snapshots and away they will go.

The middle two - I have no idea.

Of an immediate concern is the chksum errors - which are often caused by cabling errors. So you MAY have hardware issues.

Whats your hardware?

Tha_reaper · October 5, 2025, 6:12pm

The checksum errors were there from before my rebuild, so most likely caused by the same problem that caused the corrupted files. They have not increased since the rebuild.

My hardware:
Case jonsbo N2
Motherboard: CWWK 12th/13th generation Core 8505 NAS/i5-12450H motherboard/4x2.5G/6 SATA3.0/6x M.2 NVMe/dual SFF-8643. (For a lack of a better name)
HDDs: 5x HSGT ultra star 6TB
Bootdrive: SSD_M.2_PCIe3_128GB_InnovationIT
apps pool: 500GB kingston SSD + 500GB samsung T5 USB SSD mirrored
RAM: Corsair DDR5 SODIMM Vengeance 2x32GB 4800

I want to replace the T5 USB SSD asap with a 500GB nvme drive, but im afraid i will need to find a way to get rid of the errors first.

Tha_reaper · October 5, 2025, 6:19pm

i have deleted the 2 snapshots, and indeed 4 of the 6 errors went away.

now im left with

  pool: apps
 state: ONLINE
status: One or more devices has experienced an error resulting in data
        corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
        entire pool from backup.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-8A
  scan: scrub repaired 0B in 00:39:14 with 4 errors on Fri Sep 26 11:14:20 2025
config:

        NAME                                      STATE     READ WRITE CKSUM
        apps                                      ONLINE       0     0     0
          mirror-0                                ONLINE       0     0     0
            1acc454d-fce4-4ed1-842b-b792957ff6e1  ONLINE       0     0     0
            cef98fd0-12ef-4c21-8942-a717cdc593e5  ONLINE       0     0     0

errors: Permanent errors have been detected in the following files:

        /mnt/.ix-apps/docker/overlay2/6a7bb2534e25a766065e7db4aa5926dfff02c25e77df992c1cf079af2c795cac/diff/usr/lib/libreoffice/program/libodbclo.so
        /mnt/.ix-apps/docker/overlay2/6a7bb2534e25a766065e7db4aa5926dfff02c25e77df992c1cf079af2c795cac/diff/usr/lib/libreoffice/share/palette/standard.sob

apparently deleting the snapshots also cleared the checksum errors

SmallBarky · October 5, 2025, 6:33pm

Is it possible to reload the docker or power off the app, remove the files and get it to reload those files? Can you figure out what those files go to? I’m not familiar with all that.

Tha_reaper · October 5, 2025, 6:46pm

good suggestion. something that im trying to figure out myself at the moment. but i have no bloody clue how to find out which container the corrupted files belong to.
So at the moment im trying to stop all my running containers, do a docker system prune -a and bring them back up to see if that helps

Tha_reaper · October 5, 2025, 7:28pm

oh my god that did the trick. I still don’t know which container was the bad guy, but the errors are gone, no data is lost, and my pool shows as healthy again.