Where is StoragePool/.system/?

TrueNAS General
1

Hi I have been running Truenas Scale 24.10.2.2 and recently I got a pool error.
I notice the corrupted files are from /var/db/system/netdata. I deleted the db files, did a scrub and run zpool clear StoragePool. Now the disk has zero errors. But when I run zpool status -v StoragePool I got the following:

    NAME                                      STATE     READ WRITE CKSUM
    StoragePool                               ONLINE       0     0     0
      mirror-0                                ONLINE       0     0     0
        80f96cf0-67f9-11ed-ade5-0cc47aa90636  ONLINE       0     0     0
        8106a979-67f9-11ed-ade5-0cc47aa90636  ONLINE       0     0     0
      mirror-1                                ONLINE       0     0     0
        8d6b4629-67f9-11ed-ade5-0cc47aa90636  ONLINE       0     0     0
        8d5efb66-67f9-11ed-ade5-0cc47aa90636  ONLINE       0     0     0

errors: Permanent errors have been detected in the following files:

    StoragePool/.system/netdata-ae32c386e13840b2bf9c0083275e7941:/dbengine/datafile-1-0000000208.ndf
    StoragePool/.system/netdata-ae32c386e13840b2bf9c0083275e7941:/dbengine/datafile-1-0000000210.ndf

truenas_admin@truenas[~]$

So where is this StoragePool/.system/ located?? I have tried to export/re-import the pool, upgraded to 25.04.1 (because it is not forcing you to use netdata) But the error is still there.

IMHO every time a disk error occur it is somehow related to netdata. I am glad you guys remove it in Fangtooth

2

And one more question: Should I (and how) to get rid of those 2 ndf files??

3

The “System Dataset” is where TrueNAS itself stores logs and as you’ve seen, netdata files. if you are seeing no other consequences of those files being corrupt I would just delete them and rescrub… (ZFS fails I/O of corrupted files anyways so they aren’t even accessible now)

On the other hand you really should figure out why your pool is experiencing corruption severe enough that your mirror based pool can’t heal them. You are lucky that it wasn’t user-data that got corrupted (if a scrub doesn’t reveal anything else).

4

Thanks for your reply. My biggest problem is I can’t find the netdata file to delete. And what is this path with a semi colon?

StoragePool/.system/netdata-ae32c386e13840b2bf9c0083275e7941:/dbengine/datafile-1-0000000208.ndf

As far as I know, my user data is not affected, but I am planning to wipe and restored from backup (my backup NAS was offline for months so I need to first redo the backup)

Intel(R) Xeon(R) CPU E5-2640 v3 @ 2.60GHz
MB: Supermicro X10 SRI-F
64 GB ECC DDR4
Rosewill RSV-L4412U 12 bays hotswap
4x Seagate Ironwolf Pro 10GB running at mirrored VDEV (main storage)
2X WD blue SN570 1TB mirrored (Apps and VM)
Intel X540 T2 (genuine with yottamark)
5

Thanks for your reply. My biggest problem is I can’t find the netdata file to delete. And what is this path with a semi colon?

StoragePool/.system/netdata-ae32c386e13840b2bf9c0083275e7941:/dbengine/datafile-1-0000000208.ndf

As far as I know, my user data is not affected, but I am planning to wipe and restored from backup (my backup NAS was offline for months so I need to first redo the backup)

Finally, I forgot to include the hardware spec. See my signature

6

the “System dataset” is much like the docker and instances datasets where they are not shown in the Datasets UI and have “Legacy” mountpoints. (ZFS’s legacy here is misleading and just means it’s manually mounted)

you can find the netdata files under /var/db/system/netdata/dbengine/

7

This is all I found:

root@truenas[/var/db/system/netdata/dbengine]# ls -l
total 29118
-rw------- 1 netdata netdata 23805952 Jul 3 17:48 datafile-1-0000000001.ndf
-rw------- 1 netdata netdata 12517376 Jul 4 03:00 datafile-1-0000000002.ndf
-rw------- 1 netdata netdata 2129920 Jul 3 17:48 journalfile-1-0000000001.njf
-rw------- 1 netdata netdata 1056768 Jul 4 02:54 journalfile-1-0000000002.njf

there is no datafile-1-0000000208.ndf