Leaked space: vdev 10, offset 0x10ffddd9c000, size 3145728

TrueNAS General
, ,
1

Hello,

while i was using zdb command it aborted after 20 minutes and i got the following:

leaked space: vdev 10, offset 0x11bd90694000, size 3145728
leaked space: vdev 10, offset 0x11bd99894000, size 6291456
leaked space: vdev 10, offset 0x11bd9a194000, size 3145728
leaked space: vdev 10, offset 0x11bd9a594000, size 6291456
leaked space: vdev 10, offset 0x11bd9ac94000, size 214958080
leaked space: vdev 10, offset 0x11c585d6f000, size 4194304
leaked space: vdev 10, offset 0x11c5950ae000, size 4194304
leaked space: vdev 10, offset 0x11c598bca000, size 1048576
leaked space: vdev 10, offset 0x11c59ca69000, size 2097152
leaked space: vdev 10, offset 0x11c5a3162000, size 3145728
leaked space: vdev 10, offset 0x11c5a6c20000, size 1048576
leaked space: vdev 10, offset 0x11c5a6dea000, size 1048576
leaked space: vdev 10, offset 0x11c5a73e6000, size 1048576
leaked space: vdev 10, offset 0x11c5ad910000, size 5242880
leaked space: vdev 10, offset 0x11c603bcc000, size 143360
leaked space: vdev 10, offset 0x11c6166c9000, size 5242880
leaked space: vdev 10, offset 0x11c61dca6000, size 5242880
leaked space: vdev 10, offset 0x11c6203a6000, size 6291456
leaked space: vdev 10, offset 0x11c6225a6000, size 4194304
leaked space: vdev 10, offset 0x11c62a76f000, size 3145728
leaked space: vdev 10, offset 0x11c62c604000, size 2097152
leaked space: vdev 10, offset 0x11c634533000, size 5242880
leaked space: vdev 10, offset 0x11c644164000, size 4194304
leaked space: vdev 10, offset 0x11c648358000, size 2097152
leaked space: vdev 10, offset 0x11c64862e000, size 1048576

There are alot more lines. What does that mean?

Thank you

2

What parameters in the zdb command? Why? What was the reason to do this in the first place?

What is your pool? What is its vdev layout? Are you using deduplication? Block-cloning? Have you ever removed an entire stripe or mirror vdev from the pool? What version of TrueNAS? ZFS?

Have you run a scrub before this? Did it result in any errors?

Try to understand from the perspective of those you’re asking for help. What is anyone supposed to do with your post?

3 Likes
3

i used this command: zdb -bbbs. The reason was to check the small files cause i am using a special vdev (2 way mirror) for metadata and small blocks. The pool is 10 x MIRROR | 2 wide with that special vdev. I am not using dedup or block cloning.

Its ElectricEel-24.10.2.4 with ZFS. I have never removed a vdev, i have only added them.

I do a Scrub and a Periodic Smart Test every Sunday. Last scan took about 1 day 21 hours 8 minutes 47 seconds

4

AFAIK if the pool is active, you might get unexpected results with some zdb debugging.

Is the pool state healthy? Does it report any errors? Have you recently run a scrub?

I wouldn’t worry about what zdb says about leaked space on an active pool if the pool is healthy and passes a scrub.

I’ll defer to the low level experts on this.

5

There are no errors and the pool is healthy:

Pool Status:

Online

Total ZFS Errors:

0

I will wait for the next scrub and disk health check and will return here :slight_smile:

Thank you

1 Like
6

Unless you’re specifically seeking out leaked space (which can happen, if you encounter metadata corruption when attempting to delete files) I recommend you add the -L parameter as well, so zdb -Lbbbs to significantly reduce the time this command takes.

Also, note that zdb has a number of dangerous flags in it, so make sure you know what you’re running and that it might generate a large amount of I/O to your pool. :slight_smile:

2 Likes
7

Going off on a limb here. The likelihood of this happening is higher with non-ECC RAM? Using ECC RAM can mitigate this?

OP has ECC RAM I believe.

8

Yes, we use ECC RAM (256 GB on that system):slight_smile:

I have tried it with zdb -Lbbbs and i got the results i needed without the leaked space errors. What does that mean? Is my pool still in good shape?

EDIT: The output of the zdb tells me that i have 45 cksum errors on my special vdev.

Thank you

9

It can help prevent that, but an interrupted deletion combined with a drive failure (eg: power outage that takes out a disk) can also cause leaked space. The list isn’t exhaustive.

I’d check up on your special vdev devices - CKSUM can often be attributed to an error in the cabling or backplane, so if they’re SATA/SAS check the cable, if they’re NVMe check that they have sufficient airflow and aren’t overheating.

2 Likes
Please help me Identify ZFS Errors