Added more drives, lost the pool

bacon · November 26, 2024, 10:57pm

Filed a bug report, let me know if you think I missed anything:
https://ixsystems.atlassian.net/browse/NAS-132753

Stux · November 26, 2024, 11:05pm

I think I got bit by this issue too, luckily, the two drives that I “lost” were just an L2ARC stripe.

But same cause, added some drives, and then the drive ids changed… and the drives that were “lost” were previously used with Windows systems, and because of Core → Scale had drive id rather than partid identifications (in core they were gptid)

Unfortunately, even if the issue is fixed in the future there are still timebombs out there waiting to go off.

Stux · November 26, 2024, 11:11pm

@HoneyBadger not sure if you’ve seen these threads… smoking guns?

HoneyBadger · November 27, 2024, 2:49am

Thanks for the ping; I’m pretty sure we manually nuke a given stretch of LBAs on drives that get added as part of the pool build process through the UI but now you’ve got me curious to verify exactly what’s done.

Stux · November 27, 2024, 6:13am

Looks like this was fixed in 24.10.0 RELEASE

github.com/truenas/middleware

NAS-131171 / 24.10.0 / fix disk.wipe (by yocalebo)

truenas:release/24.10.0 ← truenas:NAS-131171-24.10.0

opened 07:31PM - 21 Oct 24 UTC

bugclerk

+14 -3

This fixes disk.wipe to actually zero the first and last 32MB of the disk being …wiped. We have seen a few reports of users on SCALE where libblkid is reporting partition information on disks that should _not_ be there. There is a separate libblkid bug related to ZFS and the OS team is working on that separately, however, this was trivially reproducible by writing another filesystems label "magic string" and then running `disk.wipe` on it preventing that file system label from being removed. This boils down to the API in python interpreting and doing something with null characters instead of 0's. The fix is quite simple really and I've improved the `disk.wipe` test to ensure this does not get missed in the future. Tests are passing. Original PR: https://github.com/truenas/middleware/pull/14742 Jira URL: https://ixsystems.atlassian.net/browse/NAS-131171

Which now correctly nukes the first and last 32MB of the disk.

bacon · November 27, 2024, 8:34am

I can confirm that it HAS NOT been fixed. They may have fixed the wipe procedure, but they don’t actually use that procedure during format.

It’s very easy to reproduce, I’m on 24.10.0.2:

$ cat /etc/version
24.10.0.2

Let’s use the device /dev/sdd which perviously hat a vfat file system:

sudo blkid --probe /dev/sdd1
/dev/sdd1: UUID="7BCD-07F9" VERSION="FAT32" BLOCK_SIZE="512" TYPE="vfat" USAGE="filesystem" PART_ENTRY_SCHEME="gpt" PART_ENTRY_NAME="primary" PART_ENTRY_UUID="b651f7d3-02c9-495b-b306-7bb120baab30" PART_ENTRY_TYPE="ebd0a0a2-b9e5-4433-87c0-68b6b72699c7" PART_ENTRY_NUMBER="1" PART_ENTRY_OFFSET="2048" PART_ENTRY_SIZE="976771072" PART_ENTRY_DISK="8:48"

Let’s create a pool with it and see what happens. After creating the pool we have the following now:

$ sudo blkid --probe /dev/sdd1
blkid: /dev/sdd1: ambivalent result (probably more filesystems on the device, use wipefs(8) to see more details)

And we can confirm that indeed no wiping has been done on the drive:

$ sudo dd if=/dev/sdd1 count=8 bs=1024 2> /dev/null | hexdump -C
00000000  eb 58 90 6d 6b 66 73 2e  66 61 74 00 02 40 40 00  |.X.mkfs.fat..@@.|
00000010  02 00 00 00 00 f8 00 00  3f 00 ff 00 00 08 00 00  |........?.......|
00000020  d2 57 38 3a c0 d1 01 00  00 00 00 00 02 00 00 00  |.W8:............|
00000030  01 00 06 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000040  80 00 29 f9 07 cd 7b 4e  4f 20 4e 41 4d 45 20 20  |..)...{NO NAME  |
00000050  20 20 46 41 54 33 32 20  20 20 0e 1f be 77 7c ac  |  FAT32   ...w|.|
00000060  22 c0 74 0b 56 b4 0e bb  07 00 cd 10 5e eb f0 32  |".t.V.......^..2|
00000070  e4 cd 16 cd 19 eb fe 54  68 69 73 20 69 73 20 6e  |.......This is n|
00000080  6f 74 20 61 20 62 6f 6f  74 61 62 6c 65 20 64 69  |ot a bootable di|
00000090  73 6b 2e 20 20 50 6c 65  61 73 65 20 69 6e 73 65  |sk.  Please inse|
000000a0  72 74 20 61 20 62 6f 6f  74 61 62 6c 65 20 66 6c  |rt a bootable fl|
000000b0  6f 70 70 79 20 61 6e 64  0d 0a 70 72 65 73 73 20  |oppy and..press |
000000c0  61 6e 79 20 6b 65 79 20  74 6f 20 74 72 79 20 61  |any key to try a|
000000d0  67 61 69 6e 20 2e 2e 2e  20 0d 0a 00 00 00 00 00  |gain ... .......|
<.. output truncated ..>

HoneyBadger · November 27, 2024, 2:10pm

Thanks for the clear reproduction instructions @bacon

Should be fixed in 24.10.1 as shown in https://ixsystems.atlassian.net/browse/NAS-131171

bacon · November 27, 2024, 2:55pm

Has anyone actually tested it?

I did look at the pull request and there is no code change that would affect current behaviour.

All the pull request achieves is that now if you do a “quick wipe” instead of using ascii zero (0x30) it uses actual zero (0x00) for the overwrite. There is a difference in python between b'0' and b'\x00'.

I tested a quick wipe in 24.10.0.2:

$ sudo midclt call disk.wipe deviceredacted QUICK
< .. wait for job completion ..>
$ sudo dd if=/dev/deviceredacted count=8 bs=1024 2> /dev/null | hexdump -C
00000000  30 30 30 30 30 30 30 30  30 30 30 30 30 30 30 30  |0000000000000000|
*
00002000

But the issue is that there is no wipe done at all. Not the character with which the wipe is done.

The only place I could find in the middleware which uses wipe (in preparation for a new pool) is during boot pool formatting in src/middlewared/middlewared/plugins/boot_/format.py. Which isn’t relevant for regular pools.

SmallBarky · November 27, 2024, 3:15pm

Yes, we need to FLUSH the data and give it a hygienic WIPE. We don’t want previous filesystems getting stuck.

HoneyBadger · November 27, 2024, 7:19pm

I’m running 24.10.0.2 here as well, and it seems like it’s wiping with ASCII 0 (hex 0x30) when you either create a new pool or add a disk to an existing one.

truenas_admin@truenas[~]$ sudo dd if=/dev/sdd count=8 bs=1024 2>/dev/null | hexdump -C
00000000  eb 58 90 6d 6b 66 73 2e  66 61 74 00 02 10 20 00  |.X.mkfs.fat... .|
00000010  02 00 00 00 00 f8 00 00  3f 00 ff 00 00 00 00 00  |........?.......|
00000020  fe ff ff 01 00 40 00 00  00 00 00 00 02 00 00 00  |.....@..........|
00000030  01 00 06 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000040  80 00 29 c3 e2 95 05 4e  4f 20 4e 41 4d 45 20 20  |..)....NO NAME  |
00000050  20 20 46 41 54 33 32 20  20 20 0e 1f be 77 7c ac  |  FAT32   ...w|.|
00000060  22 c0 74 0b 56 b4 0e bb  07 00 cd 10 5e eb f0 32  |".t.V.......^..2|
00000070  e4 cd 16 cd 19 eb fe 54  68 69 73 20 69 73 20 6e  |.......This is n|
00000080  6f 74 20 61 20 62 6f 6f  74 61 62 6c 65 20 64 69  |ot a bootable di|
00000090  73 6b 2e 20 20 50 6c 65  61 73 65 20 69 6e 73 65  |sk.  Please inse|
000000a0  72 74 20 61 20 62 6f 6f  74 61 62 6c 65 20 66 6c  |rt a bootable fl|
000000b0  6f 70 70 79 20 61 6e 64  0d 0a 70 72 65 73 73 20  |oppy and..press |
000000c0  61 6e 79 20 6b 65 79 20  74 6f 20 74 72 79 20 61  |any key to try a|
000000d0  67 61 69 6e 20 2e 2e 2e  20 0d 0a 00 00 00 00 00  |gain ... .......|
000000e0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
000001b0  00 00 00 00 00 00 00 00  64 eb 95 05 00 00 80 00  |........d.......|
000001c0  01 00 0c fe ff ff 00 00  00 00 fe ff ff 01 00 00  |................|
000001d0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
000001f0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 55 aa  |..............U.|
00000200  52 52 61 41 00 00 00 00  00 00 00 00 00 00 00 00  |RRaA............|
00000210  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
000003e0  00 00 00 00 72 72 41 61  fc f7 1f 00 02 00 00 00  |....rrAa........|
000003f0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 55 aa  |..............U.|
00000400  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000c00  eb 58 90 6d 6b 66 73 2e  66 61 74 00 02 10 20 00  |.X.mkfs.fat... .|
00000c10  02 00 00 00 00 f8 00 00  3f 00 ff 00 00 00 00 00  |........?.......|
00000c20  fe ff ff 01 00 40 00 00  00 00 00 00 02 00 00 00  |.....@..........|
00000c30  01 00 06 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000c40  80 00 29 c3 e2 95 05 4e  4f 20 4e 41 4d 45 20 20  |..)....NO NAME  |
00000c50  20 20 46 41 54 33 32 20  20 20 0e 1f be 77 7c ac  |  FAT32   ...w|.|
00000c60  22 c0 74 0b 56 b4 0e bb  07 00 cd 10 5e eb f0 32  |".t.V.......^..2|
00000c70  e4 cd 16 cd 19 eb fe 54  68 69 73 20 69 73 20 6e  |.......This is n|
00000c80  6f 74 20 61 20 62 6f 6f  74 61 62 6c 65 20 64 69  |ot a bootable di|
00000c90  73 6b 2e 20 20 50 6c 65  61 73 65 20 69 6e 73 65  |sk.  Please inse|
00000ca0  72 74 20 61 20 62 6f 6f  74 61 62 6c 65 20 66 6c  |rt a bootable fl|
00000cb0  6f 70 70 79 20 61 6e 64  0d 0a 70 72 65 73 73 20  |oppy and..press |
00000cc0  61 6e 79 20 6b 65 79 20  74 6f 20 74 72 79 20 61  |any key to try a|
00000cd0  67 61 69 6e 20 2e 2e 2e  20 0d 0a 00 00 00 00 00  |gain ... .......|
00000ce0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000db0  00 00 00 00 00 00 00 00  64 eb 95 05 00 00 80 00  |........d.......|
00000dc0  01 00 0c fe ff ff 00 00  00 00 fe ff ff 01 00 00  |................|
00000dd0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000df0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 55 aa  |..............U.|
00000e00  52 52 61 41 00 00 00 00  00 00 00 00 00 00 00 00  |RRaA............|
00000e10  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000fe0  00 00 00 00 72 72 41 61  fc f7 1f 00 02 00 00 00  |....rrAa........|
00000ff0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 55 aa  |..............U.|
00001000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00002000

truenas_admin@truenas[~]$ sudo dd if=/dev/sdd count=8 bs=1024 2>/dev/null | hexdump -C
00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
000001c0  02 00 ee ff ff ff 01 00  00 00 ff ff ff 01 00 00  |................|
000001d0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
000001f0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 55 aa  |..............U.|
00000200  45 46 49 20 50 41 52 54  00 00 01 00 5c 00 00 00  |EFI PART....\...|
00000210  29 6b d2 71 00 00 00 00  01 00 00 00 00 00 00 00  |)k.q............|
00000220  ff ff ff 01 00 00 00 00  22 00 00 00 00 00 00 00  |........".......|
00000230  de ff ff 01 00 00 00 00  7b 80 10 31 85 b0 3f 49  |........{..1..?I|
00000240  b3 3d 4f b9 2b 26 b1 98  02 00 00 00 00 00 00 00  |.=O.+&..........|
00000250  80 00 00 00 80 00 00 00  68 4e 31 5e 00 00 00 00  |........hN1^....|
00000260  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000400  c3 8c 89 6a d2 1d b2 11  99 a6 08 00 20 73 66 31  |...j........ sf1|
00000410  99 b6 9c ae 06 44 c6 47  84 b6 55 81 2c 4b bc 50  |.....D.G..U.,K.P|
00000420  00 08 00 00 00 00 00 00  ff f7 ff 01 00 00 00 00  |................|
00000430  00 00 00 00 00 00 00 00  64 00 61 00 74 00 61 00  |........d.a.t.a.|
00000440  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00002000

SmallBarky · November 27, 2024, 7:23pm

Better that than EBCDIC

Does it look any different if you just WIPE an old pool disk for destruction in the GUI?

HoneyBadger · November 27, 2024, 7:32pm

Nope, exported sdc without destroying data and then wiped it through the Storage → Disks → Wipe webUI.

truenas_admin@truenas[~]$ sudo dd if=/dev/sdc count=8 bs=1024 2>/dev/null | hexdump -C
00000000  30 30 30 30 30 30 30 30  30 30 30 30 30 30 30 30  |0000000000000000|
*
00002000

bacon · November 27, 2024, 7:42pm

In your output it seems you formatted the entire drive /dev/sdd instead of the first partition /dev/sdd1. The beginning of the drive (~1MB) will be overwritten when the GPT partition table is created, which is what you’re seeing in your output.

In order to do it properly, format the disk as follows:

$ sudo parted /dev/sdd
GNU Parted 3.5
Using /dev/sdd
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) mklabel gpt
Warning: The existing disk label on /dev/sdd will be destroyed and all data on this disk will be lost. Do you want to continue?
Yes/No? yes
(parted) mkpart fat32 1048576B 100%
(parted) q
Information: You may need to update /etc/fstab.

Then make sure to format the first partition, not the entire drive:

sudo mkfs.vfat /dev/sdd1

HoneyBadger · November 27, 2024, 7:53pm

Okay, now I’m seeing that behavior, my apologies. Let me get back with our team and see what’s what.

HoneyBadger · December 2, 2024, 3:11pm

@bacon Successfully repro’d, bug squashed.