Zpool missing after "move"

zebraartefakt · September 23, 2024, 8:03am

I just moved appartments and after trying to start my media server based on TrueNAS Scale, my main VM and its storage is missing.

What I’ve done:

Server shutdown
Changed network settings (new gateway, new bridge device IPs, new DNS server address)

When I tried to bring my VM back online, I noticed this error:
[EFAULT] VM will not start as DISK Device: /dev/zvol/VMs/debian-lzbp5 device(s) are not available.

Taking a look at the dashboard, I can see that the “VMs” Pool is offline. If I click on “Manage Devices”, I get an empty list.

I’ve set this server up quite a while ago and don’t even remember which disk this was on, but I’m sure the pool only contained one disk. Here’s my disk layout:

root@truenas[~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
sda 8:0 0 3.6T 0 disk
└─sda1 8:1 0 3.6T 0 part
sdb 8:16 0 3.6T 0 disk
└─sdb1 8:17 0 3.6T 0 part
sdc 8:32 0 3.6T 0 disk
└─sdc1 8:33 0 3.6T 0 part
nvme0n1 259:0 0 232.9G 0 disk
├─nvme0n1p1 259:1 0 1M 0 part
├─nvme0n1p2 259:2 0 512M 0 part
├─nvme0n1p3 259:3 0 216.4G 0 part
└─nvme0n1p4 259:4 0 16G 0 part
└─nvme0n1p4 253:0 0 16G 0 crypt

Output of some commands that might be useful:
root@truenas[~]# zpool status
pool: boot-pool
state: ONLINE
scan: scrub repaired 0B in 00:00:07 with 0 errors on Fri Sep 6 03:45:08 2024
config:

    NAME         STATE     READ WRITE CKSUM
    boot-pool    ONLINE       0     0     0
      nvme0n1p3  ONLINE       0     0     0

errors: No known data errors

pool: media-storage
state: ONLINE
scan: scrub repaired 0B in 03:23:01 with 0 errors on Sun Aug 18 03:23:02 2024
config:

    NAME                                      STATE     READ WRITE CKSUM
    media-storage                             ONLINE       0     0     0
      raidz1-0                                ONLINE       0     0     0
        08a9c883-6b8f-4995-a85e-6122b57f15a2  ONLINE       0     0     0
        e54a5bd7-a43b-4cf6-bb4a-9e1a7ce01462  ONLINE       0     0     0
        aeb6fe6a-427b-48dd-828b-9151de69298c  ONLINE       0     0     0

errors: No known data errors

root@truenas[~]# zpool import
no pools available to import

root@truenas[~]# zfs list
NAME boot-pool boot-pool/ROOT boot-pool/ROOT/23.10.2 boot-pool/ROOT/24.04.1.1 boot-pool/ROOT/24.04.1.1/audit boot-pool/ROOT/24.04.1.1/conf boot-pool/ROOT/24.04.1.1/data boot-pool/ROOT/24.04.1.1/etc boot-pool/ROOT/24.04.1.1/home boot-pool/ROOT/24.04.1.1/mnt boot-pool/ROOT/24.04.1.1/opt boot-pool/ROOT/24.04.1.1/root boot-pool/ROOT/24.04.1.1/usr boot-pool/ROOT/24.04.1.1/var boot-pool/ROOT/24.04.1.1/var boot-pool/ROOT/24.04.1.1/var/log boot-pool/ROOT/24.04.2 boot-pool/ROOT/24.04.2.2 boot-pool/ROOT/24.04.2.2/audit boot-pool/ROOT/24.04.2.2/conf boot-pool/ROOT/24.04.2.2/data boot-pool/ROOT/24.04.2.2/etc boot-pool/ROOT/24.04.2.2/home boot-pool/ROOT/24.04.2.2/mnt boot-pool/ROOT/24.04.2.2/opt boot-pool/ROOT/24.04.2.2/root boot-pool/ROOT/24.04.2.2/usr boot-pool/ROOT/24.04.2.2/var boot-pool/ROOT/24.04.2.2/var boot-pool/ROOT/24.04.2.2/var/log boot-pool/ROOT/24.04.2/audit boot-pool/ROOT/24.04.2/conf boot-pool/ROOT/24.04.2/data boot-pool/ROOT/24.04.2/etc boot-pool/ROOT/24.04.2/home boot-pool/ROOT/24.04.2/mnt boot-pool/ROOT/24.04.2/opt boot-pool/ROOT/24.04.2/root boot-pool/ROOT/24.04.2/usr boot-pool/ROOT/24.04.2/var boot-pool/ROOT/24.04.2/var/c boot-pool/ROOT/24.04.2/var/log boot-pool/ROOT/Initial-Install boot-pool/grub media-storage media-storage/.system media-storage/.system/config media-storage/.system/cores media-storage/.system/ctdb_shared_vol media-storage/.system/glusterd media-storage/.system/netdat media-storage/.system/rrd-ae media-storage/.system/samba4 media-storage/.system/services media-storage/.system/webui media-storage/media-nfs media-storage/windows-smb USED AVAIL REFER MOUNTPOINT
12.3G 197G 96K none
12.2G 197G 96K none
4.93G 197G 4.92G legacy
2.39G 197G 164M legacy
308K 197G 308K /audit
140K 197G 140K /conf
100K 197G 284K /data
6.71M 197G 5.66M /etc
0B 197G 132K /home
104K 197G 104K /mnt
74.1M 197G 74.1M /opt
8K 197G 164K /root
2.12G 197G 2.12G /usr
33.5M 197G 32.0M /var
/ca-certificates 96K 197G 96K /var/local/ca-certificates
576K 197G 56.8M /var/log
2.39G 197G 164M legacy
2.49G 197G 164M legacy
424K 197G 424K /audit
140K 197G 140K /conf
776K 197G 284K /data
6.63M 197G 5.66M /etc
300K 197G 132K /home
96K 197G 96K /mnt
74.1M 197G 74.1M /opt
388K 197G 164K /root
2.12G 197G 2.12G /usr
128M 197G 31.0M /var
/ca-certificates 96K 197G 96K /var/local/ca-certificates
96.5M 197G 39.5M /var/log
700K 197G 700K /audit
140K 197G 140K /conf
132K 197G 280K /data
6.65M 197G 5.65M /etc
84K 197G 132K /home
104K 197G 104K /mnt
74.1M 197G 74.1M /opt
96K 197G 164K /root
2.12G 197G 2.12G /usr
32.7M 197G 31.2M /var
a-certificates 96K 197G 96K /var/local/ca-certificates
432K 197G 37.3M /var/log
8K 197G 2.33G /
8.20M 197G 8.20M legacy
2.41T 4.73T 40.7G /mnt/media-storage
1.49G 4.73T 1.23G legacy
s-ae32c386e13840b2bf9c0083275e7941 16.7M 4.73T 16.7M legacy
128K 1024M 128K legacy
128K 4.73T 128K legacy
139K 4.73T 139K legacy
a-ae32c386e13840b2bf9c0083275e7941 248M 4.73T 248M legacy
32c386e13840b2bf9c0083275e7941 128K 4.73T 128K legacy
943K 4.73T 352K legacy
128K 4.73T 128K legacy
128K 4.73T 128K legacy
2.26T 4.73T 2.26T /mnt/media-storage/media-nfs
104G 4.73T 104G /mnt/media-storage/windows-smb

Does anyone have an idea how to get my VM data back?

EDIT: Just noticed I was smart enough to implement a periodic snapshot task for the dataset. I’ve never worked with these snapshots, but I should be able to recover the data from them, right?

Johnny_Fartpants · September 23, 2024, 8:27am

Hello and welcome to the forums.

Do you know how your VM pool was configured i.e was it a single disk, a mirror or something else?

Snapshots do not protect you from a pool loss they protect against file deletion etc.

zebraartefakt · September 23, 2024, 8:32am

I am pretty sure the pool consisted of a single disk. I believe it used to use some space on the nvme-disk (possibly nvme0n1p3?).

Johnny_Fartpants · September 23, 2024, 8:34am

Ah right so are you saying that you partitioned the nvme-disk you use for boot and used part of it to build another pool?

zebraartefakt · September 23, 2024, 8:35am

256GB was overkill for just TrueNAS so I repurposed some of that space.
Probably not my brightest moment in hindsight.

Johnny_Fartpants · September 23, 2024, 8:37am

Yeah it’s widely agreed that pools should comprise of individual devices and not to partition devices and build multiple pools from them. As you have demonstrated it’s possible but doesn’t always end well.

Johnny_Fartpants · September 23, 2024, 8:40am

Its a long shot but depending on how much you want that pool back you could try temporarily booting from another device like a USB stick and then try a zpool import while that nvme is not operating as the boot device.

zebraartefakt · September 23, 2024, 8:57am

That’s a great idea! I’ll give that a try and will report back. Thanks!