24.10.2 setup breaks after app update

TrueNAS General
,
1

I have a severe problem: I had 24.10.2 running stable for weeks now.
Friday I noticed new app versions were available and I performed an “update all”.
On Saturday I noticed the the TrueNas GUI was inaccessible, so I tried a reboot.
But the boot hangs forever at:
(1 of 2) Job ix-zfs.service/start running (31s / 15min 25s).

Booting TrueNas debug does not help: it hangs at the same point.

So I decided to rebuild: reinstalled from my 24.04.02 USB stick, upgraded to 24.10.2 and reloaded my 24.10.1 configuration. The system reboots resulting in the same permanent hang at “Job ix-zfs.service/start running”.

I don’t know hoe to proceed: help much appreciated!

2

Reinstall 24.10.1 instead and restore your config and see if it imports the pools.

1 Like
3

Downloaded 24.10.1 TrueNas Scale iso, burned it to USB and booted it.
Loaded 24.10.1 configuration (and rebooted).
Boot hangs again on "Job ix-zfs.service/start running”

Please help: I am out of clues, what can I do get the system running again or at least regain access to my data?

4

Sorry - I am out of expertise and ideas.

My guess is that ZFS is neither importing the pool nor failing to import the pool, but is instead hanging when trying to import the pool.

This could be indicative of a hardware issue with one of the drives whereby the drive I/O never completes, or possibly something else entirely. And without it booting to the point that you can run diagnostic tools, I am stumped to know what to suggest. Sorry.

@honeybadger?

5

This looks like there’s an issue with the pool, or a drive that’s a member of it.

I’d suggest that if you’re able to boot without a hang on a fresh install, do that, don’t import your configuration yet, and we’ll query your drives with smartctl and other tools to see what’s what.

The ODROID H3 is fairly lightweight hardware, but I believe the onboard SATA controller is still an Intel-based one from the N-series chips, and shouldn’t be subject to the oddities that can occasionally crop up with other models.

6

Thank you for your offer, I just performed a fresh 24.10.1 install.

The disk devices present in /dev are sda, sda1, sdb, sdb1, sdb2, sdb3 and nvme0n1.
Below my analysis and attempted steps to resolve.
Result:

==> zpool import: reports error “ZFS label hostid mismatch”
Description: “The pool has been written to from another host” is highly unlikely…

==> sgdisk -p /dev/sdb2: reports error “Found invalid GPT and valid MBR”)
Can I restore the GPT from the valid MBR without data loss with this command?
sgdisk /dev/sdb2 -g

@HoneyBadger: Please advise on this idea/command?
Can I retry the ZFS import from that point?

Analysis:

root@truenas[~]# smartctl -i /dev/sda
smartctl 7.4 2023-08-01 r5530 [x86_64-linux-6.6.44-production+truenas] (local build)
Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Device Model: WDC WD60EFPX-68C5ZN0
Serial Number: WD-WX72D645FD95
LU WWN Device Id: 5 0014ee 26bc31003
Firmware Version: 81.00A81
User Capacity: 6,001,175,126,016 bytes [6.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 5400 rpm
Form Factor: 3.5 inches
Device is: Not in smartctl database 7.3/5528
ATA Version is: ACS-3 T13/2161-D revision 5
SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Thu Mar 13 21:57:45 2025 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

root@truenas[~]# smartctl -i /dev/sdb
smartctl 7.4 2023-08-01 r5530 [x86_64-linux-6.6.44-production+truenas] (local build)
Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Device Model: WDC WD60EFPX-68C5ZN0
Serial Number: WD-WX62D64HV8DJ
LU WWN Device Id: 5 0014ee 2c118fdeb
Firmware Version: 81.00A81
User Capacity: 6,001,175,126,016 bytes [6.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 5400 rpm
Form Factor: 3.5 inches
Device is: Not in smartctl database 7.3/5528
ATA Version is: ACS-3 T13/2161-D revision 5
SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Thu Mar 13 21:58:00 2025 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

root@truenas[~]#

root@truenas[~]# zpool list -v
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
boot-pool 5.45T 4.10G 5.45T - - 0% 0% 1.00x ONLINE -
sdb3 5.46T 4.10G 5.45T - - 0% 0.07% - ONLINE

root@truenas[~]# lsblk -o PATH,PARTUUID
PATH PARTUUID
/dev/sda
/dev/sda1 f80ccafc-dfae-45f6-94b2-5586fbcf0b77
/dev/sdb
/dev/sdb1 08667c64-fb76-40e3-9b75-0419bbea0439
/dev/sdb2 54588ad2-65c4-413d-a0d5-b301d800d49a
/dev/sdb3 fbc0c771-bbda-40ed-9d80-d578b0982213
/dev/nvme0n1

root@truenas[~]# sgdisk -p /dev/sda
Disk /dev/sda: 11721045168 sectors, 5.5 TiB
Model: WDC WD60EFPX-68C
Sector size (logical/physical): 512/4096 bytes
Disk identifier (GUID): 7DEC1114-7270-464C-B215-C4F7412E329F
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 11721045134
Partitions will be aligned on 2048-sector boundaries
Total free space is 5228 sectors (2.6 MiB)

Number Start (sector) End (sector) Size Code Name
1 4096 11721043968 5.5 TiB BF01

root@truenas[~]# sgdisk -p /dev/sda1
Creating new GPT entries in memory.
Disk /dev/sda1: 11721039873 sectors, 5.5 TiB
Sector size (logical/physical): 512/4096 bytes
Disk identifier (GUID): F734817C-C818-4104-B25A-B193C9A78A8D
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 11721039839
Partitions will be aligned on 2048-sector boundaries
Total free space is 11721039806 sectors (5.5 TiB)

Number Start (sector) End (sector) Size Code Name

root@truenas[~]# sgdisk -p /dev/sdb
Disk /dev/sdb: 11721045168 sectors, 5.5 TiB
Model: WDC WD60EFPX-68C
Sector size (logical/physical): 512/4096 bytes
Disk identifier (GUID): A827F6E0-36D1-4850-BF92-FE3DB16C249A
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 11721045134
Partitions will be aligned on 2048-sector boundaries
Total free space is 4062 sectors (2.0 MiB)

Number Start (sector) End (sector) Size Code Name
1 4096 6143 1024.0 KiB EF02
2 6144 1054719 512.0 MiB EF00
3 1054720 11721045134 5.5 TiB BF01

root@truenas[~]# sgdisk -p /dev/sdb1
Creating new GPT entries in memory.
Disk /dev/sdb1: 2048 sectors, 1024.0 KiB
Sector size (logical/physical): 512/4096 bytes
Disk identifier (GUID): 6C8881DB-5A9A-49B7-B315-0276022C54B7
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 2014
Partitions will be aligned on 2048-sector boundaries
Total free space is 1981 sectors (990.5 KiB)

Number Start (sector) End (sector) Size Code Name

root@truenas[~]# sgdisk -p /dev/sdb2

Found invalid GPT and valid MBR; converting MBR to GPT format
in memory.

Disk /dev/sdb2: 1048576 sectors, 512.0 MiB
Sector size (logical/physical): 512/4096 bytes
Disk identifier (GUID): 364DD23E-969B-4568-9E8F-4E3B391310AD
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 1048542
Partitions will be aligned on 2048-sector boundaries
Total free space is 1048509 sectors (512.0 MiB)

Number Start (sector) End (sector) Size Code Name

root@truenas[~]# sgdisk -p /dev/sdb3
Creating new GPT entries in memory.
Disk /dev/sdb3: 11719990415 sectors, 5.5 TiB
Sector size (logical/physical): 512/4096 bytes
Disk identifier (GUID): F01AA165-79EB-4D3C-B9F0-DB49A7D8CCC3
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 11719990381
Partitions will be aligned on 2048-sector boundaries
Total free space is 11719990348 sectors (5.5 TiB)

Number Start (sector) End (sector) Size Code Name

root@truenas[~]# sgdisk -p /dev/nvme0n1
Creating new GPT entries in memory.
Disk /dev/nvme0n1: 1953525168 sectors, 931.5 GiB
Model: Samsung SSD 980 1TB
Sector size (logical/physical): 512/512 bytes
Disk identifier (GUID): 7FDBB0D0-1EF1-47A0-9F9C-CC8A5165B1EA
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 1953525134
Partitions will be aligned on 2048-sector boundaries
Total free space is 1953525101 sectors (931.5 GiB)

Number Start (sector) End (sector) Size Code Name

Try to resolve:

root@truenas[~]# zpool import
pool: tank
id: 15582745427217404183
state: DEGRADED
status: The pool was last accessed by another system.
action: The pool can be imported despite missing or damaged devices. The
fault tolerance of the pool may be compromised if imported.
see: Message ID: ZFS-8000-EY — OpenZFS documentation
config:

    tank                                      DEGRADED
      mirror-0                                DEGRADED
        10880048546260789043                  UNAVAIL
        f80ccafc-dfae-45f6-94b2-5586fbcf0b77  ONLINE
      indirect-1                              ONLINE

root@truenas[~]# zpool import -f
This results in exactly the same results, nothing seems to happen.

root@truenas[~]# sgdisk -v /dev/sdb2

Found invalid GPT and valid MBR; converting MBR to GPT format
in memory.

No problems found. 1048509 free sectors (512.0 MiB) available in 1
segments, the largest of which is 1048509 (512.0 MiB) in size.