24.10.2 setup breaks after app update

TrueNAS General
,
1

I have a severe problem: I had 24.10.2 running stable for weeks now.
Friday I noticed new app versions were available and I performed an “update all”.
On Saturday I noticed the the TrueNas GUI was inaccessible, so I tried a reboot.
But the boot hangs forever at:
(1 of 2) Job ix-zfs.service/start running (31s / 15min 25s).

Booting TrueNas debug does not help: it hangs at the same point.

So I decided to rebuild: reinstalled from my 24.04.02 USB stick, upgraded to 24.10.2 and reloaded my 24.10.1 configuration. The system reboots resulting in the same permanent hang at “Job ix-zfs.service/start running”.

I don’t know hoe to proceed: help much appreciated!

2

Reinstall 24.10.1 instead and restore your config and see if it imports the pools.

1 Like
3

Downloaded 24.10.1 TrueNas Scale iso, burned it to USB and booted it.
Loaded 24.10.1 configuration (and rebooted).
Boot hangs again on "Job ix-zfs.service/start running”

Please help: I am out of clues, what can I do get the system running again or at least regain access to my data?

4

Sorry - I am out of expertise and ideas.

My guess is that ZFS is neither importing the pool nor failing to import the pool, but is instead hanging when trying to import the pool.

This could be indicative of a hardware issue with one of the drives whereby the drive I/O never completes, or possibly something else entirely. And without it booting to the point that you can run diagnostic tools, I am stumped to know what to suggest. Sorry.

@honeybadger?

5

This looks like there’s an issue with the pool, or a drive that’s a member of it.

I’d suggest that if you’re able to boot without a hang on a fresh install, do that, don’t import your configuration yet, and we’ll query your drives with smartctl and other tools to see what’s what.

The ODROID H3 is fairly lightweight hardware, but I believe the onboard SATA controller is still an Intel-based one from the N-series chips, and shouldn’t be subject to the oddities that can occasionally crop up with other models.

6

Thank you for your offer, I just performed a fresh 24.10.1 install.

The disk devices present in /dev are sda, sda1, sdb, sdb1, sdb2, sdb3 and nvme0n1.
Below my analysis and attempted steps to resolve.
Result:

==> zpool import: reports error “ZFS label hostid mismatch”
Description: “The pool has been written to from another host” is highly unlikely…

==> sgdisk -p /dev/sdb2: reports error “Found invalid GPT and valid MBR”)
Can I restore the GPT from the valid MBR without data loss with this command?
sgdisk /dev/sdb2 -g

@HoneyBadger: Please advise on this idea/command?
Can I retry the ZFS import from that point?

Analysis:

root@truenas[~]# smartctl -i /dev/sda
smartctl 7.4 2023-08-01 r5530 [x86_64-linux-6.6.44-production+truenas] (local build)
Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Device Model: WDC WD60EFPX-68C5ZN0
Serial Number: WD-WX72D645FD95
LU WWN Device Id: 5 0014ee 26bc31003
Firmware Version: 81.00A81
User Capacity: 6,001,175,126,016 bytes [6.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 5400 rpm
Form Factor: 3.5 inches
Device is: Not in smartctl database 7.3/5528
ATA Version is: ACS-3 T13/2161-D revision 5
SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Thu Mar 13 21:57:45 2025 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

root@truenas[~]# smartctl -i /dev/sdb
smartctl 7.4 2023-08-01 r5530 [x86_64-linux-6.6.44-production+truenas] (local build)
Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Device Model: WDC WD60EFPX-68C5ZN0
Serial Number: WD-WX62D64HV8DJ
LU WWN Device Id: 5 0014ee 2c118fdeb
Firmware Version: 81.00A81
User Capacity: 6,001,175,126,016 bytes [6.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 5400 rpm
Form Factor: 3.5 inches
Device is: Not in smartctl database 7.3/5528
ATA Version is: ACS-3 T13/2161-D revision 5
SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Thu Mar 13 21:58:00 2025 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

root@truenas[~]#

root@truenas[~]# zpool list -v
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
boot-pool 5.45T 4.10G 5.45T - - 0% 0% 1.00x ONLINE -
sdb3 5.46T 4.10G 5.45T - - 0% 0.07% - ONLINE

root@truenas[~]# lsblk -o PATH,PARTUUID
PATH PARTUUID
/dev/sda
/dev/sda1 f80ccafc-dfae-45f6-94b2-5586fbcf0b77
/dev/sdb
/dev/sdb1 08667c64-fb76-40e3-9b75-0419bbea0439
/dev/sdb2 54588ad2-65c4-413d-a0d5-b301d800d49a
/dev/sdb3 fbc0c771-bbda-40ed-9d80-d578b0982213
/dev/nvme0n1

root@truenas[~]# sgdisk -p /dev/sda
Disk /dev/sda: 11721045168 sectors, 5.5 TiB
Model: WDC WD60EFPX-68C
Sector size (logical/physical): 512/4096 bytes
Disk identifier (GUID): 7DEC1114-7270-464C-B215-C4F7412E329F
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 11721045134
Partitions will be aligned on 2048-sector boundaries
Total free space is 5228 sectors (2.6 MiB)

Number Start (sector) End (sector) Size Code Name
1 4096 11721043968 5.5 TiB BF01

root@truenas[~]# sgdisk -p /dev/sda1
Creating new GPT entries in memory.
Disk /dev/sda1: 11721039873 sectors, 5.5 TiB
Sector size (logical/physical): 512/4096 bytes
Disk identifier (GUID): F734817C-C818-4104-B25A-B193C9A78A8D
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 11721039839
Partitions will be aligned on 2048-sector boundaries
Total free space is 11721039806 sectors (5.5 TiB)

Number Start (sector) End (sector) Size Code Name

root@truenas[~]# sgdisk -p /dev/sdb
Disk /dev/sdb: 11721045168 sectors, 5.5 TiB
Model: WDC WD60EFPX-68C
Sector size (logical/physical): 512/4096 bytes
Disk identifier (GUID): A827F6E0-36D1-4850-BF92-FE3DB16C249A
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 11721045134
Partitions will be aligned on 2048-sector boundaries
Total free space is 4062 sectors (2.0 MiB)

Number Start (sector) End (sector) Size Code Name
1 4096 6143 1024.0 KiB EF02
2 6144 1054719 512.0 MiB EF00
3 1054720 11721045134 5.5 TiB BF01

root@truenas[~]# sgdisk -p /dev/sdb1
Creating new GPT entries in memory.
Disk /dev/sdb1: 2048 sectors, 1024.0 KiB
Sector size (logical/physical): 512/4096 bytes
Disk identifier (GUID): 6C8881DB-5A9A-49B7-B315-0276022C54B7
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 2014
Partitions will be aligned on 2048-sector boundaries
Total free space is 1981 sectors (990.5 KiB)

Number Start (sector) End (sector) Size Code Name

root@truenas[~]# sgdisk -p /dev/sdb2

Found invalid GPT and valid MBR; converting MBR to GPT format
in memory.

Disk /dev/sdb2: 1048576 sectors, 512.0 MiB
Sector size (logical/physical): 512/4096 bytes
Disk identifier (GUID): 364DD23E-969B-4568-9E8F-4E3B391310AD
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 1048542
Partitions will be aligned on 2048-sector boundaries
Total free space is 1048509 sectors (512.0 MiB)

Number Start (sector) End (sector) Size Code Name

root@truenas[~]# sgdisk -p /dev/sdb3
Creating new GPT entries in memory.
Disk /dev/sdb3: 11719990415 sectors, 5.5 TiB
Sector size (logical/physical): 512/4096 bytes
Disk identifier (GUID): F01AA165-79EB-4D3C-B9F0-DB49A7D8CCC3
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 11719990381
Partitions will be aligned on 2048-sector boundaries
Total free space is 11719990348 sectors (5.5 TiB)

Number Start (sector) End (sector) Size Code Name

root@truenas[~]# sgdisk -p /dev/nvme0n1
Creating new GPT entries in memory.
Disk /dev/nvme0n1: 1953525168 sectors, 931.5 GiB
Model: Samsung SSD 980 1TB
Sector size (logical/physical): 512/512 bytes
Disk identifier (GUID): 7FDBB0D0-1EF1-47A0-9F9C-CC8A5165B1EA
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 1953525134
Partitions will be aligned on 2048-sector boundaries
Total free space is 1953525101 sectors (931.5 GiB)

Number Start (sector) End (sector) Size Code Name

Try to resolve:

root@truenas[~]# zpool import
pool: tank
id: 15582745427217404183
state: DEGRADED
status: The pool was last accessed by another system.
action: The pool can be imported despite missing or damaged devices. The
fault tolerance of the pool may be compromised if imported.
see: Message ID: ZFS-8000-EY — OpenZFS documentation
config:

    tank                                      DEGRADED
      mirror-0                                DEGRADED
        10880048546260789043                  UNAVAIL
        f80ccafc-dfae-45f6-94b2-5586fbcf0b77  ONLINE
      indirect-1                              ONLINE

root@truenas[~]# zpool import -f
This results in exactly the same results, nothing seems to happen.

root@truenas[~]# sgdisk -v /dev/sdb2

Found invalid GPT and valid MBR; converting MBR to GPT format
in memory.

No problems found. 1048509 free sectors (512.0 MiB) available in 1
segments, the largest of which is 1048509 (512.0 MiB) in size.

7

@etorix I based much of my analysis on your previous replies in other posts.
Maybe you can advise me?
Results so far (see the top of my previous post):

==> zpool import: reports error “ZFS label hostid mismatch”
Description: “The pool has been written to from another host”
Can’t think of any other host, so this seems highly unlikely…

==> sgdisk -p /dev/sdb2: reports error “Found invalid GPT and valid MBR”)
My hunch is that this is the root cause of the import problem.
Can I restore the GPT from the valid MBR without data loss using this command?
sgdisk /dev/sdb2 -g

If that succeeds, is it safe to retry the ZFS import from that point?

8

  1. The lsblk command you ran wasn’t my standard diagnostic version but it is good enough for me to see that sdb is your boot drive and sda the remaining drive from your tank pool.

  2. It doesn’t make any sense to do e.g. sgdisk -p /dev/sda1 so any error messages here should be ignored - only sgdisk -p /dev/sda makes sense. The output from this was:

    root@truenas[~]# sgdisk -p /dev/sda
Disk /dev/sda: 11721045168 sectors, 5.5 TiB
Model: WDC WD60EFPX-68C
Sector size (logical/physical): 512/4096 bytes
Disk identifier (GUID): 7DEC1114-7270-464C-B215-C4F7412E329F
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 11721045134
Partitions will be aligned on 2048-sector boundaries
Total free space is 5228 sectors (2.6 MiB)

Number    Start (sector)    End (sector)       Size     Code Name
1                   4096     11721043968    5.5 TiB     BF01

    IIRC, BF01 is a ZFS partition. So the GPT table(s) look fine.

  3. If you "restore the MBR partition to /dev/sdb3, your copy of TrueNAS will be gone, because this is the partition that your boot-pool is in.

  4. The command you need to attempt to force import the pool is: sudo zpool import -f -R /mnt tank. If you don’t specify a pool name (or maybe -a) then zpool import doesn’t attempt to import anything.

If you get the pool online, don’t power off or reboot - but instead ask here what to do next.

9

I tried the command: zpool import -f -R /mnt tank
This caused the GUI to become unreachable and a reboot shortly afterwards. I have been searching for a log to find what happened, but did not find anything useful yet.

10

It sounds like perhaps it caused Linux to hang and then a watchdog time caused a reboot.

What was the pool situation after the reboot?

@HoneyBadger ?

11

P.S. The following command might also give us some diagnostic information (it tests to see whether an import with a roll-back might work - but it doesn’t actually import so it won’t cause another hang):

  • zpool import -F -n -R /mnt tank
12

After the reboot nothing is changed.

13

The command ‘zpool import -F -n -R /mnt tank’ runs less than a minute before the system reboots. I don’t see any output or diagnostics

Can I somehow test or check the TrueNas install on my nvm drive? I would like to exclude that drive and the install from causing this.

1 Like
14

Well zpool import -F -n causing a reboot was disappointing.

Do you have IPMI or a monitor attached to the server so you can see what happens real-time?

Would it help to turn off any watchdog timer in BIOS?

15

I currently have a monitor and keyboard attached to the server for install and troubleshooting.

What do you mean by ‘watchdog timer in BIOS’?
I am not aware of this, scrolled through the BIOS settings, but can’t find or do not know where to look…

16

A watchdog timer is hardware which watches for a flag to be reset by the O/S and reboots if it doesn’t get reset (i.e. because the O/S has crashed or hung).

I have no idea whether Odroid boards have them or not, but it would be the most obvious explanation for a spontaneous reboot.

17

I searched the BIOS documentation, Odroid H4 has a watchdog timer but there is no mention of it in H3.

18

Performed a fresh 24.10.2 install from USB on nvm01 and executed the ‘zpool import -F -n -R /mnt tank’ command. To my surprise this now executes fine and suggests to do a force import:

root@truenas[~]# zpool import -F -n -R /mnt tank
cannot import ‘tank’: pool was previously in use from another system.
Last accessed by truenas (hostid=2d05407e) at Sun Mar 16 06:49:04 2025
The pool can be imported, use ‘zpool import -f’ to import the pool.

However I don’t understand the time stamp mentioned, I have not touched the system today until after 12 o’clock but the time stamp mentions 06:49?

What do you suggest @Protopia ?

19

But you already tried it with -f (lower case) and it rebooted, and -f is different from -F (uppercase).

Despite a fresh install, I doubt it will import with -f any differently from last time, and I suspect that this is another example of ZFS telling you it will import but when you try it then it fails.

But at this point I am not sure what you have to lose, so try:

  • zpool import -f -R /mnt tank and if that fails (or it reboots)
  • zpool import -F -R /mnt tank
20

The difference is that I booted a fresh install of 24.10.2 now in stead of 24.10.1.

I tried ‘zpool import -f -R /mnt tank’
This results in a reboot after half a minute.

Logged in again and executed 'zpool import -F -R /mnt tank`
The result is a reboot after a minute.