Imported pool from failed system not mounting

I have an HP DL380 Gen8 that was originally Core, then promoted to Scale. Understanding (now) that it is against recommendations to install on the SD card, it seems the SD card failed. So I tried to install on an actual drive. Thankfully I just did an update before the failure and have the backup config (with seed!), so hopefully I have everything I need. I tried…everything I could find and think of to install on an SSD, and could not get it going. So the only way I could was to a flash drive, but before you scream…I just need to get the data off. I have a new installation, on redundant SSDs, on a Gen9 server that supports UEFI boot. I uploaded the config file to the installation on the Gen8, fixed up networking, and am seemingly close. I see the 2 pools, Storage and rpool. The drives show in each pool, they show healthy, I can do scrubs, it shows how much data is used in the pool, my datasets are there…but they won’t mount. I manually created the /mnt/Storage/ folders, and now I don’t get the "Path does not exist errors when I click on those in the datasets view, but nothing is actually there. If I try to unmount or export, it fails because one of them could not be found to unmount, so the whole process fails. I have done multiple reboots.

I am just hoping to get my data off and onto the new installation, then I can move the drives to a new system later with a proper installation, UEFI, etc etc.

Running: ElectricEel-24.10.2.1

We need a lot more details than this to be able to help, so please run the following diagnostic commands and post the output here each in a separate </> box:

  • lsblk -bo NAME,LABEL,MAJ:MIN,TRAN,ROTA,ZONED,VENDOR,MODEL,SERIAL,PARTUUID,START,SIZE,PARTTYPENAME
  • sudo zpool status -v
  • sudo zpool import
  • lspci
  • sudo sas2flash -list
  • sudo sas3flash -list
  • sudo storcli show all

Also was your pool under Core encrypted? And if so was it Geli encrypted?

truenas_admin@truenas[~]$ lsblk -bo NAME,LABEL,MAJ:MIN,TRAN,ROTA,ZONED,VENDOR,MODEL,SERIAL,PARTUUID,START,SIZE,PARTTYPENAME
NAME   LABEL        MAJ:MIN TRAN ROTA ZONED VENDOR   MODEL            SERIAL                                                                                                                   PARTUUID                               START          SIZE PARTTYPENAME
sda                   8:0   sas     1 none  ATA      ST4000VN008-2DR1 ZGY4JN8X                                                                                                                                                              4000787030016 
├─sda1 hestia:swap1   8:1           1 none                                                                                                                                                     9f0b9cfe-ab4a-11ed-bbea-ecb1d7ab5b50     128    2147483648 FreeBSD swap
└─sda2 Storage        8:2           1 none                                                                                                                                                     a0163b9c-ab4a-11ed-bbea-ecb1d7ab5b50 4194432 3998639460352 FreeBSD ZFS
sdb                   8:16  sas     1 none  ATA      ST4000VN008-2DR1 ZDHAFBKC                                                                                                                                                              4000787030016 
├─sdb1 hestia:swap3   8:17          1 none                                                                                                                                                     9ee765b1-ab4a-11ed-bbea-ecb1d7ab5b50     128    2147483648 FreeBSD swap
└─sdb2 Storage        8:18          1 none                                                                                                                                                     9fe58229-ab4a-11ed-bbea-ecb1d7ab5b50 4194432 3998639460352 FreeBSD ZFS
sdc                   8:32  sas     1 none  ATA      ST4000VN008-2DR1 ZDHAF98A                                                                                                                                                              4000787030016 
├─sdc1 hestia:swap1   8:33          1 none                                                                                                                                                     9ece9d38-ab4a-11ed-bbea-ecb1d7ab5b50     128    2147483648 FreeBSD swap
└─sdc2 Storage        8:34          1 none                                                                                                                                                     9ffd405d-ab4a-11ed-bbea-ecb1d7ab5b50 4194432 3998639460352 FreeBSD ZFS
sdd                   8:48  sas     1 none  ATA      ST4000VN008-2DR1 ZGY4JM6N                                                                                                                                                              4000787030016 
├─sdd1 hestia:swap3   8:49          1 none                                                                                                                                                     9ec2b344-ab4a-11ed-bbea-ecb1d7ab5b50     128    2147483648 FreeBSD swap
└─sdd2 Storage        8:50          1 none                                                                                                                                                     a0095fde-ab4a-11ed-bbea-ecb1d7ab5b50 4194432 3998639460352 FreeBSD ZFS
sde                   8:64  sas     0 none  ATA      PNY CS1311 120GB PNY17172209870102111                                                                                                                                                   120034123776 
└─sde1 Storage        8:65          0 none                                                                                                                                                     9de7c167-ab4a-11ed-bbea-ecb1d7ab5b50     128  120034037760 FreeBSD ZFS
sdf                   8:80  sas     1 none  ATA      ST4000VN008-2DR1 ZDHAFBLW                                                                                                                                                              4000787030016 
├─sdf1 hestia:swap0   8:81          1 none                                                                                                                                                     9ef28e38-ab4a-11ed-bbea-ecb1d7ab5b50     128    2147483648 FreeBSD swap
└─sdf2 Storage        8:82          1 none                                                                                                                                                     9ff0b33d-ab4a-11ed-bbea-ecb1d7ab5b50 4194432 3998639460352 FreeBSD ZFS
sdg                   8:96  sas     1 none  ATA      ST4000VN008-2DR1 ZGY4JLS3                                                                                                                                                              4000787030016 
├─sdg1 hestia:swap2   8:97          1 none                                                                                                                                                     9df11cf4-ab4a-11ed-bbea-ecb1d7ab5b50     128    2147483648 FreeBSD swap
└─sdg2 Storage        8:98          1 none                                                                                                                                                     9ea1e25c-ab4a-11ed-bbea-ecb1d7ab5b50 4194432 3998639460352 FreeBSD ZFS
sdh                   8:112 sas     1 none  ATA      ST4000VN008-2DR1 ZDHAFAFK                                                                                                                                                              4000787030016 
├─sdh1 hestia:swap2   8:113         1 none                                                                                                                                                     9eff1c97-ab4a-11ed-bbea-ecb1d7ab5b50     128    2147483648 FreeBSD swap
└─sdh2 Storage        8:114         1 none                                                                                                                                                     a02125e2-ab4a-11ed-bbea-ecb1d7ab5b50 4194432 3998639460352 FreeBSD ZFS
sdi                   8:128 sas     0 none  ATA      Samsung SSD 850  S21TNSAG125762B                                                                                                                                                        120034123776 
├─sdi1                8:129         0 none                                                                                                                                                     a22c5e20-c313-49f4-8962-6b6fbc0cb3ad      34       1031168 BIOS boot
├─sdi2                8:130         0 none                                                                                                                                                     3f9516b8-74cd-4df4-b012-7ba4a323d277    2048    1073741824 EFI System
└─sdi3 rpool          8:131         0 none                                                                                                                                                     9ca3f769-a1ba-41d4-80f4-21ac5eb95849 2099200  118110552576 Solaris /usr & Apple ZFS
sdj                   8:144 sas     1 none  ATA      ST4000VN008-2DR1 ZGY4JNLK                                                                                                                                                              4000787030016 
├─sdj1 hestia:swap0   8:145         1 none                                                                                                                                                     9edbc5e2-ab4a-11ed-bbea-ecb1d7ab5b50     128    2147483648 FreeBSD swap
└─sdj2 Storage        8:146         1 none                                                                                                                                                     9fc79df9-ab4a-11ed-bbea-ecb1d7ab5b50 4194432 3998639460352 FreeBSD ZFS
sdk                   8:160 sas     0 none  ATA      Samsung SSD 850  S21TNSAG125747M                                                                                                                                                        120034123776 
├─sdk1                8:161         0 none                                                                                                                                                     f61d4722-11d7-400c-ac52-e7ea080bedf7      34       1031168 BIOS boot
├─sdk2                8:162         0 none                                                                                                                                                     683758d2-2c65-49f7-813b-8c7fac415ee0    2048    1073741824 EFI System
└─sdk3 rpool          8:163         0 none                                                                                                                                                     09acac3e-663b-48f0-a3ec-330c9765de85 2099200  118110552576 Solaris /usr & Apple ZFS
sdl                   8:176 usb     1 none   USB     SanDisk 3.2Gen1  050114b8d06c8a9fb07b39c904f31b8f31a3db06b9175519bcfa99d72fefe97d5a4400000000000000000000b98311a5ff8c0d10815581077fab7d10                                                30765219840 
├─sdl1                8:177         1 none                                                                                                                                                     26b6b2a5-2c84-11f0-8795-ecb1d7ab5b50      40        524288 BIOS boot
└─sdl2 boot-pool      8:178         1 none                                                                                                                                                     271721f3-2c84-11f0-8795-ecb1d7ab5b50    1064   30752636928 FreeBSD ZFS
zd0                 230:0           0 none                                                                                                                                                                                                  1099511660544 
truenas_admin@truenas[~]$ sudo zpool status -v
[sudo] password for truenas_admin: 
  pool: Storage
 state: ONLINE
  scan: scrub repaired 0B in 4 days 00:25:07 with 0 errors on Mon May 12 23:14:10 2025
config:

        NAME                                      STATE     READ WRITE CKSUM
        Storage                                   ONLINE       0     0     0
          raidz1-0                                ONLINE       0     0     0
            9ff0b33d-ab4a-11ed-bbea-ecb1d7ab5b50  ONLINE       0     0     0
            9ea1e25c-ab4a-11ed-bbea-ecb1d7ab5b50  ONLINE       0     0     0
            sdh2                                  ONLINE       0     0     0
            9fc79df9-ab4a-11ed-bbea-ecb1d7ab5b50  ONLINE       0     0     0
          raidz1-1                                ONLINE       0     0     0
            9ffd405d-ab4a-11ed-bbea-ecb1d7ab5b50  ONLINE       0     0     0
            sda2                                  ONLINE       0     0     0
            9fe58229-ab4a-11ed-bbea-ecb1d7ab5b50  ONLINE       0     0     0
            a0095fde-ab4a-11ed-bbea-ecb1d7ab5b50  ONLINE       0     0     0
        logs
          9de7c167-ab4a-11ed-bbea-ecb1d7ab5b50    ONLINE       0     0     0

errors: No known data errors

  pool: boot-pool
 state: ONLINE
status: One or more features are enabled on the pool despite not being
        requested by the 'compatibility' property.
action: Consider setting 'compatibility' to an appropriate value, or
        adding needed features to the relevant file in
        /etc/zfs/compatibility.d or /usr/share/zfs/compatibility.d.
config:

        NAME        STATE     READ WRITE CKSUM
        boot-pool   ONLINE       0     0     0
          sdl2      ONLINE       0     0     0

errors: No known data errors

  pool: rpool
 state: ONLINE
status: Some supported and requested features are not enabled on the pool.
        The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(7) for details.
config:

        NAME                                      STATE     READ WRITE CKSUM
        rpool                                     ONLINE       0     0     0
          mirror-0                                ONLINE       0     0     0
            9ca3f769-a1ba-41d4-80f4-21ac5eb95849  ONLINE       0     0     0
            09acac3e-663b-48f0-a3ec-330c9765de85  ONLINE       0     0     0

errors: No known data errors
truenas_admin@truenas[~]$ sudo zpool import

no pools available to import
truenas_admin@truenas[~]$ lspci
00:00.0 Host bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 DMI2 (rev 04)
00:01.0 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 1a (rev 04)
00:01.1 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 1b (rev 04)
00:02.0 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 2a (rev 04)
00:02.1 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 2b (rev 04)
00:02.2 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 2c (rev 04)
00:02.3 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 2d (rev 04)
00:03.0 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 3a (rev 04)
00:03.1 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 3b (rev 04)
00:03.2 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 3c (rev 04)
00:03.3 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 3d (rev 04)
00:04.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 0 (rev 04)
00:04.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 1 (rev 04)
00:04.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 2 (rev 04)
00:04.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 3 (rev 04)
00:04.4 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 4 (rev 04)
00:04.5 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 5 (rev 04)
00:04.6 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 6 (rev 04)
00:04.7 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 7 (rev 04)
00:05.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 VTd/Memory Map/Misc (rev 04)
00:05.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 IIO RAS (rev 04)
00:05.4 PIC: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 IOAPIC (rev 04)
00:11.0 PCI bridge: Intel Corporation C600/X79 series chipset PCI Express Virtual Root Port (rev 05)
00:1a.0 USB controller: Intel Corporation C600/X79 series chipset USB2 Enhanced Host Controller #2 (rev 05)
00:1c.0 PCI bridge: Intel Corporation C600/X79 series chipset PCI Express Root Port 1 (rev b5)
00:1c.7 PCI bridge: Intel Corporation C600/X79 series chipset PCI Express Root Port 8 (rev b5)
00:1d.0 USB controller: Intel Corporation C600/X79 series chipset USB2 Enhanced Host Controller #1 (rev 05)
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev a5)
00:1f.0 ISA bridge: Intel Corporation C600/X79 series chipset LPC Controller (rev 05)
00:1f.2 SATA controller: Intel Corporation C600/X79 series chipset 6-Port SATA AHCI Controller (rev 05)
01:00.0 System peripheral: Hewlett-Packard Company Integrated Lights-Out Standard Slave Instrumentation & System Support (rev 05)
01:00.1 VGA compatible controller: Matrox Electronics Systems Ltd. MGA G200EH
01:00.2 System peripheral: Hewlett-Packard Company Integrated Lights-Out Standard Management Processor Support and Messaging (rev 05)
01:00.4 USB controller: Hewlett-Packard Company Integrated Lights-Out Standard Virtual USB Controller (rev 02)
02:00.0 RAID bus controller: Hewlett-Packard Company Smart Array Gen8 Controllers (rev 01)
03:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM57810 10 Gigabit Ethernet (rev 10)
03:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM57810 10 Gigabit Ethernet (rev 10)
1f:08.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 QPI Link 0 (rev 04)
1f:08.2 Performance counters: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 QPI Link 0 (rev 04)
1f:08.6 System peripheral: Intel Corporation Device 0e86 (rev 04)
1f:09.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 QPI Link 1 (rev 04)
1f:09.2 Performance counters: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 QPI Link 1 (rev 04)
1f:09.6 System peripheral: Intel Corporation Device 0e96 (rev 04)
1f:0a.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Power Control Unit 0 (rev 04)
1f:0a.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Power Control Unit 1 (rev 04)
1f:0a.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Power Control Unit 2 (rev 04)
1f:0a.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Power Control Unit 3 (rev 04)
1f:0b.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 UBOX Registers (rev 04)
1f:0b.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 UBOX Registers (rev 04)
1f:0c.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04)
1f:0c.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04)
1f:0c.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04)
1f:0c.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04)
1f:0c.4 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04)
1f:0d.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04)
1f:0d.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04)
1f:0d.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04)
1f:0d.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04)
1f:0d.4 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04)
1f:0e.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Home Agent 0 (rev 04)
1f:0e.1 Performance counters: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Home Agent 0 (rev 04)
1f:0f.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Target Address/Thermal Registers (rev 04)
1f:0f.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 RAS Registers (rev 04)
1f:0f.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder Registers (rev 04)
1f:0f.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder Registers (rev 04)
1f:0f.4 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder Registers (rev 04)
1f:0f.5 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder Registers (rev 04)
1f:10.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 Thermal Control 0 (rev 04)
1f:10.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 Thermal Control 1 (rev 04)
1f:10.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 ERROR Registers 0 (rev 04)
1f:10.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 ERROR Registers 1 (rev 04)
1f:10.4 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 Thermal Control 2 (rev 04)
1f:10.5 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 Thermal Control 3 (rev 04)
1f:10.6 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 ERROR Registers 2 (rev 04)
1f:10.7 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 ERROR Registers 3 (rev 04)
1f:13.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 R2PCIe (rev 04)
1f:13.1 Performance counters: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 R2PCIe (rev 04)
1f:13.4 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 QPI Ring Registers (rev 04)
1f:13.5 Performance counters: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 QPI Ring Performance Ring Monitoring (rev 04)
1f:16.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 System Address Decoder (rev 04)
1f:16.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Broadcast Registers (rev 04)
1f:16.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Broadcast Registers (rev 04)
20:00.0 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port in DMI2 Mode (rev 04)
20:01.0 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 1a (rev 04)
20:01.1 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 1b (rev 04)
20:02.0 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 2a (rev 04)
20:02.1 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 2b (rev 04)
20:02.2 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 2c (rev 04)
20:02.3 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 2d (rev 04)
20:03.0 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 3a (rev 04)
20:03.1 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 3b (rev 04)
20:03.2 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 3c (rev 04)
20:03.3 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 3d (rev 04)
20:04.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 0 (rev 04)
20:04.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 1 (rev 04)
20:04.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 2 (rev 04)
20:04.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 3 (rev 04)
20:04.4 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 4 (rev 04)
20:04.5 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 5 (rev 04)
20:04.6 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 6 (rev 04)
20:04.7 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 7 (rev 04)
20:05.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 VTd/Memory Map/Misc (rev 04)
20:05.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 IIO RAS (rev 04)
20:05.4 PIC: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 IOAPIC (rev 04)
3f:08.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 QPI Link 0 (rev 04)
3f:08.2 Performance counters: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 QPI Link 0 (rev 04)
3f:08.6 System peripheral: Intel Corporation Device 0e86 (rev 04)
3f:09.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 QPI Link 1 (rev 04)
3f:09.2 Performance counters: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 QPI Link 1 (rev 04)
3f:09.6 System peripheral: Intel Corporation Device 0e96 (rev 04)
3f:0a.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Power Control Unit 0 (rev 04)
3f:0a.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Power Control Unit 1 (rev 04)
3f:0a.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Power Control Unit 2 (rev 04)
3f:0a.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Power Control Unit 3 (rev 04)
3f:0b.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 UBOX Registers (rev 04)
3f:0b.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 UBOX Registers (rev 04)
3f:0c.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04)
3f:0c.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04)
3f:0c.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04)
3f:0c.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04)
3f:0c.4 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04)
3f:0d.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04)
3f:0d.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04)
3f:0d.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04)
3f:0d.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04)
3f:0d.4 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04)
3f:0e.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Home Agent 0 (rev 04)
3f:0e.1 Performance counters: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Home Agent 0 (rev 04)
3f:0f.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Target Address/Thermal Registers (rev 04)
3f:0f.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 RAS Registers (rev 04)
3f:0f.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder Registers (rev 04)
3f:0f.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder Registers (rev 04)
3f:0f.4 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder Registers (rev 04)
3f:0f.5 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder Registers (rev 04)
3f:10.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 Thermal Control 0 (rev 04)
3f:10.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 Thermal Control 1 (rev 04)
3f:10.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 ERROR Registers 0 (rev 04)
3f:10.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 ERROR Registers 1 (rev 04)
3f:10.4 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 Thermal Control 2 (rev 04)
3f:10.5 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 Thermal Control 3 (rev 04)
3f:10.6 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 ERROR Registers 2 (rev 04)
3f:10.7 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 ERROR Registers 3 (rev 04)
3f:13.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 R2PCIe (rev 04)
3f:13.1 Performance counters: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 R2PCIe (rev 04)
3f:13.4 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 QPI Ring Registers (rev 04)
3f:13.5 Performance counters: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 QPI Ring Performance Ring Monitoring (rev 04)
3f:16.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 System Address Decoder (rev 04)
3f:16.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Broadcast Registers (rev 04)
3f:16.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Broadcast Registers (rev 04)
truenas_admin@truenas[~]$ sudo sas2flash -list
LSI Corporation SAS2 Flash Utility
Version 20.00.00.00 (2014.09.18) 
Copyright (c) 2008-2014 LSI Corporation. All rights reserved 

        No LSI SAS adapters found! Limited Command Set Available!
        ERROR: Command Not allowed without an adapter!
        ERROR: Couldn't Create Command -list
        Exiting Program.
truenas_admin@truenas[~]$ sudo sas3flash -list
Avago Technologies SAS3 Flash Utility
Version 16.00.00.00 (2017.05.02) 
Copyright 2008-2017 Avago Technologies. All rights reserved.

        No Avago SAS adapters found! Limited Command Set Available!
        ERROR: Command Not allowed without an adapter!
        ERROR: Couldn't Create Command -list
        Exiting Program.
truenas_admin@truenas[~]$ sudo storcli show all
CLI Version = 007.2807.0000.0000 Dec 22, 2023
Operating system = Linux 6.6.44-production+truenas
Status Code = 0
Status = Success
Description = None

Number of Controllers = 0
Host Name = truenas
Operating System  = Linux 6.6.44-production+truenas

Encrypted under Core…I am going to say no. But the pool was successfully working under Scale for quite some time until it just wouldn’t boot anymore (SD card failure from the looks of it). How can I tell if it was encrypted? If you would have to decrypt it each time you boot up, I will say I don’t recall having to do that. Otherwise if the key was what you can save in the backup when you do an update, I have done that.

Sorry - a couple more commands:

  • sudo zfs list
  • sudo ls -al /mnt

And which pool isn’t accessible?

Theories: Could be mounts incorrect (which is what we are going to look at next), could be permissions (because you lost all your special users and groups).

1 Like
truenas_admin@truenas[~]$ sudo zfs list
[sudo] password for truenas_admin: 
NAME                                                                 USED  AVAIL  REFER  MOUNTPOINT
Storage                                                             2.28T  19.0T   198K  /mnt/Storage
Storage/ISOs                                                        1.94G  19.0T  1.94G  /mnt/Storage/ISOs
Storage/Proxmox Backups                                             1.10T  19.6T   581G  -
Storage/WolfPack                                                     731G  19.0T   731G  /mnt/Storage/WolfPack
Storage/iocage                                                      10.2M  19.0T  9.42M  /mnt/Storage/iocage
Storage/iocage/download                                              140K  19.0T   140K  /mnt/Storage/iocage/download
Storage/iocage/images                                                140K  19.0T   140K  /mnt/Storage/iocage/images
Storage/iocage/jails                                                 140K  19.0T   140K  /mnt/Storage/iocage/jails
Storage/iocage/log                                                   140K  19.0T   140K  /mnt/Storage/iocage/log
Storage/iocage/releases                                              140K  19.0T   140K  /mnt/Storage/iocage/releases
Storage/iocage/templates                                             140K  19.0T   140K  /mnt/Storage/iocage/templates
Storage/ix-applications                                              293G  19.0T   227K  /mnt/Storage/ix-applications
Storage/ix-applications/catalogs                                     140K  19.0T   140K  /mnt/Storage/ix-applications/catalogs
Storage/ix-applications/default_volumes                              140K  19.0T   140K  /mnt/Storage/ix-applications/default_volumes
Storage/ix-applications/k3s                                         1.17G  19.0T  1.17G  /mnt/Storage/ix-applications/k3s
Storage/ix-applications/k3s/kubelet                                  686K  19.0T   686K  legacy
Storage/ix-applications/releases                                     292G  19.0T   140K  /mnt/Storage/ix-applications/releases
Storage/ix-applications/releases/minio                               292G  19.0T   151K  /mnt/Storage/ix-applications/releases/minio
Storage/ix-applications/releases/minio/charts                        860K  19.0T   860K  /mnt/Storage/ix-applications/releases/minio/charts
Storage/ix-applications/releases/minio/volumes                       292G  19.0T   140K  /mnt/Storage/ix-applications/releases/minio/volumes
Storage/ix-applications/releases/minio/volumes/ix_volumes            292G  19.0T   151K  /mnt/Storage/ix-applications/releases/minio/volumes/ix_volumes
Storage/ix-applications/releases/minio/volumes/ix_volumes/export     292G  19.0T   207G  /mnt/Storage/ix-applications/releases/minio/volumes/ix_volumes/export
Storage/ix-applications/releases/minio/volumes/ix_volumes/pgBackup   140K  19.0T   140K  /mnt/Storage/ix-applications/releases/minio/volumes/ix_volumes/pgBackup
Storage/ix-applications/releases/minio/volumes/ix_volumes/pgData     140K  19.0T   140K  /mnt/Storage/ix-applications/releases/minio/volumes/ix_volumes/pgData
Storage/llm                                                          140K  60.0G   140K  /mnt/Storage/llm
Storage/stable-diffusion                                             114G   136G   114G  /mnt/Storage/stable-diffusion
boot-pool                                                           7.24G  20.4G    24K  none
boot-pool/.system                                                   1.30G  20.4G  1.23G  legacy
boot-pool/.system/configs-a2e19c264e06446d8804a27c3d55486f            24K  20.4G    24K  legacy
boot-pool/.system/configs-ae32c386e13840b2bf9c0083275e7941            83K  20.4G    83K  legacy
boot-pool/.system/cores                                               24K  1024M    24K  legacy
boot-pool/.system/netdata-a2e19c264e06446d8804a27c3d55486f          2.72M  20.4G  2.72M  legacy
boot-pool/.system/netdata-ae32c386e13840b2bf9c0083275e7941          63.4M  20.4G  63.4M  legacy
boot-pool/.system/nfs                                                 26K  20.4G    26K  legacy
boot-pool/.system/rrd-a2e19c264e06446d8804a27c3d55486f              2.83M  20.4G  2.83M  legacy
boot-pool/.system/samba4                                             214K  20.4G   114K  legacy
boot-pool/.system/services                                            24K  20.4G    24K  legacy
boot-pool/.system/syslog-a2e19c264e06446d8804a27c3d55486f             75K  20.4G    75K  legacy
boot-pool/.system/webui                                               24K  20.4G    24K  legacy
boot-pool/ROOT                                                      5.94G  20.4G    24K  none
boot-pool/ROOT/24.04.2.5                                            2.28G  20.4G   164M  legacy
boot-pool/ROOT/24.04.2.5/audit                                      14.5K  20.4G  32.5K  /audit
boot-pool/ROOT/24.04.2.5/conf                                       42.5K  20.4G  42.5K  /conf
boot-pool/ROOT/24.04.2.5/data                                         29K  20.4G  7.43M  /data
boot-pool/ROOT/24.04.2.5/etc                                        3.53M  20.4G  3.06M  /etc
boot-pool/ROOT/24.04.2.5/home                                          0B  20.4G    27K  /home
boot-pool/ROOT/24.04.2.5/mnt                                          24K  20.4G    24K  /mnt
boot-pool/ROOT/24.04.2.5/opt                                        72.0M  20.4G  72.0M  /opt
boot-pool/ROOT/24.04.2.5/root                                          1K  20.4G    39K  /root
boot-pool/ROOT/24.04.2.5/usr                                        1.89G  20.4G  1.89G  /usr
boot-pool/ROOT/24.04.2.5/var                                        19.7M  20.4G  18.9M  /var
boot-pool/ROOT/24.04.2.5/var/ca-certificates                          24K  20.4G    24K  /var/local/ca-certificates
boot-pool/ROOT/24.04.2.5/var/log                                     647K  20.4G  1.56M  /var/log
boot-pool/ROOT/24.10.2.1                                            2.36G  20.4G   165M  legacy
boot-pool/ROOT/24.10.2.1/audit                                      1.69M  20.4G  1.67M  /audit
boot-pool/ROOT/24.10.2.1/conf                                       6.48M  20.4G  6.48M  /conf
boot-pool/ROOT/24.10.2.1/data                                       7.55M  20.4G  7.43M  /data
boot-pool/ROOT/24.10.2.1/etc                                        3.60M  20.4G  3.20M  /etc
boot-pool/ROOT/24.10.2.1/home                                       59.5K  20.4G  36.5K  /home
boot-pool/ROOT/24.10.2.1/mnt                                          24K  20.4G    24K  /mnt
boot-pool/ROOT/24.10.2.1/opt                                          24K  20.4G    24K  /opt
boot-pool/ROOT/24.10.2.1/root                                         71K  20.4G    41K  /root
boot-pool/ROOT/24.10.2.1/usr                                        2.15G  20.4G  2.15G  /usr
boot-pool/ROOT/24.10.2.1/var                                        29.6M  20.4G  19.2M  /var
boot-pool/ROOT/24.10.2.1/var/ca-certificates                          24K  20.4G    24K  /var/local/ca-certificates
boot-pool/ROOT/24.10.2.1/var/log                                    9.97M  20.4G  3.10M  /var/log
boot-pool/ROOT/24.10.2.1/var/log/journal                            6.73M  20.4G  6.73M  /var/log/journal
boot-pool/ROOT/Initial-Install                                         1K  20.4G  1.30G  legacy
boot-pool/ROOT/default                                              1.30G  20.4G  1.30G  legacy
boot-pool/grub                                                      1.63M  20.4G  1.63M  legacy
rpool                                                               1.27G   104G   104K  /mnt/rpool
rpool/ROOT                                                          1.27G   104G    96K  /mnt/rpool/ROOT
rpool/ROOT/pve-1                                                    1.27G   104G  1.27G  /mnt
rpool/data                                                            96K   104G    96K  /mnt/rpool/data
truenas_admin@truenas[~]$ sudo ls -al /mnt
total 70
drwxr-xr-x 19 root root  25 May 12 19:43 .
drwxr-xr-x 21 root root  29 Mar 31 13:03 ..
drwxr-xr-x  6 root root   6 May 12 19:43 Storage
lrwxrwxrwx  1 root root   7 Jun 22  2023 bin -> usr/bin
drwxr-xr-x  5 root root  13 Mar 20 15:04 boot
drwxr-xr-x  4 root root  16 Jun 22  2023 dev
drwxr-xr-x 91 root root 183 Mar 20 15:05 etc
drwxr-xr-x  2 root root   2 Mar  2  2023 home
lrwxrwxrwx  1 root root   7 Jun 22  2023 lib -> usr/lib
lrwxrwxrwx  1 root root   9 Jun 22  2023 lib32 -> usr/lib32
lrwxrwxrwx  1 root root   9 Jun 22  2023 lib64 -> usr/lib64
lrwxrwxrwx  1 root root  10 Jun 22  2023 libx32 -> usr/libx32
drwxr-xr-x  2 root root   2 Jun 22  2023 media
drwxr-xr-x  2 root root   2 Mar 20 15:05 mnt
drwxr-xr-x  2 root root   2 Jun 22  2023 opt
drwxr-xr-x  2 root root   2 Mar  2  2023 proc
drwx------  3 root root   6 Mar 20 15:04 root
drwxr-xr-x  4 root root   4 May  8 22:04 rpool
drwxr-xr-x  4 root root   5 Jun 22  2023 run
lrwxrwxrwx  1 root root   8 Jun 22  2023 sbin -> usr/sbin
drwxr-xr-x  2 root root   2 Jun 22  2023 srv
drwxr-xr-x  2 root root   2 Mar  2  2023 sys
drwxrwxrwt  2 root root   2 Jun 22  2023 tmp
drwxr-xr-x 14 root root  14 Jun 22  2023 usr
drwxr-xr-x 11 root root  13 Jun 22  2023 var

/mnt/Storage, and its included folders, may only exist from being manually created before…not sure if that matters. For instance, I still get a path error on one of the datasets that I did not make a folder for.

Storage is the pool not accessible, but I don’t know about the other. I don’t think I have anything on it to care…? But I will check that out.

I did (at some point in troubleshooting) recreate my user with id 1000, but I did not pursue as it seemed like a future issue, as it was not mounting.

When you reinstalled, did you restore a copy of your System Configuration file?

Also, sudo ls -alR /mnt/Storage/.

Yes, I did do a restore.

truenas_admin@truenas[~]$ sudo ls -alR /mnt/Storage/
[sudo] password for truenas_admin: 
/mnt/Storage/:
total 19
drwxr-xr-x  6 root root  6 May 12 19:43 .
drwxr-xr-x 19 root root 25 May 12 19:43 ..
drwxr-xr-x  2 root root  2 May 12 19:43 ISOs
drwxr-xr-x  2 root root  2 May 12 19:43 WolfPack
drwxr-xr-x  2 root root  2 May 12 19:43 ix-applications
drwxr-xr-x  2 root root  2 May 12 19:43 stable-diffusion

/mnt/Storage/ISOs:
total 9
drwxr-xr-x 2 root root 2 May 12 19:43 .
drwxr-xr-x 6 root root 6 May 12 19:43 ..

/mnt/Storage/WolfPack:
total 9
drwxr-xr-x 2 root root 2 May 12 19:43 .
drwxr-xr-x 6 root root 6 May 12 19:43 ..

/mnt/Storage/ix-applications:
total 9
drwxr-xr-x 2 root root 2 May 12 19:43 .
drwxr-xr-x 6 root root 6 May 12 19:43 ..

/mnt/Storage/stable-diffusion:
total 9
drwxr-xr-x 2 root root 2 May 12 19:43 .
drwxr-xr-x 6 root root 6 May 12 19:43 ..

Is this maybe biting me in the rear? Why would two things be mounted on /mnt…?

Well…crap. That was it. I exported both, and only imported Storage, and it looks like stuff is there. That was stupid on my part…I can pull data off of this pool then tackle the other one. I imagine I should move that mountpoint to something else then export/import again so it mounts correctly…hopefully.

Thank you. I should have checked the mountpoints a little more.

You should be able to fix mount points without having to remove your data

That looks suspiciously like a proxmox-created ZFS pool…

https://pve.proxmox.com/wiki/ZFS_on_Linux

This explains why your /mnt folder has so many subfolders. What are you trying to do here?

1 Like

Indeed, there’s no mention of Proxmox or there being a VM on the server.
Possibly a replication target from a Proxmox host?

There was a VM, but it was just a Proxmox Backup server that I think I gave 1TB to, as a dataset under /storage I believe. I don’t care about restoring any of that though as I have other things in place. I claim the only drive I tried to do a PVE install on I pulled so it didn’t boot to that. In my attempts to get TrueNAS running again I tried installing onto an SSD with no luck, then installed PVE and was going to do TrueNAS as a VM passing the controller through, but that did not work either. I ended up installing onto a USB drive, and that worked. So I am not entirely sure what that is…

Somewhere in that install, swap drive, and VM dance you appear to have imported the Proxmox rpool that was leftover on one/some of the drives, possibly using the UI or using zpool import -R /mnt rpool in a shell.

I definitely did add via the UI, yes. Either I used it for the VM disks/data, which I no longer care about. Or it was a previous stripe array that was a network share for Steam games I think…either way, I don’t think I am too worried about it as it isn’t something that I need to worry about recovering. Just happy to have the Storage pool back, and a new machine with proper install drives (plural, thankfully), to start using instead.

It took me a minute to understand what you meant…I see what you mean now. I think I was too focused on trying to get that one pool, that I was ignoring everything else. Yep…data in /mnt is totally a linux install structure, ain’t it. Guess I will dig through that and see what exactly it was. Thanks everyone for the help!