Degraded pool, how to recover

TrueNAS General
1

Hi apologies if I miss any info or am just a newb in general. I am below even a hobbiest and set this up 5+ years ago so am really foggy on the details of the initial setup.

With that out of the way I have TrueNAS installed on a home server via esxi (6.7) something recently happened that has taken out my pool despite the drives still existing. Apparently some drive was remove (not one of my core storage drives) that has degraded the pool.

In esxi there was a missing PCI device passthrough though I can’t for the life of me figure out what it was… I’m hoping I may be able to get some information out of the TrueNAS config that could help me reverse engineer it and resolve the issue, or maybe rebuild the pool without it, as I said all 6 of my core drives are still there and this is what I thought comprised the entirety of the pool.

Any tips much appreciated, thanks

root@freenas[~]# zpool import
   pool: tank
     id: 18113911827104284058
  state: DEGRADED
status: One or more devices are missing from the system.
 action: The pool can be imported despite missing or damaged devices.  The
        fault tolerance of the pool may be compromised if imported.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-2Q
 config:

        tank                                            DEGRADED
          raidz2-0                                      ONLINE
            gptid/7d91ef16-2280-11ea-8bb5-000c29164792  ONLINE
            gptid/7f1eff26-2280-11ea-8bb5-000c29164792  ONLINE
            gptid/80a224f1-2280-11ea-8bb5-000c29164792  ONLINE
            gptid/821b7417-2280-11ea-8bb5-000c29164792  ONLINE
            gptid/83a3d034-2280-11ea-8bb5-000c29164792  ONLINE
            gptid/85136c7f-2280-11ea-8bb5-000c29164792  ONLINE
2

Did you have a special metadata device on a physical NVMe drive? (perhaps the missing pci passthru)

3

Its possible, but the SSD is the drive that esxi is installed on and when I tired to pass that through as a PCI device it blew up my image entirely and had to restore esxi

root@freenas[~]# zpool status -x
  pool: freenas-boot
 state: DEGRADED
status: One or more devices has been removed by the administrator.
        Sufficient replicas exist for the pool to continue functioning in a
        degraded state.
action: Online the device using zpool online' or replace the device with
        'zpool replace'.
  scan: scrub repaired 0B in 00:00:47 with 0 errors on Tue Aug 26 03:45:47 2025
config:

        NAME            STATE    READ WRITE CKSUM
        freenas-boot  DEGRADED 0    0     0
          mirror-0        DEGRADED 0    0     0
            da0p2         ONLINE   0    0     0
            da1p2         REMOVED  0    0     0
4

You say esxi, but are you talking about truenas?

5

No, the SSD in the system that is not in the main pool storage is where esxi is installed, it runs the VM that truenas is on and when I passed that device through all the VMs became corrupt and I had to restore ESXI to reverse the passthrough… Sorry I know this part is not strictly TrueNas related.

I’ve looked through my esxi logs and I had to install some VIB that are related to sdd data center tools so perhaps I need to reinstall that to get another passthrough option. Will investigate

6

Is there any way to get info on what this device was?

7

If you’re running on ESXi, it should be a vmdk in the VM folder. Figure out which vmdk is not attached and connect back to the VM. Or you can create a new virtual disk and do a replace operation in TrueNAS to resilver the boot pool.

As for your tank pool, there don’t appear to be any missing drives in the output you posted. I’d run a scrub on it and see if it all checks out.

8

Thanks very much, ill have a look over the next few days

9

Thanks to your help so far I have my pool recovered and the files contained accessible. My one remaining issue is that a jail I had no longer connects to the network correctly. It cannot ping the default gateway (100% packet loss) but can ping TrueNAS on 192.168.50.212. I can’t see what’s wrong in the configuration. If you have any thoughts of what I can try I’d appreciate the advice.

ifconfig from jail

root@plex:~ # ifconfig -a
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pflog0: flags=0<> metric 0 mtu 33160
        groups: pflog
epair0b: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8<VLAN_MTU>
        ether 00:0c:29:14:fa:0a
        hwaddr 02:14:d2:0e:27:0b
        inet 192.168.50.213 netmask 0xffffff00 broadcast 192.168.50.255
        inet6 fe80::20c:29ff:fe14:fa0a%epair0b prefixlen 64 scopeid 0x3
        groups: epair
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>

ifconfig from TrueNAS host

root@freenas[~]# ifconfig -a
vmx0: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: vmx0
        options=4a400b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,RXCSUM_IPV6,NOMAP>
        ether 00:0c:29:05:1b:d2
        inet 192.168.50.212 netmask 0xffffff00 broadcast 192.168.50.255
        media: Ethernet autoselect
        status: active
        nd6 options=9<PERFORMNUD,IFDISABLED>
vmx1: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
        description: Storage Network
        options=4e403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6,NOMAP>
        ether 00:0c:29:05:1b:dc
        inet 10.55.1.2 netmask 0xffff0000 broadcast 10.55.255.255
        media: Ethernet autoselect
        status: active
        nd6 options=9<PERFORMNUD,IFDISABLED>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pflog0: flags=0<> metric 0 mtu 33160
        groups: pflog
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 58:9c:fc:10:ff:9c
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: vnet0.12 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 6 priority 128 path cost 2000
        member: vmx0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 1 priority 128 path cost 2000
        groups: bridge
        nd6 options=9<PERFORMNUD,IFDISABLED>
vnet0.12: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: associated with jail: plex as nic: epair0b
        options=8<VLAN_MTU>
        ether 00:0c:29:14:fa:09
        hwaddr 02:14:d2:0e:27:0a
        groups: epair
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        nd6 options=2b<PERFORMNUD,ACCEPT_RTADV,IFDISABLED,AUTO_LINKLOCAL>
10

Wait, is this still TrueNAS Core with FreeBSD?

Sorry, I never did have the jails and their networking figured out. If you were running a VM, I know the networking backwards and forwards.

11

Yea it is, I’ll persevere and see if I can find a solution. Either to this issue or just spin up a VM specifically for it. My files are safe so that’s the main thing. Many thanks