What are the best poractices for safely restoring a mirror pool in passthrough?

[juspolo]

Hi everyone,

I’m setting up a new TrueNAS installation on Proxmox and I need some advice on safely restoring a RAID 1 pool.

Here’s my situation:

• I have two drives that were part of a RAID 1 pool in a previous TrueNAS setup.
• These drives contain important data that I can’t afford to lose and were actually my backup drives. Unfortunately, I don’t have additional backups due to size constraints.
• I’ve passed one drive through to the new TrueNAS VM using Proxmox, but I haven’t imported the pool yet.

My questions:

• Should I import one drive first to check if it’s recognized correctly, or import both at once?
• Are there potential issues with drive recognition due to the passthrough setup that I should be aware of?

The drives were previously used in a passthrough setup on Proxmox for years without issues (from FreeNAS to TrueNAS Core). However, due to a potential compromise, I had to wipe the configuration. I recall being able to check drive status and SMART data in the past, but in this new setup, those details aren’t showing.

Any tips or steps to ensure a smooth and safe recovery would be greatly appreciated!

Thanks in advance for your help.

[Farout]

@juspolo

Are there potential issues with drive recognition due to the passthrough setup that I should be aware of?

Yes, see: Resource - "Absolutely must virtualize TrueNAS!" ... a guide to not completely losing your data. | TrueNAS Community

Also there seems to be an ongoing issue of Proxmox auto importing the Truenas Pool and corrupting it!

Can't import datapool

IIRC there is currently a bug / bad design decision in proxmox where it auto-imports zpools into the host OS. If the zpool is currently imported / in use in a guest as well this will cause pool corruption. ZFS is not a clustered filesystem and so it cannot be consumed by both host and guest simultaneously. IIRC the only way to avoid the data corruption is HBA passthrough. @HoneyBadger has more details.

[juspolo]

Thanks for pointing me to the guide!

The good news is that I was cautious enough to pass through only one drive. If I understand correctly, the best approach might be to run TrueNAS directly on bare metal and manage any VMs from there. Does that sound right?

Given that, could I simply install TrueNAS bare metal and re-import the pool? I’m concerned that the drive I passed through might be corrupted now, but I’m hoping the second drive, which I haven’t passed through yet, could be used to repair the first one via the RAID 1 setup. Am I thinking along the right lines?

I want to ensure I’m taking all the necessary precautions before proceeding.

Thanks again for your help!

P.s. If I got it right, TrueNAS Scale seems to be the best choice for running VMs. Am I right?

[Farout]

If you wanna use proxmox, you have to passthrough a whole controller to truenas. Either the onboard sata controller or an HBA in IT mode. For example an LSI 9300-8i.
Then you need to blacklist the HBA so proxmox will never ever see it and try to import the pool.

If you wanna run VMs on Truenas I highly suggest,you sidegrade to SCALE, as it uses KVM as the hypervisor, which is the same as Proxmox.

As of today, VM configuration on Truenas is a bit lackluster. Lots of things missing in the GUI. Some things can be done via CLI.

However, it works ( I run 2 gaming VMs on Truenas SCALE). Performance is near bare metal.

[Davvo]

You should not use hardware RAID with ZFS, since ZFS used software RAID (RAIDZ).

Suggested readings:

Introduction to ZFS

What's all the noise about HBAs, and why can't I use a RAID controller?

ZFS Storage Pool Layout

This resource was originally created by user: @Davvo on the TrueNAS Community Forums Archive. https://www.truenas.com/community/resources/zfs-storage-pool-layout.201/download This amazing document, created by iXsystems in February 2022 as a “White Paper”, cleanly explains how to qualify pool performance touching briefly on how ZFS stores data and presents the advantages, performance and disadvantages of each pool layout (striped vdev, mirrored vdev, raidz vdev). It also presents three common …

[juspolo]

Thanks for the detailed advice!

Before I proceed, I want to ensure I’m taking the safest approach to restore my pool without risking data loss. Here’s what I’m considering:

• Restoring the Potentially Corrupted Drive First: Given that one of the drives might be corrupted from the passthrough setup, I thought about starting with this one. Does this approach make sense, or should I handle the drives differently?

• TrueNAS SCALE Installation: I’m thinking of installing TrueNAS SCALE and then attempting to import the pool. However, the pool wasn’t exported from the previous install, and I’ve read only after wiping the system in the documentation that pools should be exported and that there might be issues when importing pools, especially when drives are potentially corrupted. What are the best practices to import a non-exported pool?

• Handling Potential Disk Corruption: What should I expect if the disk passed through is corrupted? Is there a safe way to import the pool that could minimize the risk of data loss, even if one of the drives is compromised?

I want to make sure I’m not missing anything critical before moving forward.

Thanks again for all the help!

[juspolo]

Thanks for the advice and links!

Yes, I’m aware that hardware RAID isn’t recommended with ZFS. Fortunately, when I set up the system, I did some research and configured the pool using software RAID in mirroring. I apologize for any confusion caused by the incorrect terminology.

Just to recap, my pool was set up with ZFS software RAID mirror, and I’m looking for the safest way to restore it given the current situation.

[juspolo]

Hi everyone!

Long story short, I’ve been trying to determine the status of a drive that was connected to TrueNAS while still visible in Proxmox, without mounting it. After doing some research, I decided to check the pool directly using Proxmox.

I ran the following commands to see if there were any issues:

zdb -e -C wolf

This sould check the configuration of the pool.

zdb -e -bcs {pool_name}

This should provide a detailed report, including usage, checkpoints, configuration, and block statuses.

Here are the results:

zdb -e -C myPool

MOS Configuration:
        version: 5000
        name: 'myPool'
        state: 0
        txg: 40502284
        pool_guid: 7282203610654777039
        errata: 0
        hostid: 113288809
        hostname: ''
        com.delphix:has_per_vdev_zaps
        vdev_children: 1
        vdev_tree:
            type: 'root'
            id: 0
            guid: 7282203610654777039
            children[0]:
                type: 'mirror'
                id: 0
                guid: 16063361651154799650
                metaslab_array: 35
                metaslab_shift: 35
                ashift: 12
                asize: 5999022833664
                is_log: 0
                create_txg: 4
                com.delphix:vdev_zap_top: 1861
                children[0]:
                    type: 'disk'
                    id: 0
                    guid: 15142413831892053089
                    path: '/dev/gptid/e7572135-fb54-11e6-ac75-00237dfcb67a'
                    whole_disk: 1
                    DTL: 2736
                    create_txg: 4
                    com.delphix:vdev_zap_leaf: 1862
                children[1]:
                    type: 'disk'
                    id: 1
                    guid: 4993797005918395778
                    path: '/dev/gptid/e85811c2-fb54-11e6-ac75-00237dfcb67a'
                    whole_disk: 1
                    DTL: 2735
                    create_txg: 4
                    com.delphix:vdev_zap_leaf: 1863
        features_for_read:
            com.delphix:hole_birth
            com.delphix:embedded_data

zdb -e -bcs myPool
Traversing all blocks to verify metadata checksums and verify nothing leaked ...
loading concrete vdev 0, metaslab 173 of 174 ...
4.01T completed ( 727MB/s) estimated time remaining: 0hr 00min 00sec          
        No leaks (block sum matches space maps exactly)

        bp count:              37821583
        ganged count:                 0
        bp logical:       4730154636800      avg: 125064
        bp physical:      4404997525504      avg: 116467     compression:   1.07
        bp allocated:     4410484064256      avg: 116612     compression:   1.07
        bp deduped:                   0    ref>1:      0   deduplication:   1.00
        bp cloned:                    0    count:      0
        Normal class:     4410483310592     used: 74.20%
        Embedded log class         684032     used:  0.00%

        additional, non-pointer bps of type 0:     858449
        Dittoed blocks on same vdev: 707345

                            capacity   operations   bandwidth  ---- errors ----
description                used avail  read write  read write  read write cksum
myPool                    4.01T 1.43T   440     0 3.52M     0     0     0     0
  mirror                  4.01T 1.43T   440     0 3.52M     0     0     0     0
    /dev/sdb2                           219     0 1.76M     0     0     0     0
    /dev/sda2                           221     0 1.76M     0     0     0     0

From what I understand, it seems like both drives are okay and not corrupted, so I should be able to proceed with the import in truenas as previously suggested. Am I right?

Lastly, I noticed that the last command shows only 173 metaslabs of 174 were loaded. Is this something I should be concerned about?

Thanks in advance for your insights!

[juspolo]

A brief update: as stated, I opted to use directly TrueNAS on the server, and I successfully imported the pool.

Thanks to everyone.