Replacing single drive MetaDataVdev with an other drive (migrate NVME to (smaller) SATA)

TrueNAS General
,
1

Hello,

As private user I build my TrueNas Scale system based on an AMD B550 motherboard. That motherboard has a limited number of PCIE-lanes, which as consequence that I can only connect two NVME-SSD’s

Up to now I use one for Virtual Machines etc. things which should be fast. And I use the the other NVME-slot for an 1TB-SSD which I use as MetaDataVdef in combination with a big harddrive. That is OK , however with one big problem …
I only have two NVME-slots :frowning:

So at the moment I am considering to replace the NVME MetaDataVdef with a SATA SSD to free up the NVME-slot.

A few questions:

  1. how to migrate the MetaDataVdef from actual NVME SSD to a SATA SSD
  2. I do have a 500 GB SATA-SSD, it would be nice to use that one. But of course it is a smaller size. I did configure the MetaDataVdef to catch as much data as technical possible, trying to use the related HD only for bigger files (> 500k or 1 MB). I think I can not see nor change that via the GUI.
  3. I will probably lose some performance by swapping the NVME for a SATA drive. But I do not know how much …

Of course I could also buy and install a big (4TB) SATA SSD instate of a 4TB NVME SSD. However I think that is a bad Idea since SATA SSD’s are as expensive as NVME SSD’s and of course 10 times slower and not a future proof investment.

So is this migration possible? (if so how)
Has any one experiences with this kind of set-up?

1 Like
2

Post your hardware please - and the output from zpool status please.

1 Like
3

You should do the other way around, using a SATA SSD for your VMs and mirroring your single metadata drive: if you lose that drive, you lose the entire pool. You should also not use a HDD in a metadata VDEV, especially if you mix it with SSDs.

Metadata VDEVs should have the same level of redundancy of your data VDEVs.

Nevertheless, if you want to use a SATA SSD instead of the NVMe SSD for your metadata VDEV you have to connect the SATA SSD and software replace the NVMe drive with the SATA one via the GUI.

Do as @NugentS wrote, then we can give you a way out of your self-created minefield.
4

The trouble is that he cannot replace - the SSD is smaller than the NVMe.

If his pool is NOT RAIDz then he MIGHT be able to remove the svdev in which case there is light at the end of the tunnel.

1 Like
5

I know when I was playing with SCALE using VM drives, I was able to remove the Metadata Vdev without issue, but it was a single drive stripe.

What is your use case? I’m curious if you need the Metadata Vdev in the first place.

6

You can only remove metadata vdevs if there are “no top-level raidz vdevs”

This command supports removing hot spare, cache, log, and both mirrored and non-redundant primary top-level vdevs, including dedup and special vdevs.

1 Like
7

I am not yet sure what to do but note that I am facing the following limitations:

  • I only have four SATA ports and two NVME-connections
  • One SATA-port is used for the system drive (I hate that, but can’t change)
  • All disk functions are “per physical disk”, so I can not partition a drive / ssd to use it for multiple purposes
  • Big drives are all HD and beside that they are much cheaper

I tried to use these limited options in the best possible way even more restricted by interfaces than by cost

  • one SATA for the system disk
  • one SATA for ^archives^
  • one SATA combined with a (NVME) SSD for Metadata) as fast big HD
  • one SATA for temp files etc
  • one NVME for VM’s etc

Given that situation, there is no physical space for redundancy. However now and than I copy data to the ^archive disk^

So within this machine the only option I have is to swap the MetaData drive with the temp SATA in order to free the NVME-slots for extention of NVME -storage capacity.

I am not sure if that is worth the money, effort and risk.

I would love to have more PCI-E-lanes … but that would imply a whole new expensive (high end) system.

8

I know you dont wanna hear it, but this is a very…interesting…setup.

6 drives is plenty. 3 spinners in raidz1. 2 nvme as mirrors for vms. 1 boot ssd

Or boot from an external quality ssd, or a quality USB stick.
Easy peasy. No special vdevs needed.

You are just putting your data at risk. Dont be “that guy”.

Im sorry, but not every piece of random storage device that is lying around can be turned into a NAS.

1 Like
9

As described, your system should have a free x16 PCIe slot, which can be bifurcated 4x4x4x4 if this is a Ryzen CPU (x8x4x4 with an APU). So you can have more NVMe drives.
Single drive pools hardly levrage the benfits of ZFS; some redundancy would be better.

The good thing is that you can just remove the metadata vdev (vdeV, nof vdeF) from the GUI.
The question is where you want to go.

10

There is plenty, you just have a badly organized system.

You have 4x SATA ports and 2x NVMe slots (ignoring the x16 slot for the purpose of this organization). You could do the following:

  • 1x SATA for the boot drive
  • 2x NVMe for the VM/App pool
  • 3x SATA for your storage pool composed of 3 HDD in RAIDZ1.

You likely don’t require a metadata VDEV.
Please post the output of zpool status as requested.

11

So true.

You can do quite a bit if you are willing to use an add-on card. Bifurcation would allow you to use NVMe drives (3 or 4 depending on how many lanes are open), or there are other NVMe add-on card options that will use the x16 slot. However NVMe drives are not cheap. And then there are add-on cards for HBA’s to add hard drives. You might need a new case. You have options.

12

Note that

  • I see the archive drive and other PC is as a form or redundancy
  • I am using a AMD550 uATX bord with two 16 lane pcie and one 1 lane pcie
    a) Where using the 1 lane is disabling things, so is effectively not usable
    b) I use a CPU with build in GPU to save lanes and to allow a second GPU
    c) In the GPU-slot there is a low cost GPU intended for VM’s, however GPU share-ing seems to be an illusion is in practice not supported
    d) the other PCIE-slot is used by a 10G-NIC
  • the uATX case does not have the option for a 3e 3.5 inch HD drive (or an ATX-board)

So what I could do:

  • move the system drive to a better quality USB-stick to free up a SATA-port to connect a NVME-drive via SATA (in fact I have already connected an USB for that, just not yet using it). I really consider it as “waste” to give up an whole sata port and disk.
  • connect an NVME-driver via USB, however the result is about two times sata and 1/5 or less of an internal NVME. But as advantage, that it could be reused in future setup

As said, for the moment I am not convinced of one or more of those options.

13

One of the big things people have a difficult time with is “waste of drive soace” for say a boot drive that needs very little space but a 256GB SSD cost so little. This is one of those, you are buying the function of the device, not necessarily to capacity.

As already suggested, a USB to SSD adapter is a good boot device, just buy a good one as you get what you pay for. I still don’t know what your use case is but it sounds like you want to run some VMs and pass through a GPU card. I know some people have had success with that, and if things are not working for you there, look into proxmox or my favorite Vmware Esxi. These may be good options for you.

I hope you have your answer to your original question, it seems to have changed into a build thread.

Good luck.

14

I still did not decide if and how to extend the config. For the moment I just decided to move the boot partition to wards USB-sticks.

For that I am going to use two “Samsung FIT Plus 256GB Type A 400 MB/s USB 3.1 Flash Drive (MUF-256AB/APC)” drives as one raid-1 device connected to two 5gbit USB-ports. So it will effectively be one more reliable 256GB drive.

I need to migrate from an 500G drive, So I hope I can do migration in a simple way (via the commandline, I just do not jet know how). Of course I can do a fresh install and import the pools, but if that is a simple method … I doubt the more that NAS does normally no keyboard or display attached.

15

Export the config. Reinstall. Import the config.

16

Yep that is probably the best option. However I have to move the NAS to my workroom and back for that. I do not have a KVM :frowning:

17

Without a full description of your system and your needs, preferably in the first post, we cannot know.

Disabling? That’s strange for what is likely a chipset lane.
Not knowing more, this seems very suitable for boot from a cheap M.2 NVMe drive on a $3 adapter. Freeing all SATA ports for HDDs—ideally a 4-wide raidz2.

18

In the motherboard manual there is a sentence saying:

  • PCIEX16_2 will run in X2 mode when PCIEX1 is used

If PCIEX1 is not used than PCIEX16_2 is running in X4 mode. Since I use PCIEX16_2 for my 10G card (Mellanox ConnectX-3 PCI-E 3.0 x8), I need those 4 lanes for sufficient bandwith (4 gigabytes second).

In PCIEX16_1 (connected to the CPU) I have a low end GPU (GeForce GT 1030).

I general that grafic cards neerly always get 16 lanes of the very limited number of lanes available … but I can not change that. I am throwing at least 8 lanes away there … :frowning:

19

Mind sharing the motherboard model?

20

Motherboard ASUS TUF GAMING B550M-PLUS (WIFI)
CPU: AMD 5700G (disadvantage PCIE3, however very important with internal GPU).
2e GPU: GeForce GT 1030
92 GB RAM
1 x 16 TB HD
1 x 16 TB HD + 1x 1TB NVME
1x 4 TB NVME SSD
1x 500 GB SATA SSD
1x 500 GB SATA SSD as systemdisk/bootdrive
1x Mellanox Technologies MT27500 Family [ConnectX-3) 2 x 10G
2x Samsung FIT Plus 256GB USB planned to replace the sata bootdrive