First Time Mistakes – Now Rebuilding My TrueNAS the Right Way (Help Needed!)

TrueNAS General
1

Hi everyone!

I’m looking for some advice on how to “redo” my TrueNAS setup to improve the current pool structure and system disk usage — while keeping my data, app configurations, and permissions intact as much as possible.

Current Setup

I currently have two pools:

:small_blue_diamond: Jupiter

  • 2-disk mirror (7.14 TiB usable) for data
  • 1 SSD used as a Metadata VDEV → I’d like to remove this
  • 1 SSD used as Cache VDEV → Do I really need it?

:small_blue_diamond: Plex

  • 1 single disk (2.63 TiB usable) → I know it’s not ideal, but it’s just Plex content — I don’t mind losing this pool

System

  • TrueNAS OS is installed on an NVMe SSD
  • Machine: Lenovo P520
  • CPU: Intel Xeon W-2135
  • RAM: 32GB ECC
  • GPU: NVIDIA GeForce GTX 1060 3GB (used for plex transcoding and immich)

:thinking: Why I’m in this Situation

This was my first time using TrueNAS, and I’ve definitely learned a lot along the way.

Back then, I didn’t really know what a metadata VDEV was — I added it thinking it was beneficial, but now I realize it’s mostly useful for enterprise or very specific use cases.

Same goes for the cache VDEV — I’m mostly working with lots of photos and small video files (not large sequential files), so I’m unsure if it’s helping at all.

Now that I better understand my use case and what TrueNAS can do, I’d really like to optimize my setup properly.

My Goals

  1. Move the OS from NVMe to SATA SSD , I want to move it to a 120GB SATA SSD since the OS only uses a few GB and I’d rather free the NVMe for better use. (clean install if needed).
  2. Rebuild the Jupiter pool (without metadata VDEV, maybe drop cache too).
  3. Keep the Plex pool disposable, or rebuild it later.
  4. Preserve all my data – I’ve been doing rsync backups to another NAS, so I should be able to restore everything if needed.
  5. Preserve app configurations, dataset permissions, ACLs, and user/group settings – this took a while to fine-tune, and I’d love to avoid redoing everything.

Special Concern: Immich App

One app I’m especially concerned about is Immich – I have nearly 500GB of photos managed by it. The files are backed up, but:

  • I don’t want to lose the database records
  • I use Google Authenticator for login
  • And I’d prefer to avoid re-indexing or reimporting everything if possible

What’s the best way to preserve an app like this across a reinstall?

My Questions

  • Can I rebuild pools and still re-import datasets without breaking things?
  • How do I remove a Metadata VDEV (or is it impossible)?
  • Any advice on preserving apps and configs, especially Immich?
  • Should I export the system config, or will that bring over unwanted settings (like old pool names)?
  • Is it better to re-create datasets manually with the same settings?

Thanks a ton in advance! Any tips or best practices would be greatly appreciated. :pray:

2

As long as you have more hard drives & sata ports available; yes. Copy data to temp destination; destroy, rebuild, copy back.

Otherwise? No.

By destroying the pool :frowning:

I’d just backup whatever app settings manually, and if the app has a backup or a config file, I’d try to find it & save a copy of those as well.

You can, but it would bring things over. Honestly when I clean install & want to keep ACL, I straight up go & screenshot everything, so at least I can mindlessly recreate instead of thinking about it.

If you have a free sata slot, you could mirror the nvme boot drive & the sata boot drive. Then I guess you may have to CLI a bit & zpool detach the now missing nvme… honestly since you’re half considering a clean install, it may just be easier to do that.

It doubtfully is helping. L2ARC, SLOG, and svdev all have their place - ~gross oversimplifications ahead~ but that place very likely shouldn’t be on your home server. They are either adding nothing of value or degrading your performance. Maxing out RAM is likely fastest performance gain.

It’s all good man - many first timers falls for the ‘more hardware = more betterer’ trap.

Good reading: When to use a NVMe cache - #5 by Protopia

I sadly don’t use Immich, so can’t advise - but many apps have the option to backup configs; otherwise, you’ll have to find where those files are & save them in advance.

I’m also hoping you’re going to be using the nvme to host the apps/vms as part of the rebuild. Maybe good idea to add a 2nd one for a mirror?

3

Maybe a new video card, sort of soon? From T3 TrueNAS Tech Talk “… to the upcoming NVIDIA drivers that add support for the 50-series cards while dropping the 10-series and below.”

4

This probably won’t affect TrueNS until 26.04 at the earliest. 25.10 will support the 50xx cards, most likely at the 57x level.

5

Despite what you have been told above as Jupiter is a mirror vdev you should be able to just remove the special vdev from the pool. You can also remove the L2ARC.

But make sure you have an up to date backup first, just in case it all goes horribly wrong. I have however removed a metadata svdev from a mirror pool sucessfully

To maintain the immich setup - I would:

  1. Mirror the boot pool with an SSD (assuming the SATA SSD is => the NVMe)
  2. Remove the NVMe from the boot pool
  3. Reconfigure the BIOS boot

This way the environment is maintained without any issues (including immich)

3 Likes
6

Because you have an “all mirrors” pool you can actually remove the metadat vdev from GUI.

Most likely not. But this can be removed from GUI in any situation.
Since you have a Xeon W-2000, just add more RAM and don’t bother with L2ARC.

You do not have to rebuild anything, unless you want to replace Jupiter by a raidz(2) vdev.
Remove L2ARC and metadata vdev from GUI. Save the configuration file (with secret seed). Replace the boot device. Install TrueNAS anew and load the configuration file.
Done.

There should be nothing to do. Where is it installed?

7

Thanks a lot everyone for your helpful replies and suggestions — I really appreciate the time you took to explain things clearly! :pray:

You’ve helped me better understand the steps I need to take, and I’d like to confirm a few points to make sure I’m on the right track before I proceed.

System Config + Pool Rebuild

  • I’ll backup the system configuration from the web UI (got it).
  • Then I plan to remove the metadata and cache vdev

Boot Drive Migration (NVMe → SATA)

Since the boot drive is a 250GB NVMe SSD, and i need to move to a 120GB SATA SSD to free up the NVMe I’m considering using a third-party cloning tool on another machine (e.g., Clonezilla or similar).
Would that work safely, or would it be better to just do a fresh install on the SATA SSD and restore the system config afterward?

NVMe SSDs (Ex-Metadata + Cache VDEVs)

Now that I’ll have two 500GB NVMe SSDs freed up (previously Metadata and Cache VDEVs), I saw someone suggest using them as a dedicated pool for apps.

Could someone explain the benefits of doing this?
Would it improve app performance or isolation in any meaningful way (e.g., for apps like Immich, etc.)?

I’m curious about what others have found useful in real setups.

GPU Consideration

Lastly, about the GPU swap, yes — I’ve been thinking of switching to a passively cooled GPU to reduce noise. Might go for something basic like a fanless QUADRO or similar, since I don’t need heavy video decoding at the moment (even Plex is mostly idle whit direct stream).

Thanks again for all the guidance!
Let me know if I misunderstood anything or if you recommend a different order for the steps above.

8

Whilst the state of metadata and L2ARC vdevs is probably not held in the system configuration (because it is part of the pool definitions intrinsic to ZFS) it might still be preferable to do this the other way around as you want your system configuration backup to be as late as possible before you rebuild your boot drive.

As NugentS previously advised, just do a fresh install and restore the system config.

(As far as I know there are no cloning tools that can reduce the size of a ZFS partition. So the Clonezilla idea is a non-starter AFAIK.)

Apps tend to have active data and definitely benefit from being on SSD rather than HDD.

Virtual disks and database files act very differently from sequential files - whilst sequential files are usually read and written in their entirety and can benefit from fewer but larger block sizes and from sequential-prefetch, virtual disks etc. instead do high volumes of small 4KB reads and writes to random locations. Aside from the need for higher IOPS, you also need to avoid read and write amplification where e.g. to read a 4KB block you actually need to read 128KB and discard 124KB and to write a 4KB block, you need to read 128KB then insert the 4KB and then write out the 128KB. RAIDZ effectively creates larger blocks, and mirrors are the way to avoid this. And because virtual disks and databases have integrity requirements, you tend also to need to use synchronous writes for these special cases, and so you either need this data on SSD or you need an SSD SLOG.

So - for your use case (apps but no virtualisation), yes an SSD pool is a very useful thing to have, and you should store not only the apps themselves on it, but the data that these apps will use e.g. Plex metadata/transcoding datasets.

My old Terramaster doesn’t have a discrete GPU - only the ancient Intel integrated GPU on the CPU chip, and since I don’t have a Plex Pass, I cannot even use that and if I need to Plex transcode it has to be CPU transcoding. This means that real-time transcoding is not feasible, but background transcoding works fine. So I try to source videos at the 720p resolution I want to use, and if I can only source 1080p then I transcode it to 720p as a background task in order to reduce storage size rather than attempt real-time transcoding.

So if you need a GPU, switching to a more basic GPU will not only reduce noise but also your electricity bill. Or maybe just use CPU transcoding.

But remember that TrueNAS probably needs some sort of video output, and your MB may or may not need a discrete graphics card (could be dumb rather than GPU) to provide that.

1 Like
9

For me the important thing was that you are using rsync to backup to another NAS. I’d start with the idea of changing that to a second TrueNAS and replicating instead. That may lead you to building a low power always on TrueNAS and only firing up this blowtorch when you are backing up.

10

Alright, got it. Here’s the plan:

  • I’ll modify the jupiter pool as planned (remove the metadata and cache vdevs),
  • then back up the system config from the web UI,
  • and proceed with a fresh install on the 120GB SATA SSD, followed by restoring the system config.

As for the SSD pool, thanks a lot for the explanation. I’ll start working on these changes now and then figure out the best way to set it up so apps (like Immich, Plex, etc.) can use it.

Regarding the GPU, I’m a bit concerned the CPU alone won’t be enough:

  • My Xeon W-2135 has an average PassMark score of ~14,200

  • Plex recommends:

    • ~10,000 for a 4K → 1080p stream
    • ~17,000 for a 4K HDR stream

So the W-2135 can handle a single 4K stream, but there’s not much headroom — especially for 4K HDR or multiple concurrent streams.
And since it has no iGPU or discrete GPU, it’s stuck with CPU (software) transcoding only, no Quick Sync.

Most of my media is 4K HDR, and while I do direct stream about 90% of the time, I’d still like to have smooth access when I’m away from home (for work or vacations), so some transcoding headroom would be nice.

As for the idea of a backup TrueNAS box, I’ve been considering it — but realistically, I can’t justify the expense right now.
The spare hardware I have is worse than my current setup (Chinese LGA2011 board, 20-core Xeon, 64GB DDR3 ECC, 550W PSU), and I also don’t have extra drives to build it.

Backups are currently done via rsync to a QNAP NAS I share with a friend — we each have a 12TB disk, split into two pools, and we each use our own 6TB share. So that’s my current backup setup for now.

11

My Intel J3335 has a passmark score of 1,192 - yes not a typo, less than 1/10th of the score of your Xeon - and it performs brilliantly for my own modest needs including NAS, Plex streaming and limited background transcoding and 1Gb/s network.

So I doubt that your passmark score of 14,200 will be an issue for background transcoding.

But I appreciate that with only a single 3TB disk for storing 4K HDR media, you probably don’t have much headroom for pre-transcoding everything so you can potentially watch it on mobile over t’internet.

12

Ok, I managed to do everything!
I removed the SSDs from the Jupiter pool without any issues, did a fresh install on the new boot drive, imported the old config — and everything works! (Which I honestly wasn’t expecting to go so smoothly.)

Now I’ve created a new pool using the two 500GB NVMe SSDs, and I simply called it APP.
But I’m not really sure what to do next: when I go to Apps > Configuration > Choose Pool, and I select the new pool, all my apps disappear.

(I must admit, for a second — when I changed that setting and saw everything vanish — I died inside :sweat_smile:. Luckily, switching back to Jupiter brought them all back.)

So my question is: how can I migrate the existing apps to the new pool?
Is there a proper way to move everything over without breaking the current setup?

13

Normally that should move your apps to the new pool. Copying from HDDs may take some time.

14

So are you saying that the procedure is correct, and I just have to wait?
i’ve just tried to wait, but it just says that there are no app installed…

15

I will open a new topic since is another problem, thanks all for the support!

16

Just try to use client side transcoding for Plex. (as I remember you can set it up in the settings menu)
In that case the servver will send the full, 4kHDR stream and your client will transcode it to the resolution you want to use.
(SO no need for any additional HW on the server side only higher upload bandwidth on your ISP side)