Rebooting TN server after 3 - 4 years, and need bringing up to date! (Technical)

TrueNAS General
, , ,
1

About 3 or 4 years ago (2023) I had to mothball my server (TN community edition). Im now in the process of un-mothballing it and bringing it back live, and restoring my pools from backup. But a huge amount’s changed, and I can’t be sure what I knew is valid any more.

Level of experience + type of use:

In my past TN life I was comfortable with CLI and web UI, while I didnt know huge amounts about FreeBSD or Linux, I got in-depth with ZFS itself, down to troubleshooting and dtrace type of work if that’s a guide. I only really used SMB + iSCSI, but the pool had a fair amount of ACLs and hard links set up. It’s also one of those rare pools where dedup was rational - deduping cut data size by 70% (50 TB to under 15 TB) making storage affordable, and routine handling of 50 - 800 GB disk+VM images/files/folders. The server was sized for that workload - 256 GB RAM, a bunch of tunables, a very large and fast pool of 3 way 7200 enterprise data mirrors + metadata/SLOG Optane mirrors, for fast resilience/scrub/resilver, Supermicro board + 16 core Xeon, and 10G ethernet pipe feeding it. A lot of work and cost, but totally worth it for my use-case.

What Im trying to learn:

Any links and comments deeply appreciated!

  1. Implications of active CORE moving to Linux?
    I gather CORE’s development is now Linux based and the FreeBSD version I knew is maintenance only? I missed all the in-depth posts/guides, the implications for managing/migrating the server, or what to do/expect when I boot it up after 3 years in storage into FreeBSD CORE. Id deeply appreciate links (“how to” and technical “under the hood”) for migration and also familiarisation, and any options or considerations in doing so. No need to re-explain what Im sure has been well discussed/described, but where do I look for that info?

  2. Implications of fast dedup?
    This is a massive “hooray” - but it probably means most of my knowledge about dedup is outdated, and everything needs rechecking. No idea where to start with this, as older dedup needed considerable fine tuning and I dont know whats relevant/new in 2025 era. I do have the option to send/receive (rebuild) my pool to new disks. I found the blog posts and reviewers guide. But they dont answer much. Im really after a crash course reading up on everything about sysadmin-ing with fast dedup. Any suggestions for in-depth material/guides?

  3. Other significant changes to TrueNAS CE and ZFS?
    I know a huge amount has changed. But what are the major changes (or their subtle effects) I should look out/watch for in running a TN server, primarily as an SMB server with this type of pool? To what extent am I best to approach TN in December 2025 as a completely new to me platform, or as a lowkey evolution of a well-known platform?

    Also, as an experienced ZFS tinkerer, seeking raw pool performance/resilience so far as dedup allows, what would you say about ZFS-specific notable changes to look out for? Really worthwhile “well known” tunables, major landed features, or “under the hood”? (e.g., did we get any major updates to metaslab/space maps/ARC/caching/scrub/resilver, or anything else?) What’s likely to be useful/important for me to read up about, as I configure my server and pool for 2026 not 2023?

I appreciate these are questions that could have long answers, Im honestly after links, guides/howtos/”well-known posts” I missed, key release notes to read, and summary pointers generally - whatever you think’s worthwhile to share, or worth a read.

Huge thanks for any replies and input!

2

Answer 1: If your TN system is installed on an SSD and hasn’t been powered on for such a long time, I think the probability of you successfully booting the system is very low.

Since the release of SCALE, TN has undergone several destructive updates. Even if your system boots normally, upgrading to the latest SCALE will take a significant amount of time and may encounter bugs that most people won’t encounter. Therefore, if possible, try booting into the core, noting down key configurations such as IP address and shared paths, and then reinstalling the latest SCALE system.

Unless you have enough time, sufficient technical skills, a good reason, and a lot of luck, please don’t attempt this type of upgrade directly.

Answer 2: ZFS Fast Dedup significantly reduces memory usage, but it’s still recommended to allocate at least 5GB of memory per TB of data. Aside from that, Fast Dedup doesn’t offer many changes. Unfortunately, I haven’t found a systematic guide to Fast Dedup. My advice is as follows: First, read Jason Rose’s blog to recall some introductory information about the TrueNAS system; then, read Jim Salter’s blog for advice on ZFS; and finally, read the discussions of Fast Dedup on GitHub to learn about some technical details.

Answer 3: You should treat SCALE as a completely new platform. SCALE has undergone several major technical shifts, including moving from Docker to K3S and then back to Docker; from libvirt to Incus and then back to libvirt. These changes were disruptive and incompatible, so please be prepared and carefully read every sentence in the release notes.

3

It’s 2025 now, so wouldn’t this have been 2 years ago? Or can I not math?

No, it was k3s to Docker–there was no time before k3s when SCALE (now CE) used Docker.

I don’t think Incus is gone yet; that’s happening in 26.04 (unless iX changes their mind again).

And only the latter of these changes would have any possible effect on a migration from CORE; any apps would need to have been recreated from scratch in any event (under Docker, unless for some reason OP decided to move to 24.04 or an earlier version of SCALE).

1 Like
4

But there was a time when k3s used docker as runtime before it changed to containerd (if i remember correctly) and during that time you could run docker container with standard docker commands from cli.

1 Like
5

@apple -

Thanks so far - this is good info and somewhat reassuring.

We shall see if the server boots off its old boot pool - if not then as you point out, I can use my config backup. I suspect it will, because mothballing was in two stages - pool data backup and pool disks first, server with just system disks (not 20+ HDDs and a rack for them!) was scrubbed maybe a year or 18 mo ths ago, and is mirrored too. We shall see, probably not fatal if not. Even if 3 of 3 disks in some vdev of the original pool lost same data, I have 95% of it backed up in about 3 places, so Im hopeful but what’ll be will be, fingers crossed is all I can say. Did the best I could.

back to rebooting. Fortunately no apps, no packages, no jails, no iocages, no docker or K3S (guessing thats similar). Thinking about it even the iSCSI vol is redundant and can be deleted, the files are all on SMB volumes anyway. So just SMB with NFSv4 ACLs on the data dirs. The data pool isn’t POSIX compatible (deny + inheritance used), but I gather TN NFSv4 layer for SMB is reliable even on Linux. I think those and TN config are all there is. if SCALE can upload an old CORE config file and update its structures to current config, Im hoping it’s all good. if not, I guess boot (or reinstall), run old config, write it all down and set up on installing SCALE, or update to latest CORE then check instructions to migrate latest CORE to SCALE? Guessing Im not the only one still not migrated, so that with luck is supported.

Hopefully that and restore + scrub pool is hopefully whats needed to at least get safe and up to date? Or was CORE to SCALE a “plan migration + server wipe (if reusing same boot disks) + install SCALE + import config” situation for all users?

Have I missed anything, other than immense care on every step and reading all release notes for any upgrades/pool imports (as @apple emphasised)? Which are the release/migrate notes I need to read?

@dan - thanks likewise. 3 years ago (all of 2023+4+5), lost count :grin:

Remaining questions stand though -

  1. Given my main use was SMB + NFS ACLs, and anything else is probably safe to delete if I keep a config, do I face any deeper issues beyond taking care and double checking all commands and steps are followed?
  2. If the server boots and data is restored, is bringing it up to latest CORE and then migrating to SCALE the expected approach? If so where do I find the guide for the process?
  3. Is CORE to SCALE a fresh install or is there magic whereby it works from within the UI as an upgrade (with a fresh install to Linux done out of user sight)
  4. Where are docs and guides kept now? Especially link for the new sysadmin’s “user manual/user guide” to current TN SCALE CE (if thats the title), to make sure I get the right one and learn about the new platform before moving my server to it?
  5. @apple (and anyone) - thanks for the heads up there isnt a systematic sysadmins guide for fast dedup. That saves a lot of looking. I read the developer discussions/issue threads on it but those are largely implementation not management. I’ll reread man pages, maybe OpenZFS source for inline comments on tunables (was invaluable with old ZFS). My concern is also stuff like block sizes + space map tunables, things like that. I did a lot of tuning of those, to get dedup workable. Perhaps some are redundant or counterproductive now. Im struggling to figure how to tell. After all the entire SCALE IO subsystem is utterly different, so nothing that I set up is necessarily right on SCALE. Apart from prefetch and broad memory stuff which is obvious. Any tips how to approach learning and educating myself, and deeper understandings, very much appreciated. I might have to open a new topic some time, post the tunables I cant figure what to do with and ask help, but thats probably down the line. My concern is stuff that cant be changed without rebuilding the entire pool - basic pool definition tunables that control blocks, space maps, ddt, etc.

In the meantime many many thanks, and any further help or links about these questions please let me know! Its much appreciated.i

6

Yes. Upgrade to latest CORE (13.0-U6.7–these version numbers are getting almost as bad as in the 9.2 series), migrate to CE (formerly SCALE) 25.04, upgrade if desired (though I wouldn’t recommend upgrading to 25.10 just yet–the .0 and usually .1 releases should be considered beta quality). See:

(why iX doesn’t have this page available for later versions of CE is a mystery, but that’s iX for you)

7

You can as Dan says migrate from Core to CE (scale). Your success rate mileage may vary. Back on Bluefin Scale version era I think it was I tried the migration from the latest then core version to the current then scale version. The system was a very basic storage system with no jails, no encryption, and just a couple SMB shares attached to a pool consisting af several vdevs configured in a raidz2 config. The migration was only partially successful and created a number of operational issues that it should not have resulting in the “it’s just easier and less time consuming to install Scale from scratch than continuing to use a hammer” approach. To be fair, the docs have greatly improved since that time so there is a greater chance of a successful migration.

In the end I would still suggest anyone wanting to move from a Core system over to the Truenas CE (Scale) system to strongly consider just rebuilding the system from scratch.

If that is not possible for whatever reason, then make sure everything and I mean everything is updated on the current core system before starting; Core to the latest core version etc. and solve any issues or quirks with the system before continuing or they will bite you. If anything is encrypted you may have issues so that is something to consider.

I would suggest to read and reread again several times the migration docs Dan posted and follow any links within the migration docks and make sure you actually understand what is being said and why.

It is also a good idea to have a monitor and keyboard available to attach to the server as it may be needed and it is wise to make sure that all works correctly too before starting a migration.

Also I agree with Dan in that the latest version of Truenas CE I would use is 25.04.2.6 as 25.10.x.x is considered Beta and will be Beta until maybe .2 release and not ready for production at this time. It will likely cause issues you don’t need during a migration.

1 Like
8

One question:

Are OpenSFS pool disks compatible equally with FreeBSD CORE and Linux SCALE?
In other words, can I migrate the boot system, ensure that its all good and seems correctly done, then only after that, reattach my existing pool from CORE and it’ll equally work with ZFS under Linux identical to ZFS under FreeBSd (NFSv4 ACLs aside)?

Or are there differences that mean a FreeBSD operated pool must also be migrated to SCALE, or will be modified when first imported on a SCALE platform?

9

You should be able to do this, and even bring the pool back to CORE, as long as you don’t “upgrade” the pool.

1 Like
10

@dan - excellent, thank you! Then my path is clear. I think !!

11

That was my experience - even on a few Scale updates I did clean installs otherwise I had random driver issues randomly (to be fair my nic isn’t really kosher…)

12

With the server requiring config upload, and the pool OK on either, and no jails or apps to transfer, it honestly sounds like there’s no reason not to reinstall in this case really. If needed save original boot disks. Install on other boot disks, import config, and import pool.

1 Like
13

Config import also imports the pools, no need do it as a second step

14

Okay, time for a quick check on next steps.

Current status:

The server booted fine, and all pools are readable, but some disks are faulting, and potentially others may suddenly die during recovery or scrub due to the long period not in use (eg lubricant migration etc). SMART has limited predictive power for this.

  • Boot pool (small Intel SSD mirror) is good, so I updated from CORE 13 U5.3 to U6.8 (latest CORE), and that’s fine. Not worried about boot pool, as I have the U6.8 config saved.

  • Pool copy 1: Main data pool is 4 x 3-way mirrors (enterprise 6 - 10 TB 7200 HDDs), and 2-way mirror special (optane 905P 480GB).

    pool status -v says the data pool has lost 1/3 HDDs in one vdev, and 2/3 HDDs in a second vdev (the other vdevs are all ok). I’m proposing to buy a replacement 8TB HDD for the vdev that lost 2 HDDs. smartctl -a and -l suggest the 3rd disk in that vdev is in good shape.

  • Pool copy 2: Non-redundant full backup - There’s a single non redundant 2nd copy of the pool with no redundancy, spread across 3 single HDDs and one SSD for special. More just a last recourse backup made up of all the spare disks I had laying around, to give a slim chance of total recovery even if I lost an entire vdev in the main pool while mothballed.

  • Partial pool copy 3: Critical data also backed up to PC - critical files have a 3rd backup which is on a 20 TB HDD on my PC, marked via NFTS as read-only. Currently reported via chkdsk as good. Not checked SMART.

  • Worst case loss - at worst I wont lose too much critical stuff even if I lose any 2 of the above 3. (But obviously Id like not to be at that point).

  • Scheduled pool activity all disabled - I have disabled all scrubs, snaps, commands/scripts, and replication tasks in the Web UI, so as not to put any load on any data disks until I decide next steps and fixing redundancy. So both pools are online but idle.

Intended new pool and backup:

The data pool is going to move from HDD to datacentre SSDs. The new pool will have 3 mirrored devices:

  • 16 TB data vdev:- Kyoxia read-intensive SSDs mirrored pair
  • 8 TB data vdev - micron 4450 read-intensive SSDs mirrored pair
  • 480 GB special vdev (including fast dedup DDTs) - optane 905P mirrored pair

While redundancy goes from 3 disks to 2 per mirror, it halves the number of vdevs, and I gather that datacentre SSDs are probably more reliable by far on “total loss” and graceful failure than HDDs. I’ll use the working original HDDs to create an ongoing redundant replicated backup on a second server. Between those combined approaches, I reckon Im more than compensating for reducing the mirrors from 3 to 2 way.

Critical risks:

  1. one degraded vdev in the original HDD pool has lost 2 of 3 disks (smartctl says the 3rd is in good shape but a single disk left in a vdev is always major red flag alarm bells).
  2. the risk of sudden failure of one or more other disks, when used intensely over 1-3 days to zfs send the pool to the new SSDs, due to unpredictable mechanical issues (lubricant viscosity/movement, magnetic degradation etc) after long storage.

Proposed strategy:

As I plan to migrate the pool to the new SSDs anyway, I feel my best strategy might be:

  1. Upgrade the server to SCALE (without anything triggering pool activity). Possibly install SCALE onto new boot disks, then import saved CORE config, rather than upgrading old CORE boot pool.

  2. Once on SCALE, import the pool but dont allow scrub or any other intensive tasks. Instead, take advantage of SCALE’s sequential resilver to improve the odds of adding an extra disk to the 2/3 failed vdev, and reduce the risk of total vdev loss. Add disk, allow/trigger sequential resilver.

    Noting that sequential resilver will need a scrub later, but this seems the exact situation it’s designed for. Also, it seems best to trust the 3rd disk in that vdev as little as possible, as it’s been offline 3 years whatever SMART reports.

  3. Once the pool is redundant again, still don’t scrub. Just directly send/receive the entire pool to the new SSD disk set, to create a new pool that replicates the original, but written with fast dedup from the start under SCALE, which will help a lot. The aim here is to migrate the pool to its new disks immediately (which I’d do anyhow as soon as safe) and minimise load and duration of the process. Fast dedup will make that better all around, and the sooner the original HDD set isnt needed or relied upon, the better. Separate scrub after sequential resilver is never needed, because it’s integrity-checked anyway as part of send/receive.

Thoughts appreciated!

Does that seem sensible?

15

Found a good resource on config of fast dedup. Hope it Helps: