Core to SCALE checklist and concerns (Feedback appreciated)

TrueNAS General
, ,
1

Full disclaimer, I plan to ride with Core up through 2026, or whenever it’s no longer possible to keep my jails (and packages) updated on a FreeBSD 13.3 host.

Overview. You can skip this.

This is a simplified list that applies to myself, as someone who is running bare-metal Core without “Apps” or VMs.

I run several jails using “VNET”, where each jail is treated like its own server with its own unique IP address on the local network. They can be restricted further up with the router, independent of the TrueNAS server, since they do not share the host’s IP address.

I have some tunables applied, namely to load the Wireguard kernel module (to make it available for my jails), and to prioritize metadata in the ARC.

Native encryption is used on all datasets, including the System Dataset, which encrypts the syslog. (My syslog does not live on the boot-pool.)

Nothing complex in terms of datasets. No zvols used.

Swap partition (16-GiB) on the SSD boot device.

The boot-pool is very “quiet”. No syslogs, no System Dataset. Only the pure OS.

Questions and Concerns

.

1. SCALE recently went through (still has?) a transition of how to handle ARC, swapping, and partitioning.

Is it safe to assume that the option to swap (using the rarely touched boot device) will forever be gone? Is Linux so determined to swap, even if “swappiness” is set to the lowest level?

In Core, I currently have it set to create a 2-GiB partition at the beginning of every newly added/replaced drive. Although I don’t use swap, this 2-GiB “buffer partition” serves an important role: It safeguards you against purchasing a drive of the “same” capacity, only for it to be slightly smaller. Will SCALE allow making a “buffer partition”, even without activating it as swap? (i.e, unformatted / undefined 2-GiB partition.) Even if this hasn’t happened to someone yet, it future-proofs against any changes by the manufacturers, where 20-TiB and 20-TiB might not be exactly the same.

Has SCALE, as of 24.10, actually fixed the partitioning bug? Users bumped into issues where a larger replacement drive would not “use the remaining space”, but rather partition out the same size as the other vdev members. When adding/replacing a drive, will SCALE now just add a buffer + “the rest of the capacity”? It’s hard to gauge what SCALE does now.

Is the whole “not leveraging ARC as much as it could” still a thing, or has it been settled by now? Perusing the forums, I’m not really sure what’s going on.

.

2. "Linux jails" with Incus.

I know that a “hidden” dataset (ix-virt) will be created on the pool designated for the LXC containers.

Will we be able to set encryption, compression, and recordsize to this dataset (and its children) before the creation of each jail? Does it simply inherit the pool’s root dataset properties?

One great thing about Core is the control over the iocage dataset. Not only is everything encrypted, of which I never had issues, but I set my jail datasets to ZSTD-9 compression. Good space savings, without impacting performance. (Jail filesystems are mostly comprised of highly compressible files that are “write rarely, read often”.)

.

3. Relocate the syslogs back to the "System Dataset" pool.

SCALE removed the option to keep the syslogs on the System Dataset. :frowning_face: Now they must live on the boot-pool.

The boot-pool cannot be encrypted. The System Dataset can inherit encryption from the pool’s root dataset.

Why was the option to do this removed? I understand the response is “Why do you need to encrypt the logs?” and “It’s easier to debug systems with logs only on the boot device”. I’ve never had a problem before, and I like the zero-cost method of preventing any spillover of information onto a non-encrypted space.

Can this option be brought back with a warning to the user? I’m really not a fan of overdoing this “protect the users from themselves”.

Another benefit of keeping the syslogs off the boot-pool is that your logs are not tethered to your OS or boot device. Even with fresh installations, you still have a history of your logs.

By forcing the syslogs to remain on the boot-pool, the user loses this flexibility.

You should be able to toss a boot-pool, while keeping everything about your system. There’s an option to export your config and encryption keys, but no such option to export your syslogs. Why then force syslogs to live only on the “disposable” boot-pool?

.

4. Auxiliary Parameters.

I probably already know the answer to this. Will these ever return? Must those who shoot themselves in the foot really ruin the user-experience for everyone else who are more responsible?

No compromise? Not even the option to show Auxiliary Parameters, with a warning message and forgoing any bug reports from your “tainted” system?

If TrueNAS is now offered as “Community” and “Enterprise” editions, why not limit this option to the Community edition?

.

5. Tunables and Custom Jobs.

Is the proper method of setting tunables (e.g, ZFS module parameters) to create a Pre-Init script that invokes echo to the desired parameter?

With Core, I have custom Cron Jobs that run (some of them are “disabled” so that I can run them manually on-demand.) Does SCALE provide an equivalent?

.

6. The root user and the "truenas_admin" user.

I’m under the assumption that when you login via SSH, you can still switch to the user root. No different than with Core.

Is the truenas_admin user only meant for the web GUI?

Can you still login as root via the web GUI?

If you use a non-admin username, can you login to a “restricted” version of the GUI?

I’m trying to understand the purpose of a “root-like” user (truenas_admin) to use the web GUI, if normal, unprivileged users cannot. “Root by any other name…”

Maybe in the future, designated (non-admin) users can log into a “restricted” web GUI?

.

7. ACLs and Permissions.

On Core, most of my datasets don’t use ACLs. Basic Unix permissions. The ones that do, I’m assuming these are the same NFSv4 ACLs that SCALE uses as a “default” when you enable ACLs on a dataset?

Does this mean that for datasets using “ACLs” on Core, they should translate without issues under SCALE?

Am I interpreting this correctly: Core only uses NFSv4 ACLS, while SCALE lets you choose between NFSv4 ACLs (same as Core) or POSIX.1e ACLs? Stay clear of the latter and there should be no issues?

.

8. Using a config file exported by Core.

What’s the verdict on this? Due to the divergences between Core and SCALE, is it recommended to install fresh and start without a Core config file? Then just redo the settings, services, and tasks one by one?

I can only speak for myself, as a non-enterprise home user: I really don’t feel comfortable about the trend I’m seeing with SCALE in regards to “dumbing down” the GUI by removing options that were available on Core. I don’t like to be protected from myself.

Because TrueNAS now offers “Community” and “Enterprise” editions, I think the idea of more freedom (such as where to store the syslogs, dataset properties for “hidden” datasets, swap partition on boot device, et al) and Auxiliary Parameters needs to be reconsidered.

Keep the “Enterprise” edition locked down.

For the “Community” edition, allow more freedom to shoot ourselves in the foot. :pray:

“Just because the command-line allows a user to run rm -rf, does’t mean it needs to be denied to everyone.”

  • My System Dataset is encrypted, including the syslogs.

  • All my drives (new and replacements) contain a 2-GiB “buffer partition”.

  • I have a “never-touched” swap partition on my SSD boot drive.

  • All my jail datasets are encrypted, use ZSTD-9 compression, and have a 1-MiB recordsize.

  • I use Auxiliary Parameters.

So far, my house didn’t explode.

1 Like
2

I can’t answer all of your question properly, but if my experience can be usefull:

  • i didn’t find any significative difference between custom Jobs, but my use case was simple (1 curl to duckdns, multi report and NVME smart tests); fully migrated after config upload
  • i have used my core config upload without issue except some minor things >>
  • the truenas_admin user, with other groups, has been literally deleted after core config upload; i logged in GUI as root, but after have created a new admin account (and group ) following the migration path, i disable It as suggested; i struggled a bit with replication permission with my admin user, but at the end i realised that was an ACL problem in his home folder :speak_no_evil:
  • ACL on datasets used by jails on Core gaved me some little problem too, pratically broken by the missing of UserID/groupid that was in the jail… Boring check but easy to fix
  • using a VM for switch jails to docker before the jump to Scale literally saved me a lot of problem, and you can test whatever you want and fix eventually problemi without need to rush
1 Like
3

Here is another victim of the issue I brought up in point #1 in regards to the 2-GiB partition that Core creates by default, which apparently SCALE no longer does?

Seriously, iX. I could foresee this as a end-user without much technical knowledge. How was it overlooked by an enterprise software company?

Explain what I’m missing if I’ve got this wrong.

5 Likes
4

I was the victim on the exact same model being smaller than the older one. I was trying to save money by buying used drives from GoHardDrive. One of them started throwing SMART errors. So I figured I would be wise and buy the exact same model drives. That ended up not working out as the new 12 TB drives were like 1.2 GB smaller than the older exact same model 12 TB drives. I bought 4 in December and five in February. All five February drives were 1.2 GB smaller than the December purchased drives. Same model from the same vendor.

The same thing happened to my NVMEs. I had two 1 TB NVMEs from different manufactures, one well know brand the other not so much. The two were 100 MB or so different and TrueNAS was giving an error. So I ended up buying another of the more well known brand to get rid of the error.

In any case, I ended up buying four brand new 14 TB drives, so my desire to try to save money with GoHardDrive was foiled. So $500 in drives before and over $1000 in drives now because I can’t guarantee a 12 TB drive is a 12 TB drive according to TrueNAS.

In the meantime, if TruNAS were to put that swap partition in, it could make it so every drive nominally the same size is the same size that TrueNAS can use.

3 Likes
5

I feel your pain. If you only had created your pools before the (poorly overlooked) change was implemented in SCALE, you would have been spared from this irreversible issue. :frowning_face:

If it’s any consolation, @yorick created a feature request. We would love for you to vote on it. Hopefully iXsystems will realize how important it is to have this safeguard in place, which they already had for the longest time with FreeNAS, TrueNAS Core, and early versions of SCALE.

3 Likes
6

In addition to voting for the feature that would avoid this pain for vdevs created in future: The smiley face feedback in the UI, for the replacement workflow, goes to the “right people” who give engineers direction on what to implement. If you use that, be sure to add a paragraph or two on how the current behavior impacted you, and what you want instead.

1 Like
7

Shoutout to @Flo for this feature request.

8

This feature request (“reverting a bug”) was accepted. Tracked here on Jira.

2 Likes
9

I’m a little surprised that this hasn’t gotten a bit more in the way of feedback as some of these are easy to answer.

Obviously it’s been a bit of time since you posted this, and I know you’re already aware of some of the answers, but just to have an AIO summary.

  1. :white_check_mark: It sounds like the previous automatic swap use was removed in 24.04.1. The 2GiB buffer feature was restored, in 24.04.0. Lastly, ARC not using nearly as much RAM as it did on Core was also resolved with 24.04.1 (might have been 24.04.0, it’s not clear, but my system does use much more RAM for ARC than previously)

  2. :yellow_square: Currently no, but according to the podcast, most, if not all of these features will be coming in Goldeneye

  3. :red_square: Can’t comment on this one for sure as it’s not something I ever bothered changing, but as far as I can tell the feature is still gone

  4. :red_square: This one makes me sad too. If you happen to recall, you commented on a thread of mine asking for a upsmon.conf section in the NUT UI, only for a staff member to make it pretty clear they’re trying to avoid adding new ones, and even remove old ones to the extent possible. I understand if the goal is to make the values proper UI options, but DEADTIME (an obvious one to me) hasn’t even made it in

  5. :white_check_mark: You’ve been able to add custom Cron jobs and pre/post-init scripts for a while. Before the kernel parameters were quite incomplete, but as of 25.04 you can now add more of them, namely ZFS parametrs. I was really happy to see this as it makes the quota related tweaks I wrote about much easier to apply.

  6. :white_check_mark:/:yellow_square: Basically:

  • By default you have “truenas_admin” for logging into the WebUI
  • Root has no password set which prevents direct login, but you can still sudo or sudo -i into it in the UI shell or via SSH. I think currently you can still give it a password so that it can be logged into if you want (but of course it’s discouraged, and the docs note this option will be “permanently disabled in a future release”). Preventing root use and requiring doing everything through sudo or su has been a common trend in Linux for some time now.
  • Non-admin users cannot login to the Web GUI, they exist primarily for SMB/NFS shares, docker apps, ACLs, etc.

  1. :white_check_mark: It sounds like yes, Core uses NFSv4 ACLs, which were added to Scale later on and should now correctly migrate over. POSIX ACLs are optional.

  2. :white_check_mark: Dragonfish is the latest version that allows migration from Core. It seem like you have 3 options:

  • Upgrade directly to 24.04.2.5 from the UI and hope all goes well. Configure settings with no direct migration.
  • Install 24.0.4.2.5 cleanly from ISO and then once at the Web portal, use the configuration importer to upload your Core config file. Then, configure settings manually with no direct migration
  • Install 25.04 cleanly and manually re-configure everything

It hasn’t significantly impacted me yet (and funny enough the few UI replacements for things they’ve added, like ZFS kernel parameters, have actually made me more whole than less), but I’m also at least a little concerned by the “dumbing down”.

2 Likes
10

Great reply!

2 to 4 are almost deal breakers for me.

2.
I hope that by 25.10, the “Linux jails” (Incus managed LXC containers) will not obfuscate the underlying datasets and permissions, especially at creation time.

It’s a great feature to be able to fully encrypt your containers’ datasets and set their recordsize and compression settings before their datasets are populated with a base OS filesystem.

3.
This means that the boot-pool is not as “disposable” as how it used to be on FreeNAS and TrueNAS Core.

If your syslogs are forced to live on the boot-pool, then how will they survive boot drive disposals?

Isn’t that one of the selling points of a NAS as an appliance? With a brand new boot device, you can load your config and import your pools. There should be nothing important on the boot-pool beyond the NAS software/OS itself. With SCALE, this is no longer true, since your syslogs cannot be relocated to a separate pool.

Sadly, this also denies us the ability to keep our syslogs encrypted. On Core, you can relocate your syslogs to the System Dataset, which can be an encrypted dataset. Not possible with SCALE.

4.
No comment. It’s been said so many times, and I will never agree with the rationale given by iX, especially when Auxiliary Parameters can be hidden behind an “advanced” toggle that will warn the user.

We do have a “Community” Edition now, right? Arguments made for the sake of enterprise clients shouldn’t weigh as heavily on features that will only exist in a “community” edition.

11

I imagine they figure that people don’t care about losing old logs that much, or if they might have sensitive info, but I can see the value in keeping/securing them, especially in an enterprise setting so this one baffles me a bit too.

And yea, the axillary thing to me seems silly. I can empathize with the ideal of removing as much maintenance burden and possible variables when dealing with support tickets as possible, but is it really that much to deal with? A collapsible text box with a warning, in the community edition???

I get the feeling I can respect their reluctance a bit more than you care to XD, but ultimately I agree.

1 Like
12

Goldeneye: Action movie
Goldeye: A fish, also TrueNAS CE 25.10 code name

3 Likes
13

:rofl::rofl::rofl::rofl:

I actually can’t believe I managed to skim the name infrequently enough to think it was Goldeneye this entire time. Wow…

The latter makes much more sense given they’ve all been fish.

Well that’s embarrassing.