Core to SCALE checklist and concerns (Feedback appreciated)

TrueNAS General
, ,
1

Full disclaimer, I plan to ride with Core up through 2026, or whenever it’s no longer possible to keep my jails (and packages) updated on a FreeBSD 13.3 host.

Overview. You can skip this.

This is a simplified list that applies to myself, as someone who is running bare-metal Core without “Apps” or VMs.

I run several jails using “VNET”, where each jail is treated like its own server with its own unique IP address on the local network. They can be restricted further up with the router, independent of the TrueNAS server, since they do not share the host’s IP address.

I have some tunables applied, namely to load the Wireguard kernel module (to make it available for my jails), and to prioritize metadata in the ARC.

Native encryption is used on all datasets, including the System Dataset, which encrypts the syslog. (My syslog does not live on the boot-pool.)

Nothing complex in terms of datasets. No zvols used.

Swap partition (16-GiB) on the SSD boot device.

The boot-pool is very “quiet”. No syslogs, no System Dataset. Only the pure OS.

Questions and Concerns

.

1. SCALE recently went through (still has?) a transition of how to handle ARC, swapping, and partitioning.

Is it safe to assume that the option to swap (using the rarely touched boot device) will forever be gone? Is Linux so determined to swap, even if “swappiness” is set to the lowest level?

In Core, I currently have it set to create a 2-GiB partition at the beginning of every newly added/replaced drive. Although I don’t use swap, this 2-GiB “buffer partition” serves an important role: It safeguards you against purchasing a drive of the “same” capacity, only for it to be slightly smaller. Will SCALE allow making a “buffer partition”, even without activating it as swap? (i.e, unformatted / undefined 2-GiB partition.) Even if this hasn’t happened to someone yet, it future-proofs against any changes by the manufacturers, where 20-TiB and 20-TiB might not be exactly the same.

Has SCALE, as of 24.10, actually fixed the partitioning bug? Users bumped into issues where a larger replacement drive would not “use the remaining space”, but rather partition out the same size as the other vdev members. When adding/replacing a drive, will SCALE now just add a buffer + “the rest of the capacity”? It’s hard to gauge what SCALE does now.

Is the whole “not leveraging ARC as much as it could” still a thing, or has it been settled by now? Perusing the forums, I’m not really sure what’s going on.

.

2. "Linux jails" with Incus.

I know that a “hidden” dataset (ix-virt) will be created on the pool designated for the LXD containers.

Will we be able to set encryption, compression, and recordsize to this dataset (and its children) before the creation of each jail? Does it simply inherit the pool’s root dataset properties?

One great thing about Core is the control over the iocage dataset. Not only is everything encrypted, of which I never had issues, but I set my jail datasets to ZSTD-9 compression. Good space savings, without impacting performance. (Jail filesystems are mostly comprised of highly compressible files that are “write rarely, read often”.)

.

3. Relocate the syslogs back to the "System Dataset" pool.

SCALE removed the option to keep the syslogs on the System Dataset. :frowning_face: Now they must live on the boot-pool.

The boot-pool cannot be encrypted. The System Dataset can inherit encryption from the pool’s root dataset.

Why was the option to do this removed? I understand the response is “Why do you need to encrypt the logs?” and “It’s easier to debug systems with logs only on the boot device”. I’ve never had a problem before, and I like the zero-cost method of preventing any spillover of information onto a non-encrypted space.

Can this option be brought back with a warning to the user? I’m really not a fan of overdoing this “protect the users from themselves”.

.

4. Auxiliary Parameters.

I probably already know the answer to this. Will these ever return? Must those who shoot themselves in the foot really ruin the user-experience for everyone else who are more responsible?

No compromise? Not even the option to show Auxiliary Parameters, with a warning message and forgoing any bug reports from your “tainted” system?

If TrueNAS is now offered as “Community” and “Enterprise” editions, why not limit this option to the Community edition?

.

5. Tunables and Custom Jobs.

Is the proper method of setting tunables (e.g, ZFS module parameters) to create a Pre-Init script that invokes echo to the desired parameter?

With Core, I have custom Cron Jobs that run (some of them are “disabled” so that I can run them manually on-demand.) Does SCALE provide an equivalent?

.

6. The root user and the "truenas_admin" user.

I’m under the assumption that when you login via SSH, you can still switch to the user root. No different than with Core.

Is the truenas_admin user only meant for the web GUI?

Can you still login as root via the web GUI?

If you use a non-admin username, can you login to a “restricted” version of the GUI?

I’m trying to understand the purpose of a “root-like” user (truenas_admin) to use the web GUI, if normal, unprivileged users cannot. “Root by any other name…”

Maybe in the future, designated (non-admin) users can log into a “restricted” web GUI?

.

7. ACLs and Permissions.

On Core, most of my datasets don’t use ACLs. Basic Unix permissions. The ones that do, I’m assuming these are the same NFSv4 ACLs that SCALE uses as a “default” when you enable ACLs on a dataset?

Does this mean that for datasets using “ACLs” on Core, they should translate without issues under SCALE?

Am I interpreting this correctly: Core only uses NFSv4 ACLS, while SCALE lets you choose between NFSv4 ACLs (same as Core) or POSIX.1e ACLs? Stay clear of the latter and there should be no issues?

.

8. Using a config file exported by Core.

What’s the verdict on this? Due to the divergences between Core and SCALE, is it recommended to install fresh and start without a Core config file? Then just redo the settings, services, and tasks one by one?

I can only speak for myself, as a non-enterprise home user: I really don’t feel comfortable about the trend I’m seeing with SCALE in regards to “dumbing down” the GUI by removing options that were available on Core. I don’t like to be protected from myself.

Because TrueNAS now offers “Community” and “Enterprise” editions, I think the idea of more freedom (such as where to store the syslogs, dataset properties for “hidden” datasets, swap partition on boot device, et al) and Auxiliary Parameters needs to be reconsidered.

Keep the “Enterprise” edition locked down.

For the “Community” edition, allow more freedom to shoot ourselves in the foot. :pray:

“Just because the command-line allows a user to run rm -rf, does’t mean it needs to be denied to everyone.”

  • My System Dataset is encrypted, including the syslogs.

  • All my drives (new and replacements) contain a 2-GiB “buffer partition”.

  • I have a “never-touched” swap partition on my SSD boot drive.

  • All my jail datasets are encrypted, use ZSTD-9 compression, and have a 1-MiB recordsize.

  • I use Auxiliary Parameters.

So far, my house didn’t explode.

1 Like
2

I can’t answer all of your question properly, but if my experience can be usefull:

  • i didn’t find any significative difference between custom Jobs, but my use case was simple (1 curl to duckdns, multi report and NVME smart tests); fully migrated after config upload
  • i have used my core config upload without issue except some minor things >>
  • the truenas_admin user, with other groups, has been literally deleted after core config upload; i logged in GUI as root, but after have created a new admin account (and group ) following the migration path, i disable It as suggested; i struggled a bit with replication permission with my admin user, but at the end i realised that was an ACL problem in his home folder :speak_no_evil:
  • ACL on datasets used by jails on Core gaved me some little problem too, pratically broken by the missing of UserID/groupid that was in the jail… Boring check but easy to fix
  • using a VM for switch jails to docker before the jump to Scale literally saved me a lot of problem, and you can test whatever you want and fix eventually problemi without need to rush
1 Like
3

Here is another victim of the issue I brought up in point #1 in regards to the 2-GiB partition that Core creates by default, which apparently SCALE no longer does?

Seriously, iX. I could foresee this as a end-user without much technical knowledge. How was it overlooked by an enterprise software company?

Explain what I’m missing if I’ve got this wrong.

3 Likes
4

I was the victim on the exact same model being smaller than the older one. I was trying to save money by buying used drives from GoHardDrive. One of them started throwing SMART errors. So I figured I would be wise and buy the exact same model drives. That ended up not working out as the new 12 TB drives were like 1.2 GB smaller than the older exact same model 12 TB drives. I bought 4 in December and five in February. All five February drives were 1.2 GB smaller than the December purchased drives. Same model from the same vendor.

The same thing happened to my NVMEs. I had two 1 TB NVMEs from different manufactures, one well know brand the other not so much. The two were 100 MB or so different and TrueNAS was giving an error. So I ended up buying another of the more well known brand to get rid of the error.

In any case, I ended up buying four brand new 14 TB drives, so my desire to try to save money with GoHardDrive was foiled. So $500 in drives before and over $1000 in drives now because I can’t guarantee a 12 TB drive is a 12 TB drive according to TrueNAS.

In the meantime, if TruNAS were to put that swap partition in, it could make it so every drive nominally the same size is the same size that TrueNAS can use.

2 Likes
5

I feel your pain. If you only had created your pools before the (poorly overlooked) change was implemented in SCALE, you would have been spared from this irreversible issue. :frowning_face:

If it’s any consolation, @yorick created a feature request. We would love for you to vote on it. Hopefully iXsystems will realize how important it is to have this safeguard in place, which they already had for the longest time with FreeNAS, TrueNAS Core, and early versions of SCALE.

2 Likes
6

In addition to voting for the feature that would avoid this pain for vdevs created in future: The smiley face feedback in the UI, for the replacement workflow, goes to the “right people” who give engineers direction on what to implement. If you use that, be sure to add a paragraph or two on how the current behavior impacted you, and what you want instead.

1 Like
7

Shoutout to @Flo for this feature request.

8

This feature request (“reverting a bug”) was accepted. Tracked here on Jira.