TrueNAS Scale for SoHo usecases

[Whattteva]

@Steven_Ellis

I’ve been using Linux for well over 25 years, and unless this is a very small footprint embedded style system using cheap flash storage I’d always add a little swap.
Today most of my systems have NVMe backed high performance storage or at the very least SATA class flash storage for boot and swap.

No storage (not even NVMe) can match the raw performance of RAM especially in writes. This storage hierarchy (L1 > L2 > L3 > RAM > bulk storage has been true for decades and probably will continue to be true for at least another decade.

[NickF1227]

@Whattteva Generally speaking, you’re not wrong. Your point that NVME is still an order of magnitude slower than RAM isn’t lost on me here. Nor I think on OP who’s tag line reads
Architect at Red Hat and long time Linux and Open Source user.
I think he’s aware.

But, to @Steven_Ellis’s point, the “pain” felt by swap is no where near as high when we’re talking about NVME. Just purely in terms of latency, NVME is an order of magnitude faster than spinning drives. We’re talking 100s of microseconds vs 10s of milliseconds.

Regarding the issue at hand, the speed has nothing to do with the reason swap is off by default now.

When I deploy Linux VMs for XYZ application, I typically give them swap roughly equal to the amount of RAM I give them. If your webserver for an app suddenly gets hit hard, you’d prefer your webapp start swapping to disk. The result is worse performance, but you’re still serving your data.

TrueNAS SCALE isn’t a simple web app, and it’s not even a simple file server. It’s got all sorts of processes competing for RAM, whether it be SMB, KVM workloads, Kubernetes Apps, or the big one, ZFS ARC.

If you take a look at this thread: Dragonfish swap usage/high memory/memory leak issues - #62 by kris

You’ll find the reason swap was disabled has nothing to do with the relative speed of devices. It was a perfect storm of unpredictable, weird problems. In the case of what @Steven_Ellis did, it looks like a sane use of swap.

In my own limited testing before 24.04.2, I noticed that my system would start swapping for no reason, even though I had free memory, even though I had set swappiness=1. Keep an eye on it.

[Steven_Ellis]

@NickF1227

In my own limited testing before 24.04.2, I noticed that my system would start swapping for no reason, even though I had free memory, even though I had set swappiness=1. Keep an eye on it.

Don’t always assume swap is a bad thing. Good virtual memory management means unused resources, for example a bunch of memory allocated to start up a service, gets swapped out when it is unused allowing more resources for new services or cache. I remember 15+ years ago watching traditional Unix admins freak out on Linux boxes because we were using some swap space, or that they thought no memory was “free” due to system cache.

[Steven_Ellis]

@Whattteva

No storage (not even NVMe) can match the raw performance of RAM especially in writes. This storage hierarchy (L1 > L2 > L3 > RAM > bulk storage has been true for decades and probably will continue to be true for at least another decade.

Don’t disagree. In fact for some of my smaller service VMs I use zram based swap. I prefer my services to be reliable and don’t want OOM to get in the way.

[Whattteva]

@NickF1227

@Whattteva Generally speaking, you’re not wrong. Your point that NVME is still an order of magnitude slower than RAM isn’t lost on me here. Nor I think on OP who’s tag line reads
Architect at Red Hat and long time Linux and Open Source user.
I think he’s aware.

I’m not saying it is lost on anyone. I started my post with my personal preference in my systems and the reply was mentioning that NVMe is a high performance storage. So I merely made a point that not even NVMe can compete in reads, let alone writes. If anyone wants to turn swap on, it’s perfectly fine and their choice. I’m in no way against that. I simply don’t do it on my systems that have a large enough system RAM capacity.

[But, to @Steven_Ellis’s point, the “pain” felt by swap is no where near as high when we’re talking about NVME. Just purely in terms of latency, NVME is an order of magnitude faster than spinning drives. We’re talking 100s of microseconds vs 10s of milliseconds.

I never even brought up spinners here. My original point still stands. Not even NVMe holds a candle in reads. I haven’t even mentioned sync writes, which all non-enterprise NVMe’s will start to crawl on. And unfortunately, the majority of home users that come to the forums use non-enterprise drives with no PLP and usually also slower QLC-based ones due to cost efficiency.

Regarding the issue at hand, the speed has nothing to do with the reason swap is off by default now.

If you take a look at this thread: Dragonfish swap usage/high memory/memory leak issues - #62 by kris

You’ll find the reason swap was disabled has nothing to do with the relative speed of devices. It was a perfect storm of unpredictable, weird problems. In the case of what @Steven_Ellis did, it looks like a sane use of swap.

Are we reading the same post here? The post even mentions the word performance like 2-3 times, which then lead to stability issues. It goes on to mention that the problem is compounded by people using crappy boot drives that wear out early and again, would lead to terrible performance due to the drive not having spare cells left. And as I stated above, majority of home users use these questionable type of drives. And it’s apparently prevalent enough that ixSystems decided it best to just disable swap altogether.