Using RAM to cache ALL writes (random or sequential)

TrueNAS General
1

I am using SCALE 24.04.
I have a server with 16x NVMe drives and 512GB of ram.
Based on some previous guidance we configured our pool (name = nvme) to the following:

zfs set recordsize=1M nvme
zfs set primarycache=metadata nvme

However, based on some internal discussions we’d like to configure truenas to use RAM to cache ALL writes (random and sequential) and both sync and async writes. I recall there are a few different options in the way that zfs will acknowledge writes.

one option is for the write to be sent to ram AND the drive, and the write isn’t acknowledged until the drive itself confirms the write, another option is that as soon as the write hits RAM the write is acknowledged.

Now I understand that the second option is risky, but for our workloads, we would like to configure the ZFS pool to operate in this manner.

How might we insure that ALL writes sync and async (random or sequential) get cached in ram first, the write is immediately acknowledged and then written to the NVMe disks.

btw, because we have superfast NVMe drives in a 2x8 RAIDZ1 configuration we don’t use any dedicated slog or l2arc. We just want the writes going to ram then directly to the nvme’s. which are plenty fast to keep up with sustained writes in the GB/sec.

2

Hey @cyr0nk0r

What you’re asking for might be the default and current behavior, depending on what the clients are that you’re using to write to the TrueNAS system. All writes are “sent to RAM” - but synchronous writes will also be sent to the drives.

Where did you receive the guidance to set primarycache=metadata for your system?

1 Like
3

My guess would be he got it from the videos of a certain canadian youtuber who is known to loose all his archival Data…

1 Like
4

You must not use the Name That Is Not To Be Invoked!
But, you do seem to know that curse well, so you should be okay.

The penalties are worse than the Release That Shall Not Be Named!

1 Like
5

No, I got them from other members discussing the same sorts of issues with NVMe here on the truenas forums.

which are echo’d here:

6

For the record I am the author of that post.

Clarification:
Setting metadata only for ARC was meant in the context for something to try as part of troubleshooting steps and not necessarily a general recommendation for production.

There’s a reason why I said “try” in both linked posts. :slight_smile:

Depending on the workload it may help but it also will hurt in other ways.

1 Like