Does a pool of mirrored VDEVs benefit from an SLOG?

TrueNAS General
,
42

OK. Now that networking is figured out, I am back to pool and nfs performance testing.

I guess the main thing I’m after here is to have these results sanity checked and to get suggestions for improvement of pool and NFS performance. Do they make sense? Is my hardware and software working properly? Are my rationales for the results reasonable?

Again, this this pool is composed of 12x 4TB HGST 512n 7.2k SAS drives in 2 vdevs, each with 6-drives in raidz2. It’s in an Dell R720xd with 256G RAM, a LSI LSI SAS2308 (Dell H710p Mini Mono in IT mode) running at PCIe 3.0 speeds (x8 I think), and an Optane 900p PCIe card. Both servers are now running TrueNAS DragonFish 24.04 stable because I figured TrueNAS was probably more tuned for this sort of thing than plain vanilla Ubuntu Server 22.04.4. The network is a 2x10G bond that iperfs at about 19.9 Gbps. MTU of 9000 seemed to help iperf numbers some. Pretty sure just one 10G woulda given the same result.

In playing around, I noticed that when doing random reads and writes, that the size of the file your using matters A LOT. I guess its more seeking. So I wrote a script to run the same command with various file sizes. I ran the script on the server with and without an slog, and via an nfs mount from another server with and without an slog. The pool and the dataset have sync=standard.

The command I used is as follows. It’s a random read/write test with a 64k block size, iodepth=8 and numjobs=8. Maybe this simulates lots of VM storage traffic.

 fio --name=some_name --direct=1 --rw=randrw  --ioengine=libaio --runtime=60 --numjobs=8 --time_based --group_reporting --eta-newline=1 --end_fsync=1 --size=some_size --iodepth=8 --bs=64k

For each file size, the script calls sync, runs the fio test, and then runs the fio test again and records the result. It’s just a shell for loop.

Several things stick out:

  • On the server, the SLOG hurt more than it helped, but over NFS the SLOG helped tremendously. I’m guessing that despite specifying --end_fsync=1, that fio on the server didn’t sync all its writes. Maybe sync’d after a sequence of writes. But maybe NFS sync’d all the time. And maybe the reason the SLOG hurt is that caches writes in RAM are much faster than the Optane SLOG.
  • The server read and write times on the smaller file sizes seem way too fast for spinning disks. I’m guessing this is because of read and write caching in ram since I suspect many of the fio writes on the server were not sync writes.
  • The NFS performance did not change nearly as much as the server local performance did. I suspect that this is because NFS sync’s everything where the server fio didn’t. This would also explain why the SLOG helped NFS performance and didnt help (even hurt) server local performance.
  • Sometimes NFS performance exceeded server local performance. I’m not sure I can come up with as good guess as to why. Maybe the io’s were space out more? Maybe the queue depth was exhausted on the server local test due to fast RAM caching?
  • The random read and write numbers are almost identical in every case. No idea why. Guessing that everything is being cached in similarly fast ways.
  • I plan to do the test with some 25G NICs, but I doubt I’ll see any difference. One, I’m dealing with a limitation of sync writes to the pool, but it sorta seems like the network speed actually doesn’t matter much. It’s like the fact that it’s going over NFS matters more than the speed of the network. Anyone have blazing fast NFS?

Here’s the summary of the results in order of file sizes.

1M

  • Server Local:
    • read: IOPS=210k, BW=12.8GiB/s
    • write: IOPS=210k, BW=12.8GiB/s
  • Server Local (no SLOG):
    • read: IOPS=208k, BW=12.7GiB/s
    • write: IOPS=208k, BW=12.7GiB/s
  • NFS:
    • read: IOPS=3636, BW=227MiB/s
    • write: IOPS=3637, BW=227MiB/s
  • NFS (no SLOG):
    • read: IOPS=579, BW=36.2MiB/s
    • write: IOPS=588, BW=36.8MiB/s

10M

  • Server Local:
    • read: IOPS=129k, BW=8057MiB/s
    • write: IOPS=129k, BW=8055MiB/s
  • Server Local (no SLOG):
    • read: IOPS=168k, BW=10.3GiB/s
    • write: IOPS=168k, BW=10.3GiB/s
  • NFS:
    • read: IOPS=3698, BW=231MiB/s
    • write: IOPS=3698, BW=231MiB/s
  • NFS (no SLOG):
    • read: IOPS=555, BW=34.7MiB/s
    • write: IOPS=563, BW=35.2MiB/s

100M

  • Server Local:
    • read: IOPS=38.2k, BW=2389MiB/s
    • write: IOPS=38.3k, BW=2391MiB/s
  • Server Local (no SLOG):
    • read: IOPS=141k, BW=8825MiB/s
    • write: IOPS=141k, BW=8822MiB/s
  • NFS:
    • read: IOPS=3933, BW=246MiB/s
    • write: IOPS=3939, BW=246MiB/s
  • NFS (no SLOG):
    • read: IOPS=552, BW=34.6MiB/s
    • write: IOPS=561, BW=35.1MiB/s

1G

  • Server Local:
    • read: IOPS=8936, BW=559MiB/s
    • write: IOPS=8943, BW=559MiB/s
  • Server Local (no SLOG):
    • read: IOPS=7497, BW=469MiB/s
    • write: IOPS=7495, BW=468MiB/s
  • NFS:
    • read: IOPS=4525, BW=283MiB/s
    • write: IOPS=4522, BW=283MiB/s
  • NFS (no SLOG):
    • read: IOPS=540, BW=33.8MiB/s
    • write: IOPS=548, BW=34.3MiB/s

5G

  • Server Local:
    • read: IOPS=1432, BW=89.5MiB/s
    • write: IOPS=1429, BW=89.3MiB/s
  • Server Local (no SLOG):
    • read: IOPS=2372, BW=148MiB/s
    • write: IOPS=2373, BW=148MiB/s
  • NFS:
    • read: IOPS=2105, BW=132MiB/s
    • write: IOPS=2105, BW=132MiB/s
  • NFS (no SLOG):
    • read: IOPS=516, BW=32.3MiB/s
    • write: IOPS=524, BW=32.8MiB/s

10G

  • Server Local:
    • read: IOPS=1061, BW=66.3MiB/s
    • write: IOPS=1061, BW=66.4MiB/s
  • Server Local (no SLOG):
    • read: IOPS=1994, BW=125MiB/s
    • write: IOPS=1996, BW=125MiB/s
  • NFS:
    • read: IOPS=1211, BW=75.7MiB/s
    • write: IOPS=1211, BW=75.7MiB/s
  • NFS (no SLOG):
    • read: IOPS=531, BW=33.2MiB/s
    • write: IOPS=539, BW=33.7MiB/s

100G

  • Server Local:
    • read: IOPS=493, BW=30.8MiB/s
    • write: IOPS=500, BW=31.3MiB/s
  • Server Local (no SLOG):
    • read: IOPS=771, BW=48.2MiB/s
    • write: IOPS=777, BW=48.6MiB/s
  • NFS:
    • not tested
43

More comparison data points,

this is a Xeonv4 1650 box, 2xRaidZ2 (8 Sata drive) with a 900p slog.
Network is a CX3, 40GBs, test box is a vm on the TNC data store, multipath mounted to ESXi via NFS41.

READ: bw=182MiB/s (191MB/s), 182MiB/s-182MiB/s (191MB/s-191MB/s), io=10.7GiB (11.4GB), run=60016-60016msec WRITE: bw=182MiB/s (191MB/s), 182MiB/s-182MiB/s (191MB/s-191MB/s), io=10.7GiB (11.4GB), run=60016-60016msec

fio --name=random-read --direct=1 --rw=randrw --bs=64k --ioengine=libaio --iodepth=8 --runtime=60 --numjobs=8 --time_based --group_reporting --eta-newline=1 --end_fsync=1 --size=10G
random-read: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=8
…
fio-3.28
Starting 8 processes
random-read: Laying out IO file (1 file / 10240MiB)
random-read: Laying out IO file (1 file / 10240MiB)
random-read: Laying out IO file (1 file / 10240MiB)
random-read: Laying out IO file (1 file / 10240MiB)
random-read: Laying out IO file (1 file / 10240MiB)
random-read: Laying out IO file (1 file / 10240MiB)
random-read: Laying out IO file (1 file / 10240MiB)
random-read: Laying out IO file (1 file / 10240MiB)
Jobs: 8 (f=8): [m(8)][5.0%][r=145MiB/s,w=154MiB/s][r=2320,w=2471 IOPS][eta 00m:57s]
Jobs: 8 (f=8): [m(8)][8.3%][r=319MiB/s,w=319MiB/s][r=5109,w=5096 IOPS][eta 00m:55s]
Jobs: 8 (f=8): [m(8)][11.7%][r=336MiB/s,w=340MiB/s][r=5380,w=5447 IOPS][eta 00m:53s]
Jobs: 8 (f=8): [m(8)][15.0%][r=101MiB/s,w=105MiB/s][r=1608,w=1675 IOPS][eta 00m:51s]
Jobs: 8 (f=8): [m(8)][18.3%][r=48.0MiB/s,w=50.0MiB/s][r=767,w=799 IOPS][eta 00m:49s]
Jobs: 8 (f=8): [m(8)][21.7%][r=167MiB/s,w=166MiB/s][r=2674,w=2657 IOPS][eta 00m:47s]
Jobs: 8 (f=8): [m(8)][25.0%][r=258MiB/s,w=254MiB/s][r=4129,w=4063 IOPS][eta 00m:45s]
Jobs: 8 (f=8): [m(8)][28.3%][r=285MiB/s,w=279MiB/s][r=4553,w=4461 IOPS][eta 00m:43s]
Jobs: 8 (f=8): [m(8)][31.7%][r=126MiB/s,w=131MiB/s][r=2015,w=2094 IOPS][eta 00m:41s]
Jobs: 8 (f=8): [m(8)][35.0%][r=61.2MiB/s,w=56.0MiB/s][r=980,w=896 IOPS][eta 00m:39s]
Jobs: 8 (f=8): [m(8)][38.3%][r=139MiB/s,w=135MiB/s][r=2217,w=2154 IOPS][eta 00m:37s]
Jobs: 8 (f=8): [m(8)][41.7%][r=277MiB/s,w=269MiB/s][r=4438,w=4299 IOPS][eta 00m:35s]
Jobs: 8 (f=8): [m(8)][45.0%][r=276MiB/s,w=278MiB/s][r=4415,w=4454 IOPS][eta 00m:33s]
Jobs: 8 (f=8): [m(8)][48.3%][r=151MiB/s,w=148MiB/s][r=2423,w=2370 IOPS][eta 00m:31s]
Jobs: 8 (f=8): [m(8)][51.7%][r=55.0MiB/s,w=59.3MiB/s][r=879,w=948 IOPS][eta 00m:29s]
Jobs: 8 (f=8): [m(8)][55.0%][r=34.4MiB/s,w=34.4MiB/s][r=550,w=550 IOPS][eta 00m:27s]
Jobs: 8 (f=8): [m(8)][58.3%][r=144MiB/s,w=142MiB/s][r=2296,w=2273 IOPS][eta 00m:25s]
Jobs: 8 (f=8): [m(8)][61.7%][r=246MiB/s,w=255MiB/s][r=3939,w=4074 IOPS][eta 00m:23s]
Jobs: 8 (f=8): [m(8)][65.0%][r=300MiB/s,w=304MiB/s][r=4802,w=4864 IOPS][eta 00m:21s]
Jobs: 8 (f=8): [m(8)][68.3%][r=132MiB/s,w=131MiB/s][r=2109,w=2095 IOPS][eta 00m:19s]
Jobs: 8 (f=8): [m(8)][71.7%][r=56.1MiB/s,w=56.0MiB/s][r=897,w=895 IOPS][eta 00m:17s]
Jobs: 8 (f=8): [m(8)][75.0%][r=158MiB/s,w=159MiB/s][r=2524,w=2544 IOPS][eta 00m:15s]
Jobs: 8 (f=8): [m(8)][78.3%][r=256MiB/s,w=260MiB/s][r=4089,w=4163 IOPS][eta 00m:13s]
Jobs: 8 (f=8): [m(8)][81.7%][r=283MiB/s,w=282MiB/s][r=4527,w=4512 IOPS][eta 00m:11s]
Jobs: 8 (f=8): [m(8)][85.0%][r=141MiB/s,w=136MiB/s][r=2260,w=2180 IOPS][eta 00m:09s]
Jobs: 8 (f=8): [m(8)][88.3%][r=58.0MiB/s,w=57.1MiB/s][r=927,w=913 IOPS][eta 00m:07s]
Jobs: 8 (f=8): [m(8)][91.7%][r=164MiB/s,w=150MiB/s][r=2618,w=2395 IOPS][eta 00m:05s]
Jobs: 8 (f=8): [m(8)][95.0%][r=290MiB/s,w=288MiB/s][r=4643,w=4611 IOPS][eta 00m:03s]
Jobs: 8 (f=8): [m(8)][98.3%][r=268MiB/s,w=265MiB/s][r=4291,w=4233 IOPS][eta 00m:01s]
Jobs: 8 (f=8): [m(8)][100.0%][r=251MiB/s,w=255MiB/s][r=4019,w=4072 IOPS][eta 00m:00s]
random-read: (groupid=0, jobs=8): err= 0: pid=1382: Sun May 5 10:23:09 2024
read: IOPS=2910, BW=182MiB/s (191MB/s)(10.7GiB/60016msec)
slat (usec): min=3, max=37552, avg=14.58, stdev=268.46
clat (usec): min=5, max=76909, avg=9614.01, stdev=7297.66
lat (usec): min=263, max=76920, avg=9628.72, stdev=7299.62
clat percentiles (usec):
| 1.00th=[ 3916], 5.00th=[ 4621], 10.00th=[ 5014], 20.00th=[ 5604],
| 30.00th=[ 6063], 40.00th=[ 6521], 50.00th=[ 6980], 60.00th=[ 7701],
| 70.00th=[ 8717], 80.00th=[11338], 90.00th=[17957], 95.00th=[25560],
| 99.00th=[40109], 99.50th=[45351], 99.90th=[58983], 99.95th=[62129],
| 99.99th=[68682]
bw ( KiB/s): min=31744, max=372352, per=99.90%, avg=186060.24, stdev=12322.40, samples=952
iops : min= 496, max= 5818, avg=2907.18, stdev=192.54, samples=952
write: IOPS=2910, BW=182MiB/s (191MB/s)(10.7GiB/60016msec); 0 zone resets
slat (usec): min=4, max=41938, avg=18.01, stdev=342.60
clat (usec): min=808, max=95661, avg=12337.24, stdev=9220.62
lat (usec): min=1259, max=95674, avg=12355.38, stdev=9227.06
clat percentiles (usec):
| 1.00th=[ 5276], 5.00th=[ 6128], 10.00th=[ 6587], 20.00th=[ 7242],
| 30.00th=[ 7767], 40.00th=[ 8356], 50.00th=[ 8979], 60.00th=[ 9634],
| 70.00th=[11076], 80.00th=[14746], 90.00th=[23462], 95.00th=[32637],
| 99.00th=[50070], 99.50th=[57410], 99.90th=[72877], 99.95th=[76022],
| 99.99th=[82314]
bw ( KiB/s): min=31744, max=378880, per=99.89%, avg=186079.50, stdev=12257.71, samples=952
iops : min= 496, max= 5920, avg=2907.48, stdev=191.53, samples=952
lat (usec) : 10=0.01%, 20=0.01%, 100=0.01%, 250=0.01%, 500=0.01%
lat (usec) : 750=0.01%, 1000=0.01%
lat (msec) : 2=0.02%, 4=0.62%, 10=68.87%, 20=19.89%, 50=9.90%
lat (msec) : 100=0.68%
cpu : usr=0.40%, sys=0.97%, ctx=253890, majf=0, minf=128
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=174655,174692,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
READ: bw=182MiB/s (191MB/s), 182MiB/s-182MiB/s (191MB/s-191MB/s), io=10.7GiB (11.4GB), run=60016-60016msec
WRITE: bw=182MiB/s (191MB/s), 182MiB/s-182MiB/s (191MB/s-191MB/s), io=10.7GiB (11.4GB), run=60016-60016msec

Disk stats (read/write):
sdb: ios=174248/174307, merge=0/22, ticks=1668373/2141371, in_queue=3809743, util=99.90%

You see that its pretty similar to what you get, 200 MB/s

Here is a write throughput of the slog, first peak is moving the vm to the datastore, second is the test. I think its double what I see due to multipathing.

image
image1220×228 14.2 KB

44

Thanks for the comparison data. I appreciate the help.

What’s your pool again? You said raidz2 mirror? It is sync=standard?

Yeah. It’s not fair. :cry: If you got super fast network, seems like you oughta be able to get the performance you see if you do the same test on the server.

Seems like NFS is turning sync=standard to something close to sync=always.

I’m gonna experiement some more with NFS mount params sync and async. I definately don’t want sync=disabled behavior, but seems like I can watch the the SLOG to see if it behaves like sync=standard (i.e., sync when app tells it to – calls sync(), fsync(), or close() ).

And I want to figure out how to make fio sync on every write. Maybe --sync=1.

45

Its 2 raidz2 vdevs, so a stripe. sync is always as thats what ESXi is enforcing anyway when using NFS.

Lol, I got 100G networking and the most i was able to get was some 3GB/s.
I got more when using iWarp a few years back, which is why I hope that RDMA would help (as i mentioned).

With nfs the slog is the most important aspect. 3GB/s was reached with moving multiple VMs at the same time (ie multiple threads) to an NVDimm (-N) backed NVME pool. Slower nvme’s but good enough.
Unfortunatly that box is down atm or I would have provided some comparison metrics from that too (or with the single Z2 spinner dev I also had in there).

46

That’s sad. This needs to start making sense to me. I know your pool has to be performant but it seems that given sufficent network that your NFS performance should be close to as performant as your pool.

So I dont get why there are so many sync writes with NFS. Looks like NFS async is the default and it still acts just like a sync=always pool. Seems like if you could get NFS to not sync so much you could better utilize your super fast network.

Why such disparity between fio run on server and fio run on NFS?!

Pardon my storage newbieness. I know you guys have been down all these trails before. NFS daemons, TCP buffers, etc. Just seems unfathomable there would be such a disparity.

47

Its all about latency.
Thats why a “fast” nvme does not necessarily make a good slog, those with ultra low latency (to ack the write) really shine here.

Network adds a lot to latency since each write now needs to run through the network first to get ack’ed and then the ack needs to get back before the write is deemed secure.

I never played with async NFS since I always used ESXi and that forces sync on NFS to make sure the write is secure and the VMs have a consistent state even if storage (or the VM) fails.

2 Likes
48

This is not an inherent property of NFS but the way a particular product - namely ESXi - uses NFS.

Reason being that there are a ton of VMs on that host who each run their own operating system. Each of these OSs thinks it’s talking to a real disk drive when in reality there is only a VMDK file on some suitable storage.

Now each of these guest OSs also implements it’s own caching for disk access, so whenever it writes anything to its virtual disk, that is probably a synchronous write. Because data integrity, order of writes, etc. etc. The asynchronous cached write has already happened in the guest OS. Whenever the guest OS flushes anything to disk it insists that data has actually hit stable storage before it considers the task done.

Now if ESXi would lie to the guest OS and implement its own caching strategy, that would certainly lead to disaster, if e.g. the power fails. ESXi has no idea about what filesystem is at work in the guest and what the order of writes needs to be to guarantee consistency apart from performing a synchronous write operation whenever the guest demands a synchronous write operation. Which is the large majority, because the guest implements its own caching already and what must finally go to disk must go to disk and not to a queue in another lower layer.

I repeated the same point - I hope from sufficiently different angles - to make clear what is happening here.

HTH,
Patrick

2 Likes
49

Yeah. That’s what I figured in the case of VMs. I just can’t get a regular plain vanilla NFS mount to act like sync=standard. Looks like it wants to sync everything. Looks like NFS option async is the default, but I want to test it. Both client and server are Truenas, so not sure about defaults yet. /proc/mounts doesnt say one way or the other. Thanks for bearing with me as I learn.

50

OK. Here’s the same random read/write tests with a the pool reconfigured to 6x 2-drive mirror vdevs. Used to be 2x 6-drive raidz2 vdevs. Both tests were using sync=standard on the pool. 10G network.

So, looks like the mirror performance was about the same until it started getting to larger files (5G and above). This might be because the server has 256G ram and a lot is cached? But with 5GB files and larger the performance increase of the mirrors is astounding. Even NFS was faster and got up to nearly 300 MB/s on the larger files.

But all-in-all, nothing to write home about unless you are using huge files.

The % faster/slower is mirror pool test results compared to previous test results with raidz2 pool.

1M

  • Server Local:
    • read: IOPS=221k, BW=13.5GiB/s (14.5GB/s) (5.5% faster)
    • write: IOPS=221k, BW=13.5GiB/s (14.5GB/s) (5.5% faster)
  • NFS:
    • read: IOPS=3605, BW=225MiB/s (236MB/s) (1% slower)
    • write: IOPS=3608, BW=226MiB/s (236MB/s) (1% slower)

10M

  • Server Local:
    • read: IOPS=131k, BW=8217MiB/s (8616MB/s) (2% faster)
    • write: IOPS=131k, BW=8215MiB/s (8614MB/s) (2% faster)
  • NFS:
    • read: IOPS=3622, BW=226MiB/s (237MB/s) (2% slower)
    • write: IOPS=3624, BW=227MiB/s (238MB/s) (2% slower)

100M

  • Server Local:
    • read: IOPS=35.6k, BW=2226MiB/s (2334MB/s) (7% slower)
    • write: IOPS=35.6k, BW=2228MiB/s (2336MB/s) (7% slower)
  • NFS:
    • read: IOPS=3887, BW=243MiB/s (255MB/s) (1% slower)
    • write: IOPS=3893, BW=243MiB/s (255MB/s) (1% slower)

1G

  • Server Local:
    • read: IOPS=8144, BW=509MiB/s (534MB/s) (9% slower)
    • write: IOPS=8143, BW=509MiB/s (534MB/s) (9% slower)
  • NFS:
    • read: IOPS=4513, BW=282MiB/s (296MB/s) (0.5% slower)
    • write: IOPS=4510, BW=282MiB/s (296MB/s) (0.5% slower)

5G

  • Server Local:
    • read: IOPS=6533, BW=408MiB/s (428MB/s) (456% faster)
    • write: IOPS=6533, BW=408MiB/s (428MB/s) (457% faster)
  • NFS:
    • read: IOPS=2975, BW=186MiB/s (195MB/s) (48% faster)
    • write: IOPS=2975, BW=186MiB/s (195MB/s) (48% faster)

10G

  • Server Local:
    • read: IOPS=6556, BW=410MiB/s (430MB/s) (618% faster)
    • write: IOPS=6553, BW=410MiB/s (429MB/s) (617% faster)
  • NFS:
    • read: IOPS=4493, BW=281MiB/s (295MB/s) (371% faster)
    • write: IOPS=4489, BW=281MiB/s (294MB/s) (388% faster)
51

So, I think I have a change of plans based on my testing and learning.

I was gonna use the 8x 4TB drives on the XCP-ng server for a virtualized TrueNAS backup server VM (passing through the controller). Then ZFS replicate from main TrueNAS to backupt TrueNAS so I’d have two copies of the data.

But since I doubt I’ll ever have a second XCP-ng server for HA, and given the lackluster NFS performance (due to HDD performance and not being able to avoid sync writes over NFS – not blaming TrueNAS or network speeds), I’m considering using the drives on the XCP-ng server to hold the VMs themselves. I can always choose to store some VMs on TrueNAS over NFS if I want. And I can store app data on TrueNAS over NFS too like I am now. And I would still backup the VMs to TrueNAS over NFS. And I can store ISOs on TrueNAS. But the VMs should be much more snappy on local storage.

I guess I’ll use the H710p Mini controller on the XCP-ng server as an actual RAID controller with its 1GB battery-backed write cache. It’s the D1 version which means it’ll run at PCIe 3.0 speeds (might not matter with HDDs tho). Will probably use a fast RAID-10 configuration and just backup everything up on TrueNAS. XO would run as a VM on TrueNAS SCALE and store its config on TrueNAS.

The virtualized TrueNAS backup VM sorta gave me the willies anyway.

Sound reasonable?

52

You could try to see how a virtualized TrueNas would work? (With a HBA passed through) - this might alleviate the network problems.

You could then use the second box as backup.

53

Well… I don’t think I have network problems. I think I have NFS sync write problems.

54

Did one final test with a 12x 1-drive stripe vdevs. Crazy, I know, but I wanted to make the fastest pool I could to see if NFS write performance would increase.

So, the server local performance of the stripe pool was almost double that of the mirror pool (which was about the same as the raidz2 pool, except yhe mirror was much faster on files sizes >= 5GB).

But NFS random IO performance was pretty much the same on all pools, including the superfast stripe, except that the mirror pool was quite a bit faster with files 5G and larger. Not really touching the 10G network tho on writes.

Sequential read speeds are very fast over NFS tho (like over 1 GB/s), so I know that NFS can be fast. Sequential writes on the server were just shy of 14 GB/s against the super fast stripe pool, but over NFS seq write were still about 250MB/s – same ballpark as all the other pools. :man_shrugging:

Looks like NFS+HDDs just isnt very exciting.

55

try nfs with sync=disabled, thats what nfs would be really capable off if the slog did not matter.
If thats way faster, another slog might help.

56

Yikes! Fabulous suggestion! So, NFS to the crazy 12x stripe pool with sync=disabled was about twice as fast. I turned encryption off too, but it didnt seem to matter much. Tested with encryption on and off.

Even with sync=disabled there was lots of SLOG activity. Much more than with sync=standard. I guess it’s gotta write sometime, but it still looks like it was writing more often with NFS.

strip_async_noenc_slog_graph
strip_async_noenc_slog_graph2209×494 74.2 KB

During the server local test, I saw SLOG spike to about 1.5 GB/s writes. You really think adding another 900p would help? Stripe or mirror? Since it’s Optane, and I’m after SLOG speed, I’m thinking stripe.

I’m just thinking that the SLOG was used more because more writes were getting through for some reason. If sync=standard, it just can’t use the SLOG as much. In this test, the NFS throughput is about the same as the SLOG throughput. In the stripe with sync=standard, the pool itself was much faster than the NFS throughput. Maybe something else is the bottleneck and not the SLOG.

:man_shrugging:

Results by fio test file size:

1M
Server Local:

  • read: IOPS=245k, BW=14.9GiB/s (16.0GB/s)
  • write: IOPS=245k, BW=14.9GiB/s (16.0GB/s)

NFS:

  • read: IOPS=7823, BW=489MiB/s (513MB/s)
  • write: IOPS=7822, BW=489MiB/s (513MB/s)

10M
Server Local:

  • read: IOPS=164k, BW=9.99GiB/s (10.7GB/s)
  • write: IOPS=164k, BW=9.99GiB/s (10.7GB/s)

NFS:

  • read: IOPS=7640, BW=478MiB/s (501MB/s)
  • write: IOPS=7636, BW=477MiB/s (500MB/s)

100M
Server Local:

  • read: IOPS=70.2k, BW=4386MiB/s (4599MB/s)
  • write: IOPS=70.2k, BW=4389MiB/s (4602MB/s)

NFS:

  • read: IOPS=7779, BW=486MiB/s (510MB/s)
  • write: IOPS=7778, BW=486MiB/s (510MB/s)

1G
Server Local:

  • read: IOPS=17.0k, BW=1060MiB/s (1111MB/s)
  • write: IOPS=17.0k, BW=1061MiB/s (1112MB/s)

NFS:

  • read: IOPS=7754, BW=485MiB/s (508MB/s)
  • write: IOPS=7752, BW=485MiB/s (508MB/s)

5G
Server Local:

  • read: IOPS=14.8k, BW=923MiB/s (967MB/s)
  • write: IOPS=14.8k, BW=923MiB/s (968MB/s)

NFS:

  • read: IOPS=7686, BW=480MiB/s (504MB/s)
  • write: IOPS=7682, BW=480MiB/s (504MB/s)

10G
Server Local:

  • read: IOPS=14.5k, BW=904MiB/s (948MB/s)
  • write: IOPS=14.5k, BW=904MiB/s (948MB/s)

NFS:

  • read: IOPS=7611, BW=476MiB/s (499MB/s)
  • write: IOPS=7606, BW=475MiB/s (498MB/s)
57

When I tested iSCSI vs NFS, I found iSCSI performed about twice the speed of NFS…

I was using >1G files.

There is a setting for how much data is allowed to be outstanding before it is flushed to disk.

OpenZFS limits the amount of dirty data on the system according to the tunable zfs_dirty_data_max. It’s default value is 10% of memory up to 4GB .

Tuning the OpenZFS write throttle.

I suspect your Local results are approaching the iSCSI results, once your Dirty Data limit is exceeded and results have to be flushed (ie synced) to disk.

Meanwhile, many NFS clients will sync writes…

If sync perf is important, get a good slog. I use a p4801x in my server. A pair of them would be even better.

And yes, I think disabling sync is not the best way to see what a SLOG could gain you… but testing the benefits with a ram disk… is.

In that thread I mention vfs.zfs.dirty_data_sync

58

Wow! That’s an awesome and cheap test. I’ll give it a whirl. 15G or so probly enough.

59

Instructions for creating a memory device on scale are different to core, but possible.

1 Like
60

Well, I did it. Made a 20G ramdisk. Added it to the pool via command line. Didnt help. Truenas SCALE didnt use it much. If I removed the Optane, NFS performance was horrible. Dunno why it didnt like the ramdisk.

Getting nearly 10G wirespeed sequential reads over NFS tho.

Gonna do more testing. Have some Mellanox 25G CX4 cards to play with. Gonna see how fast reads can get on that. Then I’ll probably move on from this NFS testing.

61

Strange. Wonder if it knows the disk is not a good slog