Zpool sync causing extream slow writing speed (10x) on both mirrored and RAIDZ1 drive

TrueNAS General
,
1

Updated from CORE to SCALE 25.10.3, noticed significant slow SMB writing speed in both SSD pool and HDD pool while read speeds are normal. Switching off Sync speeds up writing more than 10 times (42.9MiB/s vs 644MiB/s). Is this considered normal? And why there aren’t such speed penalties in CORE SMB?

root@nas[/mnt/Misc/Cache]# fio --name=write_test --ioengine=posixaio --rw=write --bs=1m --size=10g --iodepth=32 --numjobs=1 --nrfiles=1 --directory=./ --sync=1
write_test: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=posixaio, iodepth=32
fio-3.33
Starting 1 process
write_test: Laying out IO file (1 file / 10240MiB)
Jobs: 1 (f=1): [W(1)][100.0%][w=386MiB/s][w=386 IOPS][eta 00m:00s]
write_test: (groupid=0, jobs=1): err= 0: pid=14444: Thu Apr 16 03:34:43 2026
write: IOPS=643, BW=644MiB/s (675MB/s)(10.0GiB/15907msec); 0 zone resets
slat (usec): min=6, max=480, avg=56.03, stdev=23.07
clat (usec): min=1583, max=98538, avg=49067.20, stdev=31519.68
lat (usec): min=1598, max=98606, avg=49123.23, stdev=31512.66
clat percentiles (usec):
|  1.00th=[ 8225],  5.00th=[ 9372], 10.00th=[10159], 20.00th=[11731],
| 30.00th=[14484], 40.00th=[28705], 50.00th=[60031], 60.00th=[79168],
| 70.00th=[80217], 80.00th=[80217], 90.00th=[81265], 95.00th=[81265],
| 99.00th=[82314], 99.50th=[87557], 99.90th=[98042], 99.95th=[98042],
| 99.99th=[98042]
bw (  KiB/s): min=393216, max=2883584, per=100.00%, avg=665970.94, stdev=655445.42, samples=31
iops        : min=  384, max= 2816, avg=650.35, stdev=640.09, samples=31
lat (msec)   : 2=0.05%, 10=8.96%, 20=25.70%, 50=12.48%, 100=52.81%
cpu          : usr=3.84%, sys=0.10%, ctx=2129, majf=0, minf=772
IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=22.9%, 16=54.7%, 32=22.3%, >=64=0.0%
submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete  : 0=0.0%, 4=97.4%, 8=0.1%, 16=0.0%, 32=2.6%, 64=0.0%, >=64=0.0%
issued rwts: total=0,10240,0,0 short=0,0,0,0 dropped=0,0,0,0
latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
WRITE: bw=644MiB/s (675MB/s), 644MiB/s-644MiB/s (675MB/s-675MB/s), io=10.0GiB (10.7GB), run=15907-15907msec

root@nas[/mnt/Misc/Cache]# fio --name=write_test --ioengine=posixaio --rw=write --bs=1m --size=10g --iodepth=32 --numjobs=1 --nrfiles=1 --directory=./ --sync=1
write_test: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=posixaio, iodepth=32
fio-3.33
Starting 1 process
Jobs: 1 (f=1): [W(1)][99.6%][w=32.0MiB/s][w=32 IOPS][eta 00m:01s]
write_test: (groupid=0, jobs=1): err= 0: pid=15255: Thu Apr 16 03:38:58 2026
write: IOPS=42, BW=42.9MiB/s (45.0MB/s)(10.0GiB/238850msec); 0 zone resets
slat (usec): min=8, max=722, avg=73.93, stdev=24.51
clat (msec): min=248, max=1174, avg=745.16, stdev=100.01
lat (msec): min=248, max=1174, avg=745.24, stdev=100.01
clat percentiles (msec):
|  1.00th=[  409],  5.00th=[  567], 10.00th=[  667], 20.00th=[  709],
| 30.00th=[  735], 40.00th=[  751], 50.00th=[  760], 60.00th=[  768],
| 70.00th=[  776], 80.00th=[  793], 90.00th=[  802], 95.00th=[  827],
| 99.00th=[ 1099], 99.50th=[ 1116], 99.90th=[ 1167], 99.95th=[ 1167],
| 99.99th=[ 1167]
bw (  KiB/s): min= 2048, max=126976, per=100.00%, avg=52016.60, stdev=23445.37, samples=402
iops        : min=    2, max=  124, avg=50.78, stdev=22.89, samples=402
lat (msec)   : 250=0.24%, 500=3.51%, 750=40.19%, 1000=53.84%, 2000=2.23%
cpu          : usr=0.36%, sys=0.03%, ctx=2562, majf=0, minf=1279
IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=25.0%, 16=50.0%, 32=24.9%, >=64=0.0%
submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete  : 0=0.0%, 4=97.5%, 8=0.1%, 16=0.0%, 32=2.5%, 64=0.0%, >=64=0.0%
issued rwts: total=0,10240,0,0 short=0,0,0,0 dropped=0,0,0,0
latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
WRITE: bw=42.9MiB/s (45.0MB/s), 42.9MiB/s-42.9MiB/s (45.0MB/s-45.0MB/s), io=10.0GiB (10.7GB), run=238850-238850msec
2

And while switching Sync from STANDARD to DISABLED increases fio test speed, SMB write speed still seems to be the same ~30MiB/s. But NFS share write speeds are all fast and normal.

I’m getting confused.

3

Yes.
Carefully consider whether your workload requires sync writes. If so, read into adding a SLOG.

4

I just enabled sync on my own TrueNAS Core machine and ran a test (actual file copy from an SSD pool to an HDD pool, on the same system). I’m seeing the same massive drop in write performance you described (in my case about 10x, maybe even up to 12x slower). So I don’t think SMB has anything to do with this.

Are you sure that sync was enabled on your TrueNAS Core machine?

6

Not so certain. But I’m sure that pool settings during upgrade are unchanged, just pure export and import. Maybe standard sync acts differently in SMB between CORE and SCALE.

7

I got a UPS attached, is it safe to turn off all sync settings?

8

Sync is not related to having a UPS or not…
Sync is related to workload.
Would a failed write corrupt a file system? (E.g. Virtual Machine) Sync is useful.
Would a failed write result in civil liability? (Your NAS holds a banking/trading database…) Sync is mandatory.
Can transactions be replayed? (Copying files from your desktop over SMB: If it is interrupted, do it anew.) You do not need sync.

2 Likes
9

Got it. Thanks for helping.