Windows 11 24H2 with Windows Defender forces sync writes over SMB shares, which results in very low performance on small files

TrueNAS General
, ,
1

When I copy small files (images from 10kB to 150kB) from a Windows 11 24H2 client with build in Windows Defender virus scanner active to an SMB share on TrueNAS Scale, I only get a transfer rate lower than 1MB/s.

The TrueNAS Server runs on an PowerEdge T20 with Xeon E3-1225v3 and 32GB ECC RAM. The pool tank1 consits of two mirrored SATA 3TB WD Red HDDs. During copying, continuous access of the two SATA HDDs can be heard. It looks like there is an ongoing write access for the metadata.

  • When I deactivate Windows Defender, there’s only an disk access every 5 sec like expected.
  • Copying with robocopy and Windows defender on: Disk access every 5 sec.
  • I have cross tested this with Windows 10 22H2 and Windows Defender on: Disk access every 5 sec.

All these is as expected and the performance of the copy job is much higher (20MB/s with robocopy with /MT:32) and is limited by the client.

What I have done:

  • I have tried this under TrueNAS Scale 24.04, 24.10 and 25.10-BETA.1 - Goldeye
  • The pool tank1 and the dataset tank1\dataset2 is configured with SYNC=Disabled.
  • smbd4.conf has custom option strict sync=no set over the CLI.
  • Extended the pool with an external SLOG device (Enterprise SATA SSD Samsung SM883): No write access can be seen.

But none of that helped.

With these setting zfs should cache every write access for 5 sec, but Windows Defender under Windows 11 24H2 destroys this.

What is going on here?

Please can someone replicate this effect?

Here are my configs:

root@lagerhaus[~]# zpool status tank1
  pool: tank1
 state: ONLINE
config:
        NAME                                      STATE     READ WRITE CKSUM
        tank1                                     ONLINE       0     0     0
          mirror-0                                ONLINE       0     0     0
            712e72ca-db3c-48bc-93e4-a890615fb14f  ONLINE       0     0     0
            d4fbe796-620f-4388-904f-cbbd149ff880  ONLINE       0     0     0
errors: No known data errors

smb4.cnf

# Global parameters
[global]
	bind interfaces only = Yes
	disable spoolss = Yes
	dns proxy = No
	load printers = No
	logging = file
	max log size = 5120
	min receivefile size = 16384
	passdb backend = tdbsam:/var/run/samba-cache/private/passdb.tdb
	printcap name = /dev/null
	registry shares = Yes
	restrict anonymous = 2
	server multi channel support = No
	server string = TrueNAS Server
	winbind request timeout = 2
	workgroup = PARADIES
	zfs_core:zfs_block_cloning = False
	zfs_core:zfs_integrity_streams = False
	idmap config * : read only = True
	idmap config * : range = 90000001 - 90010001
	rpc_server:mdssvc = disabled
	rpc_daemon:mdssd = disabled
	fruit:zero_file_id = False
	fruit:nfs_aces = False
	idmap config * : backend = tdb
	create mask = 0664
	directory mask = 0775
	strict sync = No

[smbshare]
	ea support = No
	path = /mnt/tank1/dataset2
	posix locking = No
	read only = No
	smbd max xattr size = 2097152
	vfs objects = streams_xattr shadow_copy_zfs ixnas zfs_core io_uring
	fruit:resource = stream
	fruit:metadata = stream

zpool iostat -v tank1 1

                                            capacity     operations     bandwidth
pool                                      alloc   free   read  write   read  write
----------------------------------------  -----  -----  -----  -----  -----  -----
tank1                                     2.13G   458G      0    377      0  4.46M
  mirror-0                                2.13G   458G      0    377      0  4.46M
    712e72ca-db3c-48bc-93e4-a890615fb14f      -      -      0    189      0  2.23M
    d4fbe796-620f-4388-904f-cbbd149ff880      -      -      0    187      0  2.23M
----------------------------------------  -----  -----  -----  -----  -----  -----
                                            capacity     operations     bandwidth
pool                                      alloc   free   read  write   read  write
----------------------------------------  -----  -----  -----  -----  -----  -----
tank1                                     2.13G   458G      0    475      0  5.02M
  mirror-0                                2.13G   458G      0    475      0  5.02M
    712e72ca-db3c-48bc-93e4-a890615fb14f      -      -      0    240      0  2.51M
    d4fbe796-620f-4388-904f-cbbd149ff880      -      -      0    235      0  2.51M
----------------------------------------  -----  -----  -----  -----  -----  -----
                                            capacity     operations     bandwidth
pool                                      alloc   free   read  write   read  write
----------------------------------------  -----  -----  -----  -----  -----  -----
tank1                                     2.13G   458G      0    408      0  5.28M
  mirror-0                                2.13G   458G      0    408      0  5.28M
    712e72ca-db3c-48bc-93e4-a890615fb14f      -      -      0    203      0  2.64M
    d4fbe796-620f-4388-904f-cbbd149ff880      -      -      0    204      0  2.64M
----------------------------------------  -----  -----  -----  -----  -----  -----
                                            capacity     operations     bandwidth
pool                                      alloc   free   read  write   read  write
----------------------------------------  -----  -----  -----  -----  -----  -----
tank1                                     2.13G   458G      0    479      0  6.42M
  mirror-0                                2.13G   458G      0    479      0  6.42M
    712e72ca-db3c-48bc-93e4-a890615fb14f      -      -      0    240      0  3.21M
    d4fbe796-620f-4388-904f-cbbd149ff880      -      -      0    239      0  3.21M
----------------------------------------  -----  -----  -----  -----  -----  -----
                                            capacity     operations     bandwidth
pool                                      alloc   free   read  write   read  write
----------------------------------------  -----  -----  -----  -----  -----  -----
tank1                                     2.13G   458G      0    458      0  6.30M
  mirror-0                                2.13G   458G      0    458      0  6.30M
    712e72ca-db3c-48bc-93e4-a890615fb14f      -      -      0    231      0  3.15M
    d4fbe796-620f-4388-904f-cbbd149ff880      -      -      0    227      0  3.15M

zilstat -i 1

root@lagerhaus[~]# zilstat -i 1
    time   cc/s   ic/s  idc/s  idb/s  iic/s  iib/s  imnc/s  imnw/s  imsc/s  imsw/s
14:26:11      0      0      0      0      0      0       0       0       0       0
14:26:12      0      0      0      0      0      0       0       0       0       0
14:26:13      0      0      0      0      0      0       0       0       0       0
14:26:14      0      0      0      0      0      0       0       0       0       0
14:26:15      0      0      0      0      0      0       0       0       0       0
14:26:16      0      0      0      0      0      0       0       0       0       0
14:26:17      0      0      0      0      0      0       0       0       0       0
14:26:18      0      0      0      0      0      0       0       0       0       0
14:26:19      0      0      0      0      0      0       0       0       0       0
14:26:20      0      0      0      0      0      0       0       0       0       0
14:26:21      0      0      0      0      0      0       0       0       0       0
14:26:22      0      0      0      0      0      0       0       0       0       0
14:26:23      0      0      0      0      0      0       0       0       0       0
14:26:24      0      0      0      0      0      0       0       0       0       0
14:26:25      0      0      0      0      0      0       0       0       0       0
14:26:26      0      0      0      0      0      0       0       0       0       0
14:26:27      0      0      0      0      0      0       0       0       0       0

Best regards
Josef

2

I thought Auxiliary Parameters were disabled to prevent tinkering? My mind is blown right now. :exploding_head:

3

There is no GUI for that. You have to use the CLI.

From post #48 of Auxiliary Parameters missed

You actually don’t have to do that. You can change settings through CLI
If you want to edit params for a particular share, first (via SSH) run:
cli
to enter CLI interface
then, run
sharing smb query
to get a list of shares. Note the one you want to edit and look at its id .
Then, run
sharing smb update id
where id is the id number of your share.
You will drop into a nano-like text editor, where you remove # to uncomment the aux conf section, and input your smb config, separated by newlines if there are multiple, like so:

…

Then you click save and quit and it should apply.
And for global options, also while in CLI mode, you run
service smb update smb_options=“”
where you put whatever options you desire inside “”, separated by \n. For example:
service smb update smb_options=“server min protocol = SMB3_11\nworm:grace_period = 86400”

Best regards
Josef

4

Aw man. :frowning_face: I thought removing Auxiliary Parameters was going to stop people from doing that.

This means that I can’t just apply my own custom options in a convenient GUI, but people who still want to tinker aimlessly can do so anyways.

5

Don’t forget to stop and start the smb Service after editing. To check the parameter use testparm in a root shell.

6

Please have a look at the CLI reference guide:

7

Hello,

Is there nobody with the same problem? In my opinion this is a general performance related bug in TrueNAS Scale.

Please try it out yourself. It’s very easy to reproduce this bug:

  • Take TrueNAS Scale, create a mirror pool (e.g., pool1), create a dataset and an SMB share.
  • Open a shell as root user on the TrueNAS console and enter the command zpool iostat -v pool1 1
  • Use a Windows 11 24H2 client with Defender On Access Scan active.
  • Create or take 1000 small files, e.g., small images < 128kB.
  • Copy the directory with the images to the TrueNAS SMB share.
  • Observe the output on the console: Do you hear or see continuous write access? Then you have very poor performance and have reproduced the effect.

Cross-check

  • On 24H2 disable Defender on Access Scan or use robocopy to copy.
  • Or use an Windows 10 22H2 Client.
  • Or use a Windows Server or OpenMediaVault with ZFS as the server.
  • Here you will see that disk access only occurs every 5 seconds.

In my opinion, this is a serious bug in TrueNAS Scale, as it is no longer possible to work properly with small files on TrueNAS.

Best regards
Josef

8

I’d say it was a problem in Windows 11, not TrueNAS.

9

Hello.

I don’t think so.

Windows 11 24H2 as client works with Windows Server and OpenMediaVault with ZFS and Samba. So Microsoft surely will not fix something in Win 11 24H2

And for this reason it is a problem of TrueNAS, probably the ZFS or Samba configuration or their settings.

If the TrueNAS team cannot find and fix the cause here, TrueNAS will be virtually unusable for Windows clients and they will be out of business.

I’ll say it again: Every Win11 24H2 client with TrueNAS SMB shares is likely to have this problem.

So, I’m calling on all TrueNAS operators: please check this on your systems. It’s really very simple.

I would like to hear from you.

Best regards
Josef

10

You can use Report a Bug in the TrueNAS GUI and link the created Jira ticket in this thread. Upper right of GUI is a smile icon for Feedback / Report a Bug.

11

Please see the Jira ticket for my bug report:
https://ixsystems.atlassian.net/browse/NAS-137511

12

The Bug Clerk Bot closed both tickets for formal reasons. No real person even have read the ticket and addressed this problem. I’m sure the same issue happens with ixsystems’ TrueNAS appliances. It’s a shame, and opening a ticket in Jira is a waste of time.

My problem report has already received 143 views in this forum, and it seems that no one has yet tested the performance of Windows 11 24H2 clients with small files on their TrueNAS server at home. If they did, they would see that there is a fundamental problem here.

As the situation stands, TrueNAS is unusable for Windows 11 24H2 clients because the performance with small files sucks.

I am very disappointed.

Best regards,
Josef

1 Like
13

I perfectly agree with your sentiment and your disappointment about the attitude towards your bug report.

Unfortunately I do not run Windows as a desktop anywhere and neither does my company.

Their enterprise customers should be interested in this, but then they probably do not run your version of TrueNAS, yet.

But again: poor judgement by the ticket triage, because if this is reproducible, it will backfire big when rolled out to enterprise installations.

Sorry
Patrick

1 Like
14

I just stumbled across your post checking the truenas forum for news, as I have just beefed up our truenas servers and am in the process of preparing to switch some workstations from w10 to w11. So that’s very relevant (and possibly very bad ) news.
I will try to find some time tomorrow to do some initial testings in this matter. Will have to quickly setup a w11 test client from scratch, but I think it’s a good idea anyways to test this on a freshly installed windows11 on bare metal.

I promise to keep you informed on my findings. If I don’t find the time tomorrow, it will be on Monday.

And I totally agree, the handling of those bug reports leaves me speechless. What a shame.

Cheers, Leif

15

Hello

Leif, thanks for the response. I’m looking forward for your findings.

I have now also set up TrueNAS Core 13.0-U6.8 in a VM and carried out a test. And lo and behold, there is only one write access to the pool disks every 5 seconds, as it should be.

It looks like there really is only a problem with TrueNAS Scale. Scale versions I have tried and seen the Problem:
24.10.2.3
25.4.2.1, 25.4.2.3
25.10-BETA.1

Best regards
Josef

16

Hello,

I can’t believe it. Something magic happened!

TrueNAS Scale 25.04.2.4 is now released:

I’ve upgrades my faulty TrueNAS system from 25.04.2.3 to 25.04.2.4 and my problem is solved! Gone! Away!

I also had the issue where SMB threads could become stuck at 100% CPU usage on Windows clients NAS-137095. Also fixed!

Now it has come to a good ending after all.

Thank you all!

Best regards
Josef

1 Like
17

Hi Josef, thank you very much for checking out that new version. That’s good news indeed.
Have been tinkering with the win 11 install right now, took some time as I had to find a spare machine that’s not in use at the moment.

Will install the update and do some tests over the weekend.

Thanks !