Full SMB auditing at high event rates causes multi-second SMB stalls (smbd blocks on journald): recommended config? (25.04.2.6)

TrueNAS General
,
1

Problem statement

Hi everyone, hoping to tap into the community’s experience here.

We need full SMB audit logging: all event types, all users, no exclusions, at a sustained rate of hundreds to thousands of events per second, eventually streamed to an external logging/SIEM system.

With auditing enabled at that volume on 25.04.2.6, we see two problems:

  1. SMB stalls: smbd blocks while handing audit events to systemd-journald, so audited operations wait. Clients see 2-5 second freezes even though storage, CPU and network are mostly idle.
  2. Audit database growth: the local SMB audit database reached 353 GB (plus a 361 GB WAL), and the daily retention cleanup now holds it for 13+ hours, longer each day.

For now we have disabled SMB auditing, which removes the stalls completely. We would like to turn it back on in full.

Where we would really value your help: is there a recommended or supported configuration for high-volume SMB auditing? For example, journald and socket tuning, sending audit events straight to remote syslog without the local journald/SQLite path, or retention settings that scale. Has anyone run full SMB auditing at this kind of volume successfully? It is entirely possible we are missing something on our side, so any pointers are welcome.

Environment

  • TrueNAS SCALE 25.04.2.6 (Fangtooth), Community Edition
  • Samba 4.21.9-truenas, OpenZFS 2.3.0
  • AMD EPYC 9124 (16c/32t), 256 GB ECC RAM
  • Pool: single RAIDZ2 vdev, 8x 61 TB NVMe; boot pool: 2x SATA SSD mirror
  • 100GbE to clients; 60-90 concurrent SMB clients
  • AD member, SMB3.11 with signing; shares are multi-protocol (SMB + NFS)
  • SMB auditing enabled on 4 shares, retention 30 days, no quota, remote logging enabled

Symptoms

  • Users report Explorer browsing and applications freezing for a few seconds at a time.

  • Server-side SMB2 profile, 30s samples at busy moments (maybe not real full pressure):

    Call Mean server time Underlying syscalls
    smb2_create 1.8 - 6.8 ms openat 5-15 us, fstat 2-4 us
    smb2_close 1.7 - 4.8 ms close 2 us
    smb2_read 0.9 - 1.6 ms async pread ~0.6-1.1 ms

    At quiet moments, create is ~0.1-0.5 ms and close ~0.03-0.1 ms.

  • Pool and ARC are not the issue: ARC demand hit rate 99.7%, drives mostly idle, CPU < 40%.

Evidence

1. smbd main threads blocked in sendto → journald

Sampling /proc/<pid>/task/<tid>/syscall and wchan for all smbd: client processes (50 Hz, 5 s, read-only):

Moment Main threads busy (non-idle) …of which sendto in unix_wait_for_peer
Quiet 2.0% 0.8%
Busy (slow opens/closes) 20.7% 19.5%
Single client connection during its stalls - 66.7%

Resolving the blocked fd via /proc/<pid>/fd and ss -xapn, the peer is /run/systemd/journal/dev-log (systemd-journald). net.unix.max_dgram_qlen = 512. Samba itself is configured with logging = file, so these sends are the audit events.

2. Client-side A/B on the same client, same moment

Linux CIFS client with two mounts from the same NAS, stat -f every 3 s for 1 min:

Share Audited? Responses > 200 ms (of 20) Worst
Share A yes 13-14 4,800 ms
Share B no (auditing disabled) 0 < 200 ms

At the same time, the smbd process serving that client showed 23-67% of its time in sendto / unix_wait_for_peer.

3. After disabling auditing and reconnecting clients

  • New SMB audit records in 30 s: ~19,600 → 0
  • smbd blocked on journald: 10-20% → 0.0%
  • Share A stat -f: 0 of 20 responses over 200 ms

4. journald behaviour

  • Audit events are written to per-UID journals (user-<uid>.journal). The journal of the busiest account rotated every 4-5 seconds:

    systemd-journald: Data hash table of .../user-<uid>.journal has a fill level at 75.0
  (8534 of 11377 items, 6553600 file size, 767 bytes per hash table item), suggesting rotation.
systemd-journald: .../user-<uid>.journal: Journal header limits reached or header out-of-date, rotating.

  • journald config: SystemMaxUse=50M (so ~6 MB per file by default), split by UID.

  • journald averaged ~18% of one core over 8 days and was restarted once by systemd (NRestarts=1). It is single-threaded; under bursts of audited activity it cannot keep up, the 512-message socket queue fills, and smbd blocks.

  • Sustained audit rate: ~650-750 events/s on average, with bursts in the thousands per second during rendering and indexing (open, read and close are each audited).

5. Audit database growth and retention cleanup

Date SMB.db SMB.db-wal On disk (lz4)
25 Sep 85.7 GB 47 GB 17 GB
30 Sep 251 GB 222 GB 61 GB
5 Oct 353 GB 361 GB ~88 GB

Sampling max(rowid) over time shows periods with almost no inserts. They line up with the daily retention task (AuditBackendService.__lifecycle_cleanup, every 86400 s), each run starting roughly 24 h after the previous one ended:

Run Approx. start Duration (no inserts)
1 Fri 21:32 ~2 h
2 Sat 23:03 ~4.8 h
3 Mon 04:17 ~5.7 h
4 Tue 10:29 ~10 h
5 Wed 22:13 ~13.7 h

middlewared log:

(ERROR) AuditBackendService.__lifecycle_cleanup():250 - SYSTEM: failed to enforce retention on audit DB.
sqlite3.OperationalError: database is locked
[SQL: DELETE FROM "audit_SYSTEM_0_1" WHERE "audit_SYSTEM_0_1".message_timestamp < ?]

midclt call audit.query '{"services":["SMB"], ...limit 2000...}' times out on this database size, and the Audit page in the UI is effectively unusable.

6. Audit setting changes only apply to new tree connects

After disabling auditing on a share, existing connections kept producing audit events for hours (long-lived connections from render nodes and services) until smbcontrol all close-share <share> forced clients to reconnect.

What we did (temporary)

  1. Disabled SMB auditing on all shares, until we find a configuration that can sustain full auditing.
  2. Forced reconnects with smbcontrol all close-share <share> in a quiet window.
  3. Verified: 0 new audit records, 0% smbd time blocked on journald, no client stalls.

Where we would appreciate guidance

  1. High-volume auditing: what is the recommended configuration for full SMB auditing (all users, all events) at hundreds to thousands of events per second, without smbd blocking on journald?
  2. Offloading: with remote logging enabled, can audit events go straight to a remote syslog/SIEM without the local journald and SQLite path, or with that path made non-blocking? Is the local audit database required?
  3. journald tuning: are there supported, persistent settings for this load (for example SplitMode, SystemMaxFileSize, SystemMaxUse, rate limits, or net.unix.max_dgram_qlen)? Changes under /etc are lost on update.
  4. Audit database size: what is the safe, supported way to shrink or reset a 353 GB SMB.db with a 361 GB WAL? Would lowering retention now just trigger an even larger DELETE?
  5. Retention cleanup: is a single DELETE holding the write lock for 10+ hours expected at this size? Is message_timestamp indexed, and is batching planned?
  6. Existing connections: should audit setting changes apply to existing tree connects, or should the UI note that clients need to reconnect?

Reproduction / measurement commands (all read-only)

Share of smbd main-thread time blocked sending to journald (5 s):

sudo python3 - << 'EOF'
import os, time
pids = [p for p in os.listdir('/proc') if p.isdigit() and os.path.exists(f'/proc/{p}/cmdline')
        and b'smbd: client' in open(f'/proc/{p}/cmdline','rb').read()]
tot = blk = 0; end = time.time() + 5
while time.time() < end:
    for p in pids:
        try:
            sc = open(f'/proc/{p}/task/{p}/syscall').read().split()
            wc = open(f'/proc/{p}/task/{p}/wchan').read().strip()
        except Exception: continue
        tot += 1; blk += (sc and sc[0] == '44' and wc == 'unix_wait_for_peer')
    time.sleep(0.02)
print(f'smbd main threads blocked in sendto/unix_wait_for_peer: {blk/tot*100:.1f}%')
EOF

New SMB audit records per 30 s:

sudo python3 -c "import sqlite3,time
m=lambda: sqlite3.connect('file:/audit/SMB.db?mode=ro',uri=True).execute('select max(rowid) from audit_SMB_0_1').fetchone()[0]
a=m(); time.sleep(30); print('new SMB audit records in 30 s:', m()-a)"

Client-side stall check (Linux CIFS client):

for i in $(seq 20); do for m in /mnt/nas/shareA /mnt/nas/shareB; do s=$(date +%s%N); stat -f "$m" >/dev/null; \
e=$(( ($(date +%s%N)-s)/1000000 )); [ $e -gt 200 ] && echo "$(date +%T) $m ${e} ms"; done; sleep 3; done

Thank you for taking the time to read this, and for any experience or advice you can share.