Possible race condition in NFS service startup on TrueNAS Core 13.0 U6.1/U6.2

SilverNetworks · July 24, 2024, 3:01pm

Hi TrueNAS Community,

I wanted to share a workaround I found for a possible race condition affecting the NFS service startup on TrueNAS Core. I’ve experienced issues where the NFS service would not function correctly after the system boot process, but restarting it manually afterward works without problems.

Issue Description:

I have several TrueNAS Core systems that are used for NFS only, and only one exhibits this behavior. When the NFS service is set to start automatically, everything appears to be working normally, but sometimes NFS clients are unable to (re)connect, which I suspect is due to a race condition or startup timing issue with NFS services.

However, all of the NFS services appear to be running normally when I check via SSH:

service rpcbind status; service mountd status; service nfsd status; service statd status; service lockd status

I get the following:

rpcbind is running as pid 3596.
mountd is running as pid 3620.
nfsd is running as pid 3633 3634.
statd is running as pid 3648.
lockd is running as pid 3666.

When I have a client successfully connected to an NFS share, and I reboot the TrueNAS Core NFS box, it will report the usual “server “servername” not responding, still trying” until the NFS box has finished booting. However, the client will simply not read/write to the share (e.g., ‘df’ will hang) until I manually stop and restart the NFS service via the TrueNAS Core GUI (sometimes it takes a few tries of switching the service off and back on again). Afterward, everything works normally again without doing anything on the client.

Workaround:

By disabling the automatic start of the NFS service and instead using an init command to start the services, I was able to resolve/workaround the issue.

Steps to Resolve/Workaround:

Disable Automatic NFS Service Start:

Navigate to Services in the TrueNAS GUI.
Disable the NFS service from starting automatically.

Create an Init Command:

Go to Tasks.
Scroll down to Init/Shutdown Scripts.
Click Add to create a new init script.
Set Type to Command.
In the Command field, enter the following command:
sleep 5; service rpcbind onestart; sleep 1; service mountd onestart; sleep 1; service nfsd onestart; sleep 1; service statd onestart; sleep 1; service lockd onestart
Set When to Post Init.
Enable the script by setting Enabled to Checked.
Set Timeout to 30 Seconds.
Save the init script.

Explanation:

The init command introduces a 5-second delay before starting the NFS-related services, followed by a 1-second delay between each service start. While it doesn’t seem to be absolutely necessary to include the delays, I’ve included them out of an abundance of caution since I suspect the issue may be timing-related.

System Details:

TrueNAS-13.0-U6.2
Supermicro X9SRL-F Motherboard, v3.3 BIOS
Intel Xeon E5-2697 v2 CPU
Dynatron R24 2U Server CPU Cooler
8x 32GB SK Hynix HMT84GL7AMR4C-RD ECC RAM
2x 120GB Kingston SSD as mirrored boot
48x Western Digital 8TB HDD
2x 4TB TeamGroup SSD as L2ARC
Intel 800GB 910 Series SSD as SLOG
Adaptec 71605 SAS Controller in HBA Mode
Chenbro NR40700 4U Chassis w/4 PSU
Intel X540-T2 Network Card

Other than the NFS service issue, the system is otherwise working perfectly. All temperatures are well within normal operating limits, eg.:

Questions:

While this workaround seems to resolve the issue, what steps can I take to further troubleshoot the issue of NFS not working properly when it is started automatically via TrueNAS Core?

Cheers,
Greg

volts · July 24, 2024, 3:29pm

Commenting to follow.

Is the machine physically similar to others that work reliably? What’s the network adapter?
I assume you’re using static IP addressing.

You could try enabling Netwait if the problem is sluggish network come-up.

SilverNetworks · July 24, 2024, 3:37pm

The network card is an Intel X540-T2. I’ve updated the original post to include this since I forgot it add it previously.

Other boxes that work reliably, some are similar, some are vastly different, but all have fully working NFS after startup. However, this particular box has the most CPU cores of any of the boxes I have (12 cores). The other boxes are all 6 cores or fewer.

IP is set static. I will try Netwait and report back.

NickF1227 · July 24, 2024, 9:20pm

Can you share your client mount options for the affected client you are referencing? It may also be helpful to see what you see in /var/log/messages on the TrueNAS just after the reboot completes.

SilverNetworks · July 25, 2024, 2:05pm

Netwait does resolve/workaround the issue as long as an external IP address is used (either valid or invalid will work). Using loopback or local IPs do not work.

In my case, I used the server’s BMC address for netwait.

Thanks for the suggestion!

SilverNetworks · July 25, 2024, 2:08pm

Client mount entry in fstab:

servername:/mnt/p1/datastore /mnt/datastore nfs defaults,sync 0 0

/var/log/messages: without netwait

Jul 25 05:17:38 servername CPU: Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz (2700.00-MHz K8-class CPU)
Jul 25 05:17:38 servername Origin="GenuineIntel"  Id=0x306e4  Family=0x6  Model=0x3e  Stepping=4
Jul 25 05:17:38 servername Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Jul 25 05:17:38 servername Features2=0x7fbee3ff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND>
Jul 25 05:17:38 servername AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
Jul 25 05:17:38 servername AMD Features2=0x1<LAHF>
Jul 25 05:17:38 servername Structured Extended Features=0x281<FSGSBASE,SMEP,ERMS>
Jul 25 05:17:38 servername Structured Extended Features3=0x9c000400<MD_CLEAR,IBPB,STIBP,L1DFL,SSBD>
Jul 25 05:17:38 servername XSAVE Features=0x1<XSAVEOPT>
Jul 25 05:17:38 servername VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr
Jul 25 05:17:38 servername TSC: P-state invariant, performance statistics
Jul 25 05:17:38 servername kernel: ix0: link state changed to UP
Jul 25 05:17:38 servername kernel: ix0: link state changed to DOWN
Jul 25 05:17:38 servername hwpmc: SOFT/16/64/0x67<INT,USR,SYS,REA,WRI> TSC/1/64/0x20<REA> IAP/4/48/0x3ff<INT,USR,SYS,EDG,THR,REA,WRI,INV,QUA,PRC> IAF/3/48/0x67<INT,USR,SYS,REA,WRI>
Jul 25 05:17:39 servername 1 2024-07-25T05:17:39.008869-07:00 servername. ntpd 3288 - - ntpd 4.2.8p15-a (1): Starting
Jul 25 05:17:39 servername 1 2024-07-25T05:17:39.009031-07:00 servername. ntpd 3288 - - Command line: /usr/sbin/ntpd -p /var/db/ntp/ntpd.pid -c /etc/ntp.conf -f /var/db/ntp/ntpd.drift -g
Jul 25 05:17:39 servername 1 2024-07-25T05:17:39.009037-07:00 servername. ntpd 3288 - - ----------------------------------------------------
Jul 25 05:17:39 servername 1 2024-07-25T05:17:39.009042-07:00 servername. ntpd 3288 - - ntp-4 is maintained by Network Time Foundation,
Jul 25 05:17:39 servername 1 2024-07-25T05:17:39.009047-07:00 servername. ntpd 3288 - - Inc. (NTF), a non-profit 501(c)(3) public-benefit
Jul 25 05:17:39 servername 1 2024-07-25T05:17:39.009051-07:00 servername. ntpd 3288 - - corporation.  Support and training for ntp-4 are
Jul 25 05:17:39 servername 1 2024-07-25T05:17:39.009056-07:00 servername. ntpd 3288 - - available at https://www.nwtime.org/support
Jul 25 05:17:39 servername 1 2024-07-25T05:17:39.009060-07:00 servername. ntpd 3288 - - ----------------------------------------------------
Jul 25 05:17:39 servername nfsd: can't register svc name
Jul 25 05:17:39 servername Security policy loaded: MAC/ntpd (mac_ntpd)
Jul 25 05:17:43 servername kernel: ix0: link state changed to UP
Jul 25 05:17:53 servername 1 2024-07-25T05:17:53.841884-07:00 servername. daemon 3361 - - 2024-07-25 05:17:53,841:wsdd WARNING(pid 3362): no interface given, using all interfaces
Jul 25 05:18:00 servername 1 2024-07-25T05:18:00.083634-07:00 servername. daemon 3636 - - 2024-07-25 05:18:00,083:wsdd WARNING(pid 3637): no interface given, using all interfaces

/var/log/messages: with netwait

Jul 25 05:42:25 servername CPU: Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz (2700.00-MHz K8-class CPU)
Jul 25 05:42:25 servername Origin="GenuineIntel"  Id=0x306e4  Family=0x6  Model=0x3e  Stepping=4
Jul 25 05:42:25 servername Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Jul 25 05:42:25 servername Features2=0x7fbee3ff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND>
Jul 25 05:42:25 servername AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
Jul 25 05:42:25 servername AMD Features2=0x1<LAHF>
Jul 25 05:42:25 servername Structured Extended Features=0x281<FSGSBASE,SMEP,ERMS>
Jul 25 05:42:25 servername Structured Extended Features3=0x9c000400<MD_CLEAR,IBPB,STIBP,L1DFL,SSBD>
Jul 25 05:42:25 servername XSAVE Features=0x1<XSAVEOPT>
Jul 25 05:42:25 servername VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr
Jul 25 05:42:25 servername TSC: P-state invariant, performance statistics
Jul 25 05:42:25 servername kernel: ix0: link state changed to UP
Jul 25 05:42:25 servername kernel: ix0: link state changed to DOWN
Jul 25 05:42:25 servername hwpmc: SOFT/16/64/0x67<INT,USR,SYS,REA,WRI> TSC/1/64/0x20<REA> IAP/4/48/0x3ff<INT,USR,SYS,EDG,THR,REA,WRI,INV,QUA,PRC> IAF/3/48/0x67<INT,USR,SYS,REA,WRI>
Jul 25 05:42:25 servername kernel: ix0: link state changed to UP
Jul 25 05:42:25 servername 1 2024-07-25T05:42:25.241835-07:00 servername. ntpd 3329 - - ntpd 4.2.8p15-a (1): Starting
Jul 25 05:42:25 servername 1 2024-07-25T05:42:25.241996-07:00 servername. ntpd 3329 - - Command line: /usr/sbin/ntpd -p /var/db/ntp/ntpd.pid -c /etc/ntp.conf -f /var/db/ntp/ntpd.drift -g
Jul 25 05:42:25 servername 1 2024-07-25T05:42:25.242002-07:00 servername. ntpd 3329 - - ----------------------------------------------------
Jul 25 05:42:25 servername 1 2024-07-25T05:42:25.242008-07:00 servername. ntpd 3329 - - ntp-4 is maintained by Network Time Foundation,
Jul 25 05:42:25 servername 1 2024-07-25T05:42:25.242012-07:00 servername. ntpd 3329 - - Inc. (NTF), a non-profit 501(c)(3) public-benefit
Jul 25 05:42:25 servername 1 2024-07-25T05:42:25.242017-07:00 servername. ntpd 3329 - - corporation.  Support and training for ntp-4 are
Jul 25 05:42:25 servername 1 2024-07-25T05:42:25.242021-07:00 servername. ntpd 3329 - - available at https://www.nwtime.org/support
Jul 25 05:42:25 servername 1 2024-07-25T05:42:25.242025-07:00 servername. ntpd 3329 - - ----------------------------------------------------
Jul 25 05:42:25 servername nfsd: can't register svc name
Jul 25 05:42:25 servername Security policy loaded: MAC/ntpd (mac_ntpd)
Jul 25 05:42:29 servername 1 2024-07-25T05:42:29.392650-07:00 servername. daemon 3397 - - 2024-07-25 05:42:29,392:wsdd WARNING(pid 3398): no interface given, using all interfaces
Jul 25 05:42:30 servername 1 2024-07-25T05:42:30.871667-07:00 servername. avahi-daemon 3381 - - XML_ParseBuffer() failed at line 1: no element found.
Jul 25 05:42:30 servername 1 2024-07-25T05:42:30.871674-07:00 servername. avahi-daemon 3381 - - Failed to load service group file /usr/local/etc/avahi/services/MIDDLEWARE.service, ignoring.
Jul 25 05:42:36 servername 1 2024-07-25T05:42:36.476032-07:00 servername. daemon 3671 - - 2024-07-25 05:42:36,475:wsdd WARNING(pid 3672): no interface given, using all interfaces

With netwait, we can see that ix0 comes up before the nfsd service is started.

Cheers

NickF1227 · July 25, 2024, 2:10pm

Is DNS hosted on the datastore on this NAS?

SilverNetworks · July 25, 2024, 2:12pm

No, DNS is external to the NAS.
I also added a host entry in the /etc/hosts file on the client in case it was DNS related, but it didn’t help.

NickF1227 · July 25, 2024, 2:14pm

Actually, I’m silly

Jul 25 05:17:38 servername kernel: ix0: link state changed to UP
Jul 25 05:17:38 servername kernel: ix0: link state changed to DOWN
Jul 25 05:17:39 servername nfsd: can't register svc name
Jul 25 05:17:43 servername kernel: ix0: link state changed to UP

Looks like this may be a spanning-tree problem. The interface is down, so it’s a DNS problem caused by the interface being down when nfsd tries to register the svc name.

Can you try changing the switch port to be portfast mode or a spanning-tree edge port ?

SilverNetworks · July 25, 2024, 2:36pm

This NAS is connected to a Netgear XS728T, which I don’t believe has spanning-tree enabled. According to the documentation, there should be STP settings available, but I don’t see them (I suppose because Global Flow Control is disabled):

Under ports, this is all it shows:

etc…

I will investigate further a bit later.

Regarding

nfsd: can't register svc name

This may be a red herring. It will come up even if NFS is manually started after the boot process has completed.

More info about it here:

…and the “here” link is broken: https://redmine.ixsystems.com/issues/11351

Cheers

NickF1227 · July 25, 2024, 2:41pm

I’m confused. You posted a screenshot of everything except the STP screen lol

SilverNetworks · July 25, 2024, 2:56pm

Whoops, my bad… I was looking down the left column for “Advanced” first

I’ve enabled fast link on port 3 where the NAS is connected:

Rebooted the server and same issue.

Cheers

NickF1227 · July 25, 2024, 2:57pm

Can you post messages again?

SilverNetworks · July 25, 2024, 2:59pm

Sure:

Jul 25 07:55:37 servername CPU: Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz (2700.00-MHz K8-class CPU)
Jul 25 07:55:37 servername Origin="GenuineIntel"  Id=0x306e4  Family=0x6  Model=0x3e  Stepping=4
Jul 25 07:55:37 servername Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Jul 25 07:55:37 servername Features2=0x7fbee3ff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND>
Jul 25 07:55:37 servername AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
Jul 25 07:55:37 servername AMD Features2=0x1<LAHF>
Jul 25 07:55:37 servername Structured Extended Features=0x281<FSGSBASE,SMEP,ERMS>
Jul 25 07:55:37 servername Structured Extended Features3=0x9c000400<MD_CLEAR,IBPB,STIBP,L1DFL,SSBD>
Jul 25 07:55:37 servername XSAVE Features=0x1<XSAVEOPT>
Jul 25 07:55:37 servername VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr
Jul 25 07:55:37 servername TSC: P-state invariant, performance statistics
Jul 25 07:55:37 servername kernel: ix0: link state changed to UP
Jul 25 07:55:37 servername kernel: ix0: link state changed to DOWN
Jul 25 07:55:37 servername hwpmc: SOFT/16/64/0x67<INT,USR,SYS,REA,WRI> TSC/1/64/0x20<REA> IAP/4/48/0x3ff<INT,USR,SYS,EDG,THR,REA,WRI,INV,QUA,PRC> IAF/3/48/0x67<INT,USR,SYS,REA,WRI>
Jul 25 07:55:37 servername 1 2024-07-25T07:55:37.972430-07:00 servername. ntpd 3288 - - ntpd 4.2.8p15-a (1): Starting
Jul 25 07:55:37 servername 1 2024-07-25T07:55:37.972589-07:00 servername. ntpd 3288 - - Command line: /usr/sbin/ntpd -p /var/db/ntp/ntpd.pid -c /etc/ntp.conf -f /var/db/ntp/ntpd.drift -g
Jul 25 07:55:37 servername 1 2024-07-25T07:55:37.972596-07:00 servername. ntpd 3288 - - ----------------------------------------------------
Jul 25 07:55:37 servername 1 2024-07-25T07:55:37.972660-07:00 servername. ntpd 3288 - - ntp-4 is maintained by Network Time Foundation,
Jul 25 07:55:37 servername 1 2024-07-25T07:55:37.972665-07:00 servername. ntpd 3288 - - Inc. (NTF), a non-profit 501(c)(3) public-benefit
Jul 25 07:55:37 servername 1 2024-07-25T07:55:37.972670-07:00 servername. ntpd 3288 - - corporation.  Support and training for ntp-4 are
Jul 25 07:55:37 servername 1 2024-07-25T07:55:37.972674-07:00 servername. ntpd 3288 - - available at https://www.nwtime.org/support
Jul 25 07:55:37 servername 1 2024-07-25T07:55:37.972678-07:00 servername. ntpd 3288 - - ----------------------------------------------------
Jul 25 07:55:38 servername nfsd: can't register svc name
Jul 25 07:55:38 servername Security policy loaded: MAC/ntpd (mac_ntpd)
Jul 25 07:55:41 servername kernel: ix0: link state changed to UP
Jul 25 07:55:52 servername 1 2024-07-25T07:55:52.456659-07:00 servername. daemon 3361 - - 2024-07-25 07:55:52,456:wsdd WARNING(pid 3362): no interface given, using all interfaces
Jul 25 07:55:59 servername 1 2024-07-25T07:55:59.774325-07:00 servername. daemon 3634 - - 2024-07-25 07:55:59,774:wsdd WARNING(pid 3635): no interface given, using all interfaces

NickF1227 · July 25, 2024, 3:09pm

Still seeing that when NFS starts when the link was DOWN, which I suspect is the issue regardless. So yes, it looks like a race condition related to your network, but it looks the change you made didn’t do anything?

volts · July 25, 2024, 3:56pm

Yeah, that was my hunch too with the netwait suggestion.

You might be able to improve link-up speed with other changes, but there’s also little reason to disable netwait.

I often put IP addresses (and services) on bridge interfaces instead of physical ones, so they come up regardless of the physical layer. You’d want to confirm that still performs well for your adapter and application/traffic, but it’s a great model in general.

SilverNetworks · July 25, 2024, 4:30pm

Enabling fast link didn’t seem to do anything on/to the NAS.

What’s interesting is that I have 3 more similar servers (one using the same X540-T2 adapter) connected to the same network and switch, but don’t have issues with NFS after a reboot.

Here’s the message log of one of them with a X540-T2 adapter:

Mar  3 08:32:16 otherserver CPU: Intel(R) Xeon(R) CPU E3-1220 v5 @ 3.00GHz (2999.99-MHz K8-class CPU)
Mar  3 08:32:16 otherserver Origin="GenuineIntel"  Id=0x506e3  Family=0x6  Model=0x5e  Stepping=3
Mar  3 08:32:16 otherserver Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Mar  3 08:32:16 otherserver Features2=0x7ffafbff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND>
Mar  3 08:32:16 otherserver AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
Mar  3 08:32:16 otherserver AMD Features2=0x121<LAHF,ABM,Prefetch>
Mar  3 08:32:16 otherserver Structured Extended Features=0x29c6fbf<FSGSBASE,TSCADJ,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,NFPUSG,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PROCTRACE>
Mar  3 08:32:16 otherserver Structured Extended Features3=0x9c002e00<MCUOPT,MD_CLEAR,TSXFA,IBPB,STIBP,L1DFL,SSBD>
Mar  3 08:32:16 otherserver XSAVE Features=0xf<XSAVEOPT,XSAVEC,XINUSE,XSAVES>
Mar  3 08:32:16 otherserver VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
Mar  3 08:32:16 otherserver TSC: P-state invariant, performance statistics
Mar  3 08:32:16 otherserver kernel: ix0: link state changed to UP
Mar  3 08:32:16 otherserver kernel: ix0: link state changed to DOWN
Mar  3 08:32:16 otherserver hwpmc: SOFT/16/64/0x67<INT,USR,SYS,REA,WRI> TSC/1/64/0x20<REA> IAP/8/48/0x3ff<INT,USR,SYS,EDG,THR,REA,WRI,INV,QUA,PRC> IAF/3/48/0x67<INT,USR,SYS,REA,WRI>
Mar  3 08:32:16 otherserver 1 2024-03-03T08:32:16.508513-08:00 otherserver. ntpd 3768 - - ntpd 4.2.8p15-a (1): Starting
Mar  3 08:32:16 otherserver 1 2024-03-03T08:32:16.508651-08:00 otherserver. ntpd 3768 - - Command line: /usr/sbin/ntpd -p /var/db/ntp/ntpd.pid -c /etc/ntp.conf -f /var/db/ntp/ntpd.drift -g
Mar  3 08:32:16 otherserver 1 2024-03-03T08:32:16.508658-08:00 otherserver. ntpd 3768 - - ----------------------------------------------------
Mar  3 08:32:16 otherserver 1 2024-03-03T08:32:16.508662-08:00 otherserver. ntpd 3768 - - ntp-4 is maintained by Network Time Foundation,
Mar  3 08:32:16 otherserver 1 2024-03-03T08:32:16.508666-08:00 otherserver. ntpd 3768 - - Inc. (NTF), a non-profit 501(c)(3) public-benefit
Mar  3 08:32:16 otherserver 1 2024-03-03T08:32:16.508670-08:00 otherserver. ntpd 3768 - - corporation.  Support and training for ntp-4 are
Mar  3 08:32:16 otherserver 1 2024-03-03T08:32:16.508674-08:00 otherserver. ntpd 3768 - - available at https://www.nwtime.org/support
Mar  3 08:32:16 otherserver 1 2024-03-03T08:32:16.508678-08:00 otherserver. ntpd 3768 - - ----------------------------------------------------
Mar  3 08:32:16 otherserver nfsd: can't register svc name
Mar  3 08:32:16 otherserver Security policy loaded: MAC/ntpd (mac_ntpd)
Mar  3 08:32:18 otherserver kernel: ix0: link state changed to UP
Mar  3 08:32:47 otherserver 1 2024-03-03T08:32:47.453244-08:00 otherserver. daemon 3841 - - 2024-03-03 08:32:47,453:wsdd WARNING(pid 3842): no interface given, using all interfaces
Mar  3 08:32:53 otherserver 1 2024-03-03T08:32:53.511374-08:00 otherserver. daemon 4071 - - 2024-03-03 08:32:53,511:wsdd WARNING(pid 4072): no interface given, using all interfaces

The ix0 interface is DOWN when NFS is started as well:

Mar  3 08:32:16 otherserver kernel: ix0: link state changed to DOWN
...
Mar  3 08:32:16 otherserver nfsd: can't register svc name
...
Mar  3 08:32:18 otherserver kernel: ix0: link state changed to UP

I still think it’s an odd networking timing issue when NFS/related services are being started which seem to be triggered perhaps with a CPU with ‘quite a few’ and ‘relatively fast’ cores, but I’m just speculating at this point. However, it does seem to me that networking related services should be started /after/ the network is fully up by default (which is what enabling netwait seems to do).

Cheers