EDIT: VMs Losing DHCP Lease on TrueNAS Core 13.0 - Network Issue or Time Sync Issue?

Apps and Virtualization
, ,
1

Hi everyone!

I’m facing a recurring, though not easily reproducible, network issue with my VMs hosted on TrueNAS Core 13.0-U6.2.

Host System:

  • TrueNAS Core 13.0-U6.2
  • Physical NICs: Intel X540-T2 (10G) and integrated 1G NIC
  • Network Configuration: The X540-T2 is in a LAGG (LACP). The integrated NIC is hosting one VM on a separate L2 network (DMZ).

VM Setup:

  • All 6 VMs are running Debian 12 (latest stable) with minimum installations.
  • VMs use VIRTIO network drivers.
  • IPs are assigned through static DHCP mappings in pfSense.
  • IPv4 only, IPv6 is disabled network wide in pfsense.
  • DHCP leases are stable for the host system and all jails (7) on the LAGG interface, as for all other devices on my LAN.

Issue:
After rebooting either the host or the VMs, all VMs lose their IP addresses at some point—typically within 15-20 minutes after a reboot. The network interface remains UP, but the IP is lost. The issue affects all VMs, regardless of whether they are on the LAGG interface or the integrated NIC.

Workaround:
Manually running dhclient on each VM restores the IP, which then remains stable until the issue recurs.

Additional Info:

  • No lease time adjustments have been made in pfSense or the VMs.
  • MAC addresses are unique for each VM.
  • Two VMs are running Docker containers, and three are running non-containerized web applications. All are equally affected.
  • VMs required manual configuration in /etc/network/interfaces due to NIC renaming after removing the CD-ROM device post-installation.

Questions:

  1. Has anyone encountered similar behavior with bhyve?
  2. Should I have set up a bridge in TrueNAS for the VMs instead of just using the LAGG interface? I saw some posts about in the old forum, but it didn’t seemed needed in my case.
  3. I understand setting static IPs directly in the VMs might be a better solution, but is there a fix for maintaining DHCP assignments?
  4. Moving to a dedicated Proxmox/XenServer machine isn’t feasible for me at the moment.

Any insights or suggestions would be greatly appreciated!

2

After further investigation, I believe the network issue with my Debian VMs might be related to a system time offset rather than a network configuration problem.

Observations:

  • Time Offset at Reboot:

    • After rebooting any Debian VM, checking the /tmp/dhclient-script.debug file shows that the initial DHCP lease time is recorded with a +2 hour offset from my local time (CEST), which was correctly set during installation.

    • For example, if the VM boots at 14:00 CEST, the DHCP lease time in the debug script is reported as 16:00 CEST.

  • Correct Lease Time After Interface Reset:

    • If I manually reset the network interface using ifdown enp0s4 && ifup enp0s4, the lease time is correctly reported in CEST without the 2-hour offset.

  • System Time Checks:

    • Running the date command at any point returns the correct local time (CEST) in the VMs.

I suspect that the issue may be related to the system time not being fully synchronized when the DHCP lease is initially obtained during the VM’s boot process. The VM might be acquiring the lease before the local time is correctly set, leading to the observed offset.

Current Configuration:

  • TrueNAS Settings: Local time is set in the TrueNAS system settings as Europe/Rome.

  • VM Settings: Each VM is configured with the “local time” option enabled by default.

  • pfSense Settings: The DHCP server on pfSense is configured to use local time and my time zone in pfsense is Europe/Rome.

Has anyone seen this kind of behaviour elsewhere?

Also please notify me if this getting OT.

Thanks!

3

The lease time reported is two hours off, and if I’m not mistaken CEST is UTC+2. Try setting the clock in the VM to UTC rather than Local (Local is, IMO, a poor default; any OS other than Windows expects the (emulated) hardware clock to be in UTC).

1 Like
4

Thanks for the answer @dan,

I thought that defaulting to local time in the VM settings was necessary to maintain coherence with the local settings with the trueNAS host, but it was my own rationalisation, so I gave it a try on your input.

I changed the System Clock of one VM from Local to UTC and I verified that over reboots the lease issuing times sync up with “my time”: the lease is no longer given out in the future, as it was.

I got caught out by the default settings, for years, it seems.

At this stage the issue could be solved? I’ll test it out for a couple of hours, but thank you in the mean time for the needed nudge to edit the VM settings.

Thanks!

5

Thanks for solution.
I just had the same exact problem in Truenas SCALE. Debian couldnt renew IPv4 after initial boot.
This only happened with IPv4. IPv6 kept working correctly.
Ubuntu didnt have this problem, only Debian.

After changing System Clock to UTC in Truenas VM settings it now works correctly.

This really needs better explanation and settings.

1 Like