Network Configuration for New Instances in TrueNAS SCALE 25.04 Fangtooth

In TrueNAS SCALE 25.04, the Virtualization feature has been transformed into “Instances”, migrating from KVM to Incus. This change has significantly altered virtual machine network configuration, making it more challenging to set up properly.

My TrueNAS physical machine has only one network port connected to a switch. In previous versions, to enable TrueNAS to ping virtual machines, I configured a network bridge called br0.

In the earlier Virtualization version, I simply needed to set the virtual machine’s network interface to bridged mode and select br0, which allowed the NAS to ping the virtual machine.

In the new Instances feature, network bridging offers two options: “Bridged NICs” and “Macvlan NICs.”

I first configured it with “Bridged NICs” connected to the br0 interface. While TrueNAS could ping the virtual machine, one issue emerged—the virtual machine couldn’t obtain IPv6 addresses from the router, possibly related to interface MAC addressing.

When I switched to “Macvlan NICs,” the virtual machine successfully obtained IPv6 addresses, but TrueNAS could no longer ping it.

Finally, I tried adding two network interfaces to the virtual machine—one “Bridged NIC” and one “Macvlan NIC”—both connected to the br0 interface. The result was as desired: the virtual machine successfully obtained IPv6 addresses, and TrueNAS could ping it.

Although this approach met my requirements, the configuration seems unusual. Do you have any best practices to recommend for this situation?

Sorry that I forgot to set the categories to Apps and Virtualization.

Honestly, one or both of these may have been an oversight/bug. Instances are a new and experimental feature that we expect to stabilize throughout the 25.04 release cycle. If you submit a bug report about Instances and IPv6 compatibility, our engineers can look into this further.

1 Like

Thank you. I will test it carefully again, and then decide whether to report the bug.

I did this same setup and had no trouble getting an IPv6 address. :thinking:

Okay, I’ll try again.

Could you please share your full network configuration here (is there an cli command available to do this instead of screenshots ?).

I’ll face the same challenge although I have two NICs for network aggregation.

If I’m not mistaken you can completely segregate instances from host and even put them on different VLAN?
So if a instance is compromised, TN’s security is not necessarily on stake?

just use the macvlan mode and they would be segregated

1 Like

@Skimmer5512 Did you ever get your IPv6 sorted? If you’re using SLAAC things can be a little temperamental. Sometimes they ask for a RA, and sometimes they wait to see a RA. I’m not sure what happens with a bridge interface.

RA = Router Advertisement.

Does anyone uses VLANs as well?

It is a pain that I am away from my TN. I have only the documentation available.

Does segregation still apply if VLANs Interface configured (and link aggregation)?

My setup shall be

  • TN (GUI, SSH etc) only reachable via VLAN_Mgtm
  • private apps reachable through VLAN_privat
    • private apps can read TN dataset locally
  • shared apps reachable through VLAN_shared

I have two NICs

So os the following hierarchy correct ?

  • en0 + en1-> bond0
  • bond0
    • VLAN_Mgtm, IP by DHCP
    • VLAN_private
      • bridge
        • Apps
    • VLAN_shared

MACVLAN on VLAN_…?

well, i just added another network interface and used the mac vlan mode

Eventually, I could spare a few hours to do my tests.
I will share more details tomorrow


Is there any reason not to put an IP on a VLAN Interface if a BRIDGE is connected to that?
I’m losing connection to the GUI after a short while if I do so.

Working

Failing after a short while


How can I map an extra NICs MAC, IP to the network interface in the TN GUI, which shows up in TN GUI Network, unlike the auto-configured bridge?

Hi @PackElend, I have a similar configuration you had, but maybe my backend network is configured differently, but I don’t assign IPs to VLAN, but assign them to the Bridge instead, then I have a static route for the separate VLAN.

what different does ot make?

I assigned IP to VLAN-NICs only when there is not a Bridge in the VLAN.
As soon as a Bridge is involved I assigned the IP to the bridge (otherewise that GUI is not reachable anymore)

By the way, why to you need Static Routes as it works without them on my side.
Is is still the case that outbound uses Default Route?

he is my full network configruation.
Using TrueNAS CLI shell results in better overview, hope that the GUI will shown the same information once.

  1. SSH into TN than open Console setup menu (CSM)
    /usr/bin/cli --menu
  2. run the CLI and run nework query
    • press 7
    • type network interface query

more details, see Using the Console Setup Menu | TrueNAS Documentation Hub and Docs Hub | Console Setup Menu Configuration.

commands to get the table as below
C:\Users\test>ssh root@10.10.9.231
(root@10.10.9.231) Password: 
Linux truenas 6.12.15-production+truenas #1 SMP PREEMPT_DYNAMIC Tue Apr 15 20:07:13 UTC 2025 x86_64

        TrueNAS (c) 2009-2025, iXsystems, Inc. dba TrueNAS
        All rights reserved.
        TrueNAS code is released under the LGPLv3 and GPLv3 licenses with some
        source files copyrighted by (c) iXsystems, Inc. All other components
        are released under their own respective licenses.

        For more information, documentation, help or support, go here:
        http://truenas.com

Warning: the supported mechanisms for making configuration changes
are the TrueNAS WebUI, CLI, and API exclusively. ALL OTHERS ARE
NOT SUPPORTED AND WILL RESULT IN UNDEFINED BEHAVIOR AND MAY
RESULT IN SYSTEM FAILURE.

Welcome to TrueNAS
Last login: Thu May 15 18:25:13 2025 from 10.10.9.1
root@truenas[~]# /usr/bin/cli --menu                                                                                                                                                                                                                  
1) Configure network interfaces
2) Configure network settings
3) Configure static routes
4) Change local administrator password
5) Create one-time password for "root"
6) Reset configuration to defaults
7) Open TrueNAS CLI Shell
8) Open Linux Shell
9) Reboot
10) Shutdown

Enter an option from 1-10: 7

Type "ls" (followed by Enter) to list available configuration options

[truenas]> network interface query
+-----------------+------------------+------------------------------+----------------+-----------+-----------+----------------+--------+----------------+-------------+-----------------+-----------------------+-------------+-------------+------------------+-------------+--------------+-------------+
| name            | type             | state.aliases                | aliases        | ipv4_dhcp | ipv6_auto | description    | mtu    | bridge_members | stp         | enable_learning | vlan_parent_interface | vlan_tag    | vlan_pcp    | xmit_hash_policy | lacpdu_rate | lag_protocol | lag_ports   |
+-----------------+------------------+------------------------------+----------------+-----------+-----------+----------------+--------+----------------+-------------+-----------------+-----------------------+-------------+-------------+------------------+-------------+--------------+-------------+
| enx0c3796000562 | PHYSICAL         | 10.10.9.231/24               | <empty list>   | true      | false     | USB            | 1500   | <undefined>    | <undefined> | <undefined>     | <undefined>           | <undefined> | <undefined> | <undefined>      | <undefined> | <undefined>  | <undefined> |
|                 |                  | fe80::e37:96ff:fe00:562/64   |                |           |           |                |        |                |             |                 |                       |             |             |                  |             |              |             |
| eno1            | PHYSICAL         | <empty list>                 | <empty list>   | false     | false     |                | <null> | <undefined>    | <undefined> | <undefined>     | <undefined>           | <undefined> | <undefined> | <undefined>      | <undefined> | <undefined>  | <undefined> |
| eno2            | PHYSICAL         | <empty list>                 | <empty list>   | false     | false     |                | <null> | <undefined>    | <undefined> | <undefined>     | <undefined>           | <undefined> | <undefined> | <undefined>      | <undefined> | <undefined>  | <undefined> |
| bond0           | LINK_AGGREGATION | fe80::3eec:efff:fe8d:8bc6/64 | <empty list>   | false     | false     | eno1 + eno2    | 1500   | <undefined>    | <undefined> | <undefined>     | <undefined>           | <undefined> | <undefined> | LAYER2+3         | SLOW        | LACP         | eno1        |
|                 |                  |                              |                |           |           |                |        |                |             |                 |                       |             |             |                  |             |              | eno2        |
| vlan130         | VLAN             | fe80::3eec:efff:fe8d:8bc6/64 | <empty list>   | false     | false     | VLAN_private   | 1500   | <undefined>    | <undefined> | <undefined>     | bond0                 | 130         | <null>      | <undefined>      | <undefined> | <undefined>  | <undefined> |
| br130           | BRIDGE           | 10.10.20.12/22               | 10.10.20.12/22 | false     | false     | bridge_PRIVATE | 1500   | vlan130        | true        | true            | <undefined>           | <undefined> | <undefined> | <undefined>      | <undefined> | <undefined>  | <undefined> |
|                 |                  | fe80::d40b:3bff:fe80:1ab5/64 |                |           |           |                |        |                |             |                 |                       |             |             |                  |             |              |             |
| vlan12          | VLAN             | fe80::3eec:efff:fe8d:8bc6/64 | <empty list>   | false     | false     | vlan_SHARED    | 1500   | <undefined>    | <undefined> | <undefined>     | bond0                 | 12          | <null>      | <undefined>      | <undefined> | <undefined>  | <undefined> |
| vlan8           | VLAN             | 10.10.8.66/24                | 10.10.8.66/24  | false     | false     | vlan_SYS       | 1500   | <undefined>    | <undefined> | <undefined>     | bond0                 | 8           | <null>      | <undefined>      | <undefined> | <undefined>  | <undefined> |
|                 |                  | fe80::3eec:efff:fe8d:8bc6/64 |                |           |           |                |        |                |             |                 |                       |             |             |                  |             |              |             |
| mac85f0685b     | PHYSICAL         | <empty list>                 | <empty list>   | false     | false     |                | <null> | <undefined>    | <undefined> | <undefined>     | <undefined>           | <undefined> | <undefined> | <undefined>      | <undefined> | <undefined>  | <undefined> |
| macd3789f96     | PHYSICAL         | <empty list>                 | <empty list>   | false     | false     |                | <null> | <undefined>    | <undefined> | <undefined>     | <undefined>           | <undefined> | <undefined> | <undefined>      | <undefined> | <undefined>  | <undefined> |
| mac0e1280bb     | PHYSICAL         | <empty list>                 | <empty list>   | false     | false     |                | <null> | <undefined>    | <undefined> | <undefined>     | <undefined>           | <undefined> | <undefined> | <undefined>      | <undefined> | <undefined>  | <undefined> |
+-----------------+------------------+------------------------------+----------------+-----------+-----------+----------------+--------+----------------+-------------+-----------------+-----------------------+-------------+-------------+------------------+-------------+--------------+-------------+
  • enx0c3796000562 is LAN2USB-Interface as backup as long as I cannot trust the given configuration 100%.
  • LAYER3+4 seems to work realaibel as well. My switch is an CRS328-4C-20S-4S+

That is my intended network (Currently)

flowchart TD
 subgraph Physical_Layer["Physical Layer"]
        en0["NIC en0"]
        en1["NIC en1"]
        bond0["LAGG bond0 <br>(LAYER 2+3)"]
  end
 subgraph VLAN_Interfaces["VLAN Interfaces"]
        vlan8["VLAN vlan8 <br><i>MGTM</i><br>IP 10...."]
        vlan130["VLAN vlan130 <br><i>private</i><br>IP n/a"]
        vlan12["VLAN vlan12 <br><i>shared</i><br>IP n/a"]
  end
 subgraph TrueNAS_Host["TrueNAS Host"]
        GUI["Web GUI/SSH on VLAN_Mgtm"]
        dataset["Datasets /mnt/pool/..."]
  end
 subgraph Private_App_Network["Private App Network"]
        br0_private["BRIDGE br130 <br><i>private</i><br>IP n/a"]
        app1["Private App 1"]
        app2["Private App 2"]
        n2["inter app communication?"]
  end
 subgraph Shared_App_Network["Shared App Network"]
        macvlan1["MACVLAN of vlan 12"]
        sApp1["Shared App 1"]
        macvlan2["MACVLAN of vlan 12"]
        sApp2["Shared App 2"]
        n1["inter app communication?"]
  end
    en0 --- bond0
    en1 --- bond0
    bond0 --- vlan8 & vlan130 & vlan12
    vlan130 --- br0_private
    br0_private --- app1 & app2
    app2 -- Host Mount --- dataset
    vlan12 --- macvlan1 & macvlan2
    macvlan1 --- sApp1
    vlan8 -.- GUI
    macvlan2 -.- sApp2
    sApp1 -.- n1
    sApp2 -.- n1
    app1 --- n2
    app2 --- n2

    vlan130@{ shape: rect}
    n2@{ shape: subproc}
    n1@{ shape: subproc}


in and outbound work well without Static Routes in my system, if that is the reason using Static Routes

debian-admin@vm-DEBIAN:~$ ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
9: eth1: <BROADCAST,MULTICAST,DYNAMIC,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 00:16:3e:55
debian-admin@vm-DEBIAN:~$ routel
Dst             Gateway         Prefsrc         Protocol Scope   Dev              Table
default         10.10.8.1                                        eth1
10.10.8.0/24                    10.10.8.3       kernel   link    eth1
10.10.8.1                                                link    eth1
10.10.8.3                       10.10.8.3       kernel   host    eth1             local
10.10.8.255                     10.10.8.3       kernel   link    eth1             local
127.0.0.0/8                     127.0.0.1       kernel   host    lo               local
127.0.0.1                       127.0.0.1       kernel   host    lo               local
127.255.255.255                 127.0.0.1       kernel   link    lo               local
debian-admin@vm-DEBIAN:~$ traceroute google.com  
traceroute to google.com (142.250.203.110), 30 hops max, 60 byte packets
 1  10.10.8.1 (10.10.8.1)  0.349 ms  0.329 ms  0.275 ms
 2  3.80.254.84.ftth.as8758.net (84.254.80.3)  1.884 ms  2.235 ms  2.564 ms
 3  gwen.glb.as8758.net (212.25.27.124)  1.556 ms  1.894 ms grace.glb.as8758.net (212.25.27.126)  1.919 ms
 4  212.25.28.50 (212.25.28.50)  0.925 ms  0.891 ms  0.821 ms
 5  glyn.glb.as8758.net (212.25.27.247)  0.903 ms glyn.glb.as8758.net (212.25.27.245)  0.947 ms glyn.glb.as8758.net (212.25.27.247)  0.882 ms
 6  83.150.38.119 (83.150.38.119)  1.121 ms  1.185 ms 72.14.198.50 (72.14.198.50)  1.214 ms
 7  142.251.53.177 (142.251.53.177)  2.524 ms 142.251.245.151 (142.251.245.151)  1.099 ms 142.251.53.177 (142.251.53.177)  2.376 ms
 8  192.178.254.76 (192.178.254.76)  1.524 ms 216.239.41.42 (216.239.41.42)  1.680 ms 142.251.70.185 (142.251.70.185)  0.930 ms
 9  zrh04s16-in-f14.1e100.net (142.250.203.110)  1.484 ms 192.178.109.54 (192.178.109.54)  1.746 ms zrh04s16-in-f14.1e100.net (142.250.203.110)  0.811 ms

what leaves me still clueless is that I still can reach the VM via VNC and the TN host IP altough the don’t share a NIC.

RealVNC® Viewer 7.13.1 (r57) x64 (Dec 6 2024 18:17:18)
OS: Microsoft Windows 11, version 24H2, x64
Desktop name: QEMU (desktop-DEBIAN)
VNC Server: 10.10.9.231::5900 (TCP)
Size: 1280 x 800
Pixel format: depth 24 (32 bpp) little-endian rgb888
VNC Server default: depth 24 (32 bpp) little-endian rgb888
Audio format: VNC Server does not support audio
Requested encoding: ZRLE2
Last-used encoding: Hextile
Line-speed estimate: 100000 kbit/s (RTT ~0ms)
Protocol version: 3.8
Security method: no encryption [VncAuth]
Connection type: Direct TCP
Extensions: