Home Assistant - How I ditched the VM and embraced The Container

Something that I learned is that macvlan resets with reboots. I modified my script to delete the old macvlan and the shim and recreate them. Then set it up with a post init script so it runs when truenas restarts.

Please be careful with suggestions from ChatGPT, it may hallucinate commands that will break your setup.

@tannisroot I’m going to be trying out a custom YAML setup for HA soon. In the network section, does it matter if existing network devices are in the range of the 192.168.0.240/29 subnet? And is the .241 host address the address of your TrueNAS server?

The biggest issue I ran into using this was mdns discovery didnt work because mdns is tied to the top level IP on the box. I use time machine so disabling mdns wasnt an option. I have my fingers crossed the mdns avahi thing is going to get fixed for gold eye.

It does, you should change the DHCP lease upper range on your router to max out at 239 (including).
The .241 is the address of the TrueNAS server yes, I use that for communicating with local services like UPS NUT.

I’m getting a variety of errors when trying to set this up, attempting to work through them one at a time. When running the macvlan commands on the host they work until the last line, which throws “Error: Invalid prefix for given prefix length” which google tells me is probably netmask related. Here are my commands:

ip link add ha-host-shim link br0 type macvlan mode bridge
ip addr add 192.168.0.250/32 dev ha-host-shim
ip link set ha-host-shim up
ip route add 192.168.0.249/29 dev ha-host-shim

My primary home network is 192.168.0.0, my truenas server is 192.168.0.250. I do have a bridge setup as br0, the address for br0 is 192.168.0.250/24. Does the /24 netmask make sense there?

I was going to atttempt to setup HA with address 192.168.0.251

This is because you modified the original IP values.
In my example, the network is specified at 241/29, which has a range of 240-248, with 241 being the host and 242-246 being usable by other clients.
You set it to 249/29, which ends beyond the 8-bit range of the block (0-255).

Ah dang. The online cidr IP calculators showed your range as 240-247 and mine as 248-255 so I thought it might work. I’ll test if /30 works or if I need to lower my IP ranges closer to yours.
Thanks

@tannisroot You have been a big help, thank you. I have Home Assistant up and running in a macvlan and device discovery works. My remaining issue is connecting it to mosquitto. I have the z2mqtt and mosquitto apps from the app catalog installed and have had those working for a while. I can connect HA to mqtt when it is on the truenas network, but not now that it is in the macvlan. You mentioned earlier you are also using z2m and mosquitto, how do you have it setup to communicate?

Edit: after doing some additional reading I may understand what to try tomorrow. I believe my shim commands might be incorrect. This page explains the issue well.
https://blog.oddbit.com/post/2018-03-12-using-docker-macvlan-networks/

It sounds like you’re running into the typical issue with Docker’s Macvlan network where containers on different networks can’t communicate easily. Since you’re using Home Assistant in a Macvlan, it might not be able to access the Mosquitto broker on the same network.

If you haven’t already, check your shim commands for setting up the network bridge between the br0 interface and your Docker containers. The OddBit guide explains the issue and provides a good approach to fixing this.

In my case, I solved it by either:

  1. Using network_mode: host for Home Assistant to allow it to access the local network (if you’re using Docker in Proxmox).
  2. Ensure your Macvlan network config includes proper routing between containers.

Let me know if that helps, or if you’d like more details on my setup!

Got it working now using the shim setup. What I didn’t initially understand was that the .241 address is more of an ‘intermediate’ IP between the MACVLAN and host and not the actual host IP. Go figures, things work correctly when you actually follow directions. :sweat_smile:

Coming back to this after living with the setup for a while, here are the main things I learned.

The biggest clarification for me was that the host shim IP is not supposed to be the normal TrueNAS host IP. It is a separate reserved IP used so the TrueNAS host can talk to containers on the macvlan network.

I also would not edit avahi-daemon.conf on TrueNAS. I tried that path early on, but it is not update-safe. The better route for me was:

  1. Leave TrueNAS mDNS alone so Time Machine/SMB discovery keeps working.
  2. Put Home Assistant on a macvlan so it has its own LAN IP.
  3. Reserve a small IP range outside DHCP for Docker/macvlan.
  4. Add a host shim interface so TrueNAS can talk back to the macvlan containers.
  5. Make the shim commands persistent with a TrueNAS post-init script.

Example layout:

  • LAN: 192.168.1.0/24
  • Gateway: 192.168.1.1
  • TrueNAS normal IP: 192.168.1.x
  • Reserved macvlan range: 192.168.1.240/29
  • Shim IP: 192.168.1.241
  • Home Assistant IP: 192.168.1.242

Compose network example:

networks:
  homeassistant_net:
    driver: macvlan
    driver_opts:
      parent: br0
    ipam:
      config:
        - subnet: 192.168.1.0/24
          ip_range: 192.168.1.240/29
          gateway: 192.168.1.1
          aux_addresses:
            host: 192.168.1.241

Host shim commands:

ip link add ha-host-shim link br0 type macvlan mode bridge
ip addr add 192.168.1.241/32 dev ha-host-shim
ip link set ha-host-shim up
ip route add 192.168.1.240/29 dev ha-host-shim

One important correction from my earlier attempt: I would route only the reserved macvlan range, not the entire LAN subnet, through the shim. So in this example I would use:

ip route add 192.168.1.240/29 dev ha-host-shim

not:

ip route add 192.168.1.0/24 dev ha-host-shim

That keeps the shim narrowly scoped to the Docker macvlan addresses.

A couple other practical notes:

  • If HA needs to talk to MQTT, Z-Wave JS UI, Matter, etc., either put those related containers on the same macvlan network or make sure the routing path between them is actually valid.
  • With macvlan, port mappings are generally not needed for the service that has its own LAN IP.
  • Make sure the macvlan IP range is excluded from DHCP on the router, otherwise Docker can eventually collide with another LAN device.
  • The .241 style shim IP should be reserved for the host shim and should not be handed out to a container.

This ended up being the cleanest compromise for me: Home Assistant gets proper LAN behavior and discovery, TrueNAS mDNS stays enabled for Time Machine, and I do not have to rely on editing system files that may get reverted during updates.

I was running Opencanary in an Incus LXC with Macvlan and was forced to use a shim in order for my Uptime Kuma (running as a Docker container in same host), to be able to ping the LXC container.

This works ok, but, it does create “suspicion” on network clients running a good anti-malware, as they recognize the Macvlan IP address as a “duplicate” and may block connectivity to/from the LXC container.

I finally looked deeper into it and recently changed it to veth type and got rid of the shim and the issues.

I just looked in TN Beta 26 and the GUI allows to add existing bridge interferences to the now virtio container and allows its own IP address, so I will use that when I migrate to TN 26.

That is useful to know. I am actually on TN 26 beta now, so I may test this directly instead of continuing to polish the macvlan/shim approach.

My macvlan/shim setup was mostly me solving the problem with the tools I had at the time while keeping Home Assistant as a Docker container and preserving normal LAN behavior/discovery. It works, but I agree it is more of a workaround than an ideal network design.

The main thing I was trying to avoid was breaking TrueNAS mDNS/Time Machine discovery or editing Avahi config directly. The shim approach got Home Assistant reachable without touching that, but if TN 26 can attach the container to an existing bridge and give it its own LAN IP cleanly through the GUI, that sounds like the better path.

I am on TN 26 beta now, so I will probably try the bridge/veth approach and compare it against my current macvlan setup.

My main validation points will be:

1. Home Assistant gets its own LAN IP
2. mDNS/discovery still works
3. TrueNAS host can reach the HA container
4. Other Docker containers can reach HA
5. Time Machine / SMB discovery is not broken
6. No shim route needed

If that works cleanly, I would consider the TN 26 bridge/veth method the better recommendation going forward.

Note that TN 26 does allow to add bridges to the virtio LXC, so the veth nic is not needed at all.

That is useful to know. I missed that TN 26 can attach the virtio LXC NIC directly to a bridge now.

I ended up moving Home Assistant into an Ubuntu LXC on TN 26 beta and gave the container its own LAN IP through the bridge. That avoided the old macvlan/shim approach entirely.

So far the important bits are working: HA is reachable on the LAN, mDNS/discovery is behaving, and the host can still reach the container. That was the main thing I was trying to solve with the shim setup before.

So yes, if the bridge attachment is available in TN 26, that seems like the much cleaner path going forward.