TrueNAS 26.0.0-BETA.2 is Now Available!

Playing around with LXC containers on a test system (as it seems like there have been a number of fixes and improvements there), it seems that IPv6 connectivity with the default bridge (truenasbr0) may be broken.

On a clean install, I verified the host has both IPv4 and IPv6 connectivity, and then I created an LXC using the Debian Trixie image. IPv4 in the container works by default. (It looks like TrueNAS runs dnsmasq behind the scenes to provide DHCP and DNS servers to the bridge.) IPv6 doesn’t, because there’s no default route in the container. Not a TrueNAS problem; the Debian LXC image is not configured to accept RAs by default.

But if I modify /etc/systemd/network/eth0.network in the container (the Debian LXC image uses systemd-networkd by default, which handles RAs internally) to set IPv6AcceptRA=true, I do get a default route, but still no IPv6 connectivity. I think this may be a TrueNAS bug, but haven’t had time to dig into it yet.

Inside the container
root@test:/# ip addr show eth0
16: eth0@if17: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 52:54:00:ea:2c:e0 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 172.200.0.71/24 metric 1024 brd 172.200.0.255 scope global dynamic eth0
       valid_lft 3573sec preferred_lft 3573sec
    inet6 fd42:4c58:43ae:0:5054:ff:feea:2ce0/64 scope global mngtmpaddr noprefixroute 
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:feea:2ce0/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever

root@test:/# ip -6 route
fd42:4c58:43ae::/64 dev eth0 proto ra metric 1024 pref medium
fe80::/64 dev eth0 proto kernel metric 256 pref medium
default nhid 4048809722 via fe80::8024:f0ff:feb0:25e5 dev eth0 proto ra metric 1024 expires 1790sec pref medium

root@test:/# ping -c1 -4 google.com
PING google.com (142.251.210.206) 56(84) bytes of data.
64 bytes from pnorda-ak-in-f14.1e100.net (142.251.210.206): icmp_seq=1 ttl=115 time=10.7 ms

--- google.com ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 10.744/10.744/10.744/0.000 ms

root@test:/# ping -c1 -6 google.com
PING google.com (2607:f8b0:4009:811::200e) 56 data bytes

--- google.com ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms

root@test:/# ping -c1 fe80::8024:f0ff:feb0:25e5
PING fe80::8024:f0ff:feb0:25e5 (fe80::8024:f0ff:feb0:25e5) 56 data bytes
64 bytes from fe80::8024:f0ff:feb0:25e5%eth0: icmp_seq=1 ttl=64 time=0.036 ms

--- fe80::8024:f0ff:feb0:25e5 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.036/0.036/0.036/0.000 ms
Outside the container
bcat@benchy:~$ ip addr show truenasbr0

7: truenasbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 82:24:f0:b0:25:e5 brd ff:ff:ff:ff:ff:ff
    inet 172.200.0.1/24 brd 172.200.0.255 scope global truenasbr0
       valid_lft forever preferred_lft forever
    inet6 fd42:4c58:43ae:0:8024:f0ff:feb0:25e5/64 scope global dynamic mngtmpaddr proto kernel_ra
       valid_lft forever preferred_lft forever
    inet6 fd42:4c58:43ae::1/64 scope global
       valid_lft forever preferred_lft forever
    inet6 fe80::8024:f0ff:feb0:25e5/64 scope link proto kernel_ll
       valid_lft forever preferred_lft forever

bcat@benchy:~$ sudo nft list ruleset
table inet truenas {
        chain pstrt.truenasbr0 {
                type nat hook postrouting priority srcnat; policy accept;
                ip saddr 172.200.0.0/24 ip daddr != 172.200.0.0/24 masquerade
                ip6 saddr fd42:4c58:43ae::/64 ip6 daddr != fd42:4c58:43ae::/64 masquerade
        }

        chain fwd.truenasbr0 {
                type filter hook forward priority filter; policy accept;
                ip version 4 oifname "truenasbr0" accept
                ip version 4 iifname "truenasbr0" accept
                ip6 version 6 oifname "truenasbr0" accept
                ip6 version 6 iifname "truenasbr0" accept
        }

        chain in.truenasbr0 {
                type filter hook input priority filter; policy accept;
                iifname "truenasbr0" tcp dport 53 accept
                iifname "truenasbr0" udp dport 53 accept
                iifname "truenasbr0" icmp type { destination-unreachable, time-exceeded, parameter-problem } accept
                iifname "truenasbr0" udp dport 67 accept
                iifname "truenasbr0" icmpv6 type { destination-unreachable, packet-too-big, time-exceeded, parameter-problem, nd-router-solicit, nd-neighbor-solicit, nd-neighbor-advert, mld2-listener-report } accept
                iifname "truenasbr0" udp dport 547 accept
        }

        chain out.truenasbr0 {
                type filter hook output priority filter; policy accept;
                oifname "truenasbr0" tcp sport 53 accept
                oifname "truenasbr0" udp sport 53 accept
                oifname "truenasbr0" icmp type { destination-unreachable, time-exceeded, parameter-problem } accept
                oifname "truenasbr0" udp sport 67 accept
                oifname "truenasbr0" icmpv6 type { destination-unreachable, packet-too-big, time-exceeded, parameter-problem, nd-router-solicit, nd-neighbor-solicit, nd-neighbor-advert, mld2-listener-report } accept
                oifname "truenasbr0" udp sport 547 accept
        }
}

So the container receives an RA and gets an IPv6 gateway it can ping, but can’t actually route through it. Unclear as yet why.

Edit: The IPv6 default route in the container also disappears after some time. Probably will have to get some pcaps to determine why (which may not be a tonight problem). It’s possible this is just a network setup issue in the container, but since I’m not doing anything fancy in there at all, I’m still suspicious it’s a host issue.

Oh, it seems IPv6 forwarding is just disabled on the TrueNAS host by default, even when containers are configured:

bcat@benchy:~$ sysctl net.ipv4.ip_forward
net.ipv4.ip_forward = 1

bcat@benchy:~$ sysctl net.ipv6.conf.all.forwarding
net.ipv6.conf.all.forwarding = 0

Flipping net.ipv6.conf.all.forwarding to 1 as a test seems to give the container IPv6 connectivity. (Probably this is only an issue because this is a clean TrueNAS install without Docker configured, which would enable IPv6 forwarding itself.)

I’ll file a bug about this. (Still unclear if the disappearing default route in the container was related or separate issue.)

Edit: Filed NAS-141468.

Regarding the reboot cause logging - does it apply to shutdowns as well? For example, will it log a shutdown initiated by a UPS?

Created another ticket/bug report just to have it closed automatically again due to bug clerk saying duplicate of NAS-140984.

New bug report is Jira - NAS-141469

We can’t view the original report, it’s not available to public, aka NAS-140984.

Have created this new report, did a fair bit of research/troubleshooting to obtain as much context and info the help diagnose and show what’s been checked. Attached debug to the report as well.

If this bug is already lodge since reports are closed as duplicates, would that not mean it’s a known issue to be worked on or? If not why does the bug system keep saying it’s duplicate, auto closes and then says can’t comment anymore and lodge another. So now I am in a circle and getting no where submitting the issue as requested again to do.

Windows SMD discovery broke. I got it up by restarting truenas-discoveryd once or twice but it craps out again right after.

Hello,

I am getting SMB share “Share Name” is unavailable because it uses a locked dataset.” after every boot on beta 2 (no problem on beta1)

However, everything works and the datasets are all unlocked.

1 Like

Updated from BETA 1 to 2.

I still get the below error when starting a container.

Validation Error

internal error: guest failed to start: Failure in libvirt_lxc startup: cannot find init path ‘/sbin/init’ relative to container root: No such file or directory

Error Name: EINVAL
Error Code: 22
Reason: internal error: guest failed to start: Failure in libvirt_lxc startup: cannot find init path '/sbin/init' relative to container root: No such file or directory
Error Class: libvirtError
Trace: Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/api/base/server/ws_handler/rpc.py", line 387, in process_method_call
    result = await method.call(app, id_, params)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/api/base/server/method.py", line 70, in call
    result = await self.middleware.call_with_audit(self.name, self.serviceobj, methodobj, params, app,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                                                   message_id=id_)
                                                   ^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1094, in call_with_audit
    result = await self._call(method, serviceobj, methodobj, params, app=app,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                              audit_callback=audit_callback_messages.append, **kwargs)
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 924, in _call
    return await self.run_in_executor(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<4 lines>...
    )
    ^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 804, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.13/concurrent/futures/thread.py", line 59, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/lib/python3/dist-packages/middlewared/api/base/decorator.py", line 217, in wrapped
    result = func(*args)
  File "/usr/lib/python3/dist-packages/middlewared/plugins/container/lifecycle.py", line 114, in start
    self.middleware.libvirt_domains_manager.containers.start(pylibvirt_obj)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/truenas_pylibvirt/domain/manager.py", line 77, in start
    if libvirtd_domain.create() < 0:
       ~~~~~~~~~~~~~~~~~~~~~~^^
  File "/usr/lib/python3/dist-packages/libvirt.py", line 1390, in create
    raise libvirtError('virDomainCreate() failed')
libvirt.libvirtError: internal error: guest failed to start: Failure in libvirt_lxc startup: cannot find init path '/sbin/init' relative to container root: No such file or directory

If I run:
sudo zfs inherit canmount z-flash/.truenas_containers/containers/Zabbix
sudo zfs mount z-flash/.truenas_containers/containers/Zabbix

The container will start but when running docker in the container and I attempt to start Zabbix I get the following error.

[root@Zabbix zabbix-docker]# docker compose --profile full -f ./docker-compose_v3_alpine_mysql_latest.yaml up -d
[+] Running 0/6
 â ¸ Container zabbix-docker-zabbix-agent-1         Starting                                                                                                                            0.3s
 â ¸ Container zabbix-docker-zabbix-web-service-1   Starting                                                                                                                            0.3s
 â ¸ Container zabbix-docker-zabbix-java-gateway-1  Starting                                                                                                                            0.3s
 â ¸ Container zabbix-docker-zabbix-snmptraps-1     Starting                                                                                                                            0.3s
 â ¸ Container zabbix-docker-db-data-mysql-1        Starting                                                                                                                            0.3s
 â ¸ Container zabbix-docker-mysql-server-1         Starting                                                                                                                            0.3s
Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: unable to apply cgroup configuration: mkdir /sys/fs/cgroup/docker/5ccd736196f480a56cef67adb651f1dfa47cfab1032c16e3616beb88e9806a4b: permission denied: unknown

The containers were running fine in 25. They have not worked sinse 26 BATA-1.

Any suggestions on where to look.

I had similar issues and for me it was permission mapping not working, however I tested stuff and did a workaround, so I reset permissions on Beta.2. So try reapplying the permissions recursively on any mapped datasets.

Permissions mapping for containers (idmapping) in BETA.1 was broken. In BETA.2 it’s fixed. This has implications for people who ran BETA.1, didn’t notice idmapping was 100% broken, and upgraded. Basically, you need to restamp permissions on the datasets that are mounted into idmapped containers.

1 Like

This particular container has no data sets mounted into the container.

The following dataset is visible which it never was before upgrading to BETA-1

The dataset is not mounted.

1 Like

How-To enable AAPL extension? Finder lookup very slow on macOS.

Try searching the online documentation for ‘AAPL extensions’ and see if you have your shares set where they are disabled. There are some notes in the 25.10 section when I did the search

1 Like

Cheers! Will update today.

This depends on context. MacOS is generally quite poor performing if you have large file counts in directories. This is because of the quantity of SMB request it makes in order to fulfill a directory listing. For example, if Windows makes say 5 requests to list a directory, and MacOS makes 5 + 2N (where N is the number of files in a directory) then MacOS will always be slower and performance will become worse as directory size grows. This is over-simplified, but it’s not hard to take a pcap and watch per-file operations that occur when you list a directory in MacOS that don’t occur in Windows or Linux clients.

Apple has some tuning instructions for this case: Adjust SMB browsing behavior in macOS - Apple Support

1 Like

I had the exact opposite. I upgraded to BETA.2 and none of my apps are there anymore

How do I get the app pool back?

Have you tried unsetting the app pool and then setting the app pool again in the TrueNAS GUI?

1 Like

After reboot of a system with encrypted datasets, a spurious alert shows in the GUI that shares related to the dataset are unavailable. The dataset is unlocked and the share is available. New beta 2 behaviour.

1 Like

You can use Report a Bug and attach a Debug Dump. I can’t tell if a previous ticket has the Debug Dump attached and may get closed without. At worst, a new ticket will get closed as a duplicate

I am getting the exact same issue on Beta 2 that was no in Beta 1 across 2 different TrueNAS systems.

Everything does work though.