TrueNAS 26 Beta 1 LXC: Docker real images fail with “failed to register layer: operation not permitted”; /dev/fuse unavailable

Apps and Virtualization
21

Interesting, I can’t repro that particular error (can’t apply capabilities: operation not permitted). Retested with new LXCs to make sure I had clean setup. Exact steps I’m following:

  1. Start with a clean install of BETA.2. Nothing done by making a pool and enabling “containers” on it.
  2. Manually run sudo sysctl net.ipv6.conf.all.forwarding=1 on the host to ensure LXCs have IPv6 connectivity (unrelated bug).
  3. Install an LXC using Debian Trixie image, configured with ID Map Type = Privileged and Capabilities Policy = Allow All.
  4. Install Docker in the LXC.
  5. Configure an IPv6 subnet for Docker’s bridge network (just using fd00::/64 for testing) by adding this to /etc/docker/daemon.json: {"ipv6": true, "fixed-cidr-v6": "fd00::/64"}. (Otherwise, running Docker containers fails with failed to disable IPv6 on container's interface eth0 errors, presumably since the LXC can’t change sysctls on its network interfaces. Again, I think this is unrelated to the other issues.)
  6. systemctl restart docker.

At that point, running hello-world works for me (but only if I pass --security-opt apparmor=unconfined to docker run, else I get some AppArmor error I don’t presently understand). However, running docker pull lscr.io/linuxserver/radarr:latest fails with mount callback failed on /var/lib/containerd/tmpmounts/containerd-mount152488362: operation not permitted as it did for me before.

To work around the remaining issues:

  1. Manually edit the libvirt XML to add the “identity” idmap (<idmap><uid start='0' target='0' count='2147483648'/><gid start='0' target='0' count='2147483648'/></idmap>).
  2. Restart the LXC (from virsh so TrueNAS doesn’t overwrite your changes to the XML).
  3. In the LXC, systemctl stop docker && systemctl stop containerd, delete /var/lib/docker & /var/lib/containerd, and then restart systemctl start containerd && systemctl start docker. (Things get wonky if you had a failed Docker setup before the XML editing; blowing away the Docker state seems to fix this.)

After that, docker run --rm hello-world works with no AppArmor flag required, and docker pull lscr.io/linuxserver/radarr:latest completes successfully.

I’m curious at what point your testing steps diverge from mine, since you get an error that I never encountered (amongst the various others I did hit :slight_smile: ).

1 Like
22

It looks like this recently submitted PR should help…. Evidently the issue was that even “privileged” containers didn’t actually get CAP_MKNOD. (I still don’t fully understand why making a “dummy” user namespace worked around that, but in any case, this seems like a better fix.)

Should be able to test this out in a TrueNAS 26 nightly and see if things work any better….