TrueNAS 26 Beta 1 LXC: Docker real images fail with “failed to register layer: operation not permitted”; /dev/fuse unavailable

I am testing TrueNAS 26 Beta 1 containers/LXC and ran into a Docker-in-LXC issue.

The LXC itself works: Ubuntu boots, networking works, SSH works, Docker starts, and hello-world runs. But real multi-layer images fail during layer extraction/registration. fuse-overlayfs also appears blocked because /dev/fuse is not available inside the LXC.

Environment

  • TrueNAS 26 Beta 1
  • TrueNAS managed LXC / container.* / libvirt LXC
  • Guest OS: Ubuntu 24.04.4 LTS
  • Privileged container
  • Capabilities policy: Allow All
  • Init process: /sbin/init
  • VirtIO NIC attached to bridge
  • Docker installed inside the LXC with Docker’s official install script
  • Docker Compose: v5.1.3

What works

docker run --rm hello-world

This works and prints the normal Hello from Docker! message.

Docker Compose also works:

docker compose version
Docker Compose version v5.1.3

What fails

A real image fails:

docker pull lscr.io/linuxserver/radarr:latest
docker run --rm lscr.io/linuxserver/radarr:latest /bin/sh -c 'echo radarr image works'

Failure:

latest: Pulling from linuxserver/radarr
ccc6d199fc4b: Extracting [==================================================>]  6.275MB/6.275MB
f6a4c3e338ed: Download complete
4c9d9a005bd6: Download complete
f63ecd910bdc: Download complete
5afbe1a83c1a: Download complete
bc8e3ac89381: Download complete
fef015f8fe2b: Download complete
41d7626389d1: Download complete
c1e5bb986614: Download complete
failed to register layer: operation not permitted

A larger Docker Compose stack also failed with:

failed to extract layer ... to overlayfs ...
mount callback failed on /var/lib/containerd/tmpmounts/containerd-mount...: operation not permitted

Things tried

Docker bridge IPv6

Docker initially failed trying to disable IPv6 on container interfaces, so I configured:

{
  "ipv6": true,
  "fixed-cidr-v6": "fd00:dead:beef:11::/64"
}

After that, hello-world worked.

fuse-overlayfs

Installed fuse-overlayfs and configured Docker:

{
  "ipv6": true,
  "fixed-cidr-v6": "fd00:dead:beef:11::/64",
  "storage-driver": "fuse-overlayfs"
}

Docker reported fuse-overlayfs, but containers failed with:

fuse: device not found, try 'modprobe fuse' first
fuse-overlayfs: cannot mount: No such file or directory

So /dev/fuse does not seem available inside the LXC.

vfs storage driver

Configured:

{
  "ipv6": true,
  "fixed-cidr-v6": "fd00:dead:beef:11::/64",
  "storage-driver": "vfs"
}

Docker showed:

Storage Driver: vfs
Docker Root Dir: /var/lib/docker

hello-world worked, but real images still failed with:

failed to register layer: operation not permitted

Disabled Docker containerd snapshotter

Also tried:

{
  "ipv6": true,
  "fixed-cidr-v6": "fd00:dead:beef:11::/64",
  "storage-driver": "vfs",
  "features": {
    "containerd-snapshotter": false
  }
}

After clearing /var/lib/docker and /var/lib/containerd, restarting Docker, and retesting, hello-world still worked but Radarr still failed with:

failed to register layer: operation not permitted

Question

Since this is TrueNAS 26 Beta 1, I am trying to understand whether this is expected beta behavior, a missing configuration option, or a bug worth reporting.

Specifically:

  1. Is Docker inside a TrueNAS-managed LXC expected to support real multi-layer images?
  2. Is there a supported way to expose /dev/fuse to a TrueNAS LXC?
  3. Is fuse-overlayfs expected to work?
  4. Are extra LXC/libvirt settings needed for Docker image layer extraction?
  5. Should Docker-in-LXC be avoided on TrueNAS containers except for trivial images?
  6. Is a VM the recommended path if I want Docker isolation?

Current takeaway: the LXC works well for native Linux services, but I cannot currently use it as a reliable Docker host. hello-world works, but real application images fail during layer extraction.

You are doing better than I have recently. I haven’t even managed to get hello world to run. I have tried several LXC images (not all of them)

1 Like

That is helpful context. In my case hello-world only started working after I enabled IPv6 in Docker’s daemon.json; before that Docker was failing when it tried to disable IPv6 on the container interface.

This is what got me past that first hurdle:

{
  "ipv6": true,
  "fixed-cidr-v6": "fd00:dead:beef:11::/64"
}

Quick update: I tested this again with a clean Debian 13 Trixie LXC.

Environment:

  • TrueNAS 26 Beta 1
  • Debian GNU/Linux 13 Trixie LXC
  • Privileged container
  • Capabilities policy: Allow All
  • VirtIO NIC attached to bridge
  • Docker installed using Docker’s official install script
  • Docker Engine 29.4.3
  • Docker Compose v5.1.3
  • Storage driver: overlayfs

Same behavior as Ubuntu:

  1. Docker initially failed on hello-world because Docker tried to disable IPv6 on the container interface.
  2. Adding this to /etc/docker/daemon.json fixed hello-world :
{
  "ipv6": true,
  "fixed-cidr-v6": "fd00:dead:beef:12::/64"
}
  1. After restarting Docker, hello-world worked.
  2. A real image still failed:
docker pull lscr.io/linuxserver/radarr:latest

Failure:

failed to extract layer (application/vnd.oci.image.layer.v1.tar+gzip sha256:ccc6d199fc4b45e9f838e26a86a6f524ea72095d94f160b4e7c5c1b8ec522ec4) to overlayfs as "extract-248583357-SJoT sha256:b35ae72661e8a5975a101e705cf33b6ac8841ddadb8e5ef8adf8edbf9dcff76c": mount callback failed on /var/lib/containerd/tmpmounts/containerd-mount1377361297: operation not permitted

So this does not appear to be Ubuntu-specific. I can reproduce the same failure pattern on both Ubuntu 24.04 and Debian 13 LXCs.

The LXC itself works, Docker can run trivial images after the IPv6 workaround, but Docker/containerd cannot extract/register real application image layers because the required mount operation is denied.

Unfortunately, I encountered the same problem. I successfully ran a Dockge, but when trying to run Qbittorrent, I got the error “failed to register layer: operation not permitted”.

I think I’ll try the nightly version later(

I have now got LXC containers to run - but DNS doesn’t work and the UI (for the console) is crap

Currently broken

Starting in 26 you need to configure a container as privileged in order to run a nested containers like docker inside an LXC. Curious to see if this helps any of you get further: https://www.truenas.com/docs/scale/26/containers/managingcontainers/#running-nested-containers.

There are some ID-mapping fixes targeted to BETA.2 that might be involved here as well.

Networking fix in BETA.2: Jira

I shall look forward to further testing when that comes available

Is there a fix to the shell as well? Making the cursor keys work correctly?

As a NAS (File Services) Beta.1 seems to be working well so far

I just gave the ticket a bump, I don’t know if it would land in BETA.2 or have to wait for the next release at this point.

I just spotted some movement on that one. Apparently RC.1

1 Like

Thanks. Just to separate the issues: my containers are already privileged with Allow All capabilities, and networking/DNS is working once configured manually.

The remaining issue is specifically Docker/containerd image extraction. hello-world works after the Docker IPv6 workaround, but real images fail during layer extraction with mount callback failed ... operation not permitted.

If the Beta 2 ID-mapping fixes are expected to affect nested Docker/containerd layer extraction, I can retest when Beta 2 lands. Otherwise this may need its own ticket separate from the DNS/networking issue.

In a quick test with BETA.2—I am also super interested in running Docker inside LXCs when 26 releases—it seems this is still broken in the same way:

failed to extract layer (application/vnd.oci.image.layer.v1.tar+gzip sha256:31e4d6f71ab5268fb1bd6f01a428f8fed0152673a89a50b401abb96b0d5aec43) to overlayfs as "extract-174963924-KOc9 sha256:092f51ea238196fa09de7d0b5e5f38f1bd7e006b0d2df79a4c56321fd397fa16": mount callback failed on /var/lib/containerd/tmpmounts/containerd-mount3277757157: operation not permitted

That’s using a privileged container with Capabilities Policy = Allow All.

So this does seem worth reporting a bug for. (But I’ve filed a mish-mash of odd networking-related bugs recently, and so I’ll leave this one for someone else to file, since I haven’t does as much testing here as others on this thread.)

1 Like

I also see a different form of unhappiness: with a privileged container on BETA.2, even hello-image image fails to start, this time with an apparmor error:

root@test:/# docker run --rm -it hello-world 
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
4f55086f7dd0: Pull complete 
d5e71e642bf5: Download complete 
Digest: sha256:96498ffd522e70807ab6384a5c0485a79b9c7c08ca79ba08623edcad1054e62d
Status: Downloaded newer image for hello-world:latest
docker: Error response from daemon: AppArmor enabled on system but the docker-default profile could not be loaded: running '/usr/sbin/apparmor_parser -Kr' failed with output: apparmor_parser: Unable to replace "docker-default".  Unknown error (30): Read-only file system

Edit: Starting the Docker container with --security-opt apparmor=unconfined addresses that issue, but it’s not clear to me if that’s the right path forward. And anyway, it’s a moot point if we still can’t pull nontrivial images.

1 Like

In Beta 2

I get

root@docker:~# sudo docker run hello-world
docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: unable to apply caps: can’t apply capabilities: operation not permitted

DNS works

The Shell cursor keys work

Docker installs, I am using a bridge interface and I do get an IP address which I can ping from externaly. I have applied the IPv6 patch, although there isn’t a file there in the first place - so I have no idea if its actually doing anything. I tried with the patch and that just winged about ipv6

So things is still broken AFAICT

[TrueNAS as a NAS still works well though]

Okay, I’ve found a very, very strange workaround. Create a container with ID Map Type = Privileged and Capabilities Policy = Allow All as before, then use virsh edit to manually add the following under <domain>…</domain>:

  <idmap>
    <uid start='0' target='0' count='2147483648'/>
    <gid start='0' target='0' count='2147483648'/>
  </idmap>

Then restart the container with virsh (not via TrueNAS GUI, or it’ll wipe out your changes).

After that, Docker works fine with the default storage driver! No need to pass /dev/fuse in and change to fuse-overlayfs. And since the idmap is basically 1:1, there’s no file permission conflicts vs. the host. A container run as UID 2999 should be able to read and write files on host mounts using that UID.

But… why?! It seems that with a new userns (even one that’s effectively a 1:1 mapping like I used above), Docker works fine. But I truly have no idea why it works, and unfortunately, there’s not many up-to-date docs about the libvirt variant of LXC (which it kind of seems like the rest of the world has moved on from?).

Anyway, TrueNAS will wipe out the custom container XML every time it touches the container (including, I think, on boot). So I think some TrueNAS fix is still needed here. But it’s… progress?

Sadly, the TrueNAS UI offers only very minimal configuration of UID and GID ranges. Not sure why it doesn’t just allow specifying lists of source:target:size tuples per container, like the underyling userns uses…. So one I don’t know of a good way to get that config in place manually.

This is not an error I encountered in my testing tonight. Are you making the container with Capabilities Policy = Allow All?

I filed NAS-141470 for this issue since it seems pretty definite that there is some bug here, and I mentioned the workaround above. Hopefully folks who understand libvirt-lxc can determine if this workaround is sane… and if not, at least it may help lead the way to a real fix.

1 Like

Yes