Docker not supported under running under containers

TrueNAS General
1

I posted a bug on Jira which was closed very quickly as an upstream issue. Fair enough.

What I did find interesting as the ix comment about docker not being supported.

“Note also that running Docker nested inside an LXC sandbox is not a supported configuration on TrueNAS - the supported path for containerised workloads is the Apps feature.”

I was / am planning on running portainer under an LXC container to replace my jailmaker setup that I currently run.

I have been testing trying to run docker under a container and failing. Something do do with the way that the container is interacting with the host (/proc/sys is RO). I have tried multiple distros with no luck.

docker run hello-world was failing, but docker run --network host hello-world was working.

I have a possible workaround - which I shall do when I get around to it

2

On which version of truenas did you try to run docker insdie an lxc?
If it’s still on 25.10 and the incus backend, there’s a bug in docker 29 which isn’t fixed in the incus version that truenas uses.
I still run docker 28 inside my incus lxc and it works

3

Can you post the ticket number? This sounds like a misunderstanding of feature support.

Version matters too, as @LarsR said, but if you’re on 26, documentation for running nested containers like Docker inside an LXC is here: https://www.truenas.com/docs/scale/26/containers/managingcontainers/#running-nested-containers

4

NAS-142351 Was closed very quickly. Jira

I am testing on 26 Beta 3

5

I can report that I have docker and portainer working on TN 26 Beta 3.

I don’t know what the long term effects are of the minor bodgery I have had to do however here it is for anyone interested.

Process:

  1. Install Container. I used Privileged & Allow All. I also used Debian Bookworm
  2. Install Docker - ignore the generated errors that don’t seem to matter as the install will continue
  3. “docker run hello-world” should fail
  4. Now run “mount -o remount,rw /proc/sys”, followed by “docker run hello-world” - which should work
  5. Note that this will not survive a reboot so

Note: Do not copy and paste below - that won’t work. Blasted forum keeps reformating what I write and removing the spaces I was using to indicate that this is what you need to enter into the edited file. So I have used italics instead

apt install nano

nano /etc/systemd/system/procsys-rw.service
[Unit]
Description=Remount /proc/sys read-write for Docker networking
DefaultDependencies=no
Before=docker.service
After=local-fs.target

[Service]
Type=oneshot
ExecStart=/bin/mount -o remount,rw /proc/sys
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target

Save the File

systemctl daemon-reload
systemctl enable procsys-rw.service
systemctl start procsys-rw.service

systemctl edit docker
[Unit]
Requires=procsys-rw.service
After=procsys-rw.service

Save the File

systemctl daemon-reload
systemctl restart docker
reboot

mount | grep "proc/sys " #Confirm this shows rw
docker run hello-world #Confirm this runs

I hope this makes sense. More testing will be required though

6

Thou shall invoke the magic of the Three Backquotes…

7

I am very glad to hear that (although the documentation if followed is not sufficient). My whole path forward is to replace jailmaker with containers

8

Yeah - it worked under incus (TN 25) - but not in the current system in 26 Beta 1,2 or 3

9

That right there has been my worry about the new containers!

I have an Ubuntu incus container, running OpenCanary and my own arp-scan discovery/reporting solution tied to multiple networks/vlans, since jailmaker came out in TN.
I never worry about it, it just works!

I have NOT been impressed, or convinced by the features lacking and adding features as the Beta matures (that’s not a beta release!).

So my suspicion about the better containers solution, keep being proven right when we continue to see basic setups not working (tells me they are not testing or using them fully).

At this point, I don’t want to continue to test or worry about the new virtio containers and just hope that jailmaker continues working, until virtio containers can do everything that jailmaker can do today!

10

Ah ok I see on the linked duplicate ticket that there’s an upstream docker issue making it not play nice with LXC: Jira.

As part of a security fix (CVE-2024-32473), Docker attempts to disable IPv6 on IPv4-only networks. However, in LXC containers, /proc/sys is mounted as read-only, which prevents Docker from applying this change and results in the failure.

11

Thus my “solution” is to mount /proc/sys as rw instead. Which fixes the problem

More alarming was the “its unsupported to run docker on a container”

12

Understood. That would be an erroneous bot response, that or it was confusing “supported” meaning possible with "supported meaning what we would provide support for to Enterprise customers.

13

I just installed 27.0.0-RC.1

This is still an issue under RC.1 - docker doesn’t work.

However the bodge / fix / workaround above does still work

root@docker-test-rc1:/# docker run hello-world

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (amd64)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://hub.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/get-started/

root@docker-test-rc1:/#