Linux Jails (sandboxes / containers) with Jailmaker

truenas-fan · June 8, 2024, 9:07pm

Sorry, “odd network behavior” is not an IT term.
If not documented somewhere, you don’t remember the specific odd issues?

I have been running that jail with that config, for 3 months now and I have been very happy with the stability.

I believe that you did observe issues, but without any idea of what kind of issues, so we can either agree/disagree or test/confirm, it’s just words.

Thanks for the work/tool, I just never go for “because” as there has to be a technical reason, one way or another.

skittlebrau · June 8, 2024, 11:12pm

Well that would be because I’m a graphic designer and not an IT person

For more background, my bridges were bound to VLAN interfaces and there were four in total. I then assigned all of these to the container using macvlan (also tried ipvlan) and then had docker networks inside the container using these connections.

By odd I mean dropped packets after intermittent periods of time (hours or sometimes days) or complete loss of connection after heavy network traffic.

I asked Jip-Hop about it at the time and he suggested that combining macvlan/ipvlan was the cause.

truenas-fan · June 9, 2024, 3:39am

Thanks. That gives us direction.

I can almost remember some very odd TCP/network issues at the beginning of SCALE.
I still observe some sporadic ping timeouts to the TN SCALE IPs when I upgrade/restart certain apps, which had nothing to do with jails, correct?

But, I believe I have observed certain other sporadic network issues that COULD be explained by my specific setup, as you said.

We are going to get to the bottom of this, cause it is, or it is not.
(but it has NEVER been bad enough that it “suddenly” stopped communicating!).

(I went to this rabbit hole 1, 2 years ago, trying to figure out how, as there was NOT, and STILL, no DEFINITIVE guide for semi-complex, VLAN included setups!).

LarsR · June 9, 2024, 9:42pm

So i’ve now migrated most of my apps from truecharts to jailmaker and portainer and i must say i’m impressed at how much less resources jailmaker uses. There’s only three apps left for me to migrate, but that will have to wait until my holiday starts. The most tricky for me was how to figure out traefik and ingress for the apps, but as soon as i’ve had it figured out on one app the setup was then pretty easy. I don’t even know if i’ll migrate to the native docker solution or just keep portainer running in the jail.

There’s one question i still have: If i want to add a gpu to more then one app, do i have to increase the count from 1 to 2, 3 ,4 or 5?

skittlebrau · June 9, 2024, 11:58pm

Are you referring to the config/settings for a jailmaker jail? ‘1’ just means yes/true and isn’t like the GPU allocation setting in TrueNAS Apps. You don’t need to worry about allocating resources for the GPU in the same way.

If Intel GPU/iGPU, then it simply passes /dev/dri whereas NVIDIA it passes different directories (I forget which). In my particular case, I have an Intel iGPU and Intel Arc GPU, so both are available in the jail.

rexzhang · June 14, 2024, 6:30am

If you want to use a specific MAC address in Macvlan mode so that the router can assign a fixed IP address via DHCP, you can refer to this setting

jails/docker/config

systemd_nspawn_user_args=--system-call-filter='add_key keyctl bpf'
    --network-macvlan=enp5s0

in jail /etc/systemd/network/mv-dhcp.network

[Match]
Virtualization=container
Name=mv-*

[Link]
MACAddress=C0:00:00:20:00:62

[Network]
DHCP=yes
LinkLocalAddressing=ipv6

[DHCPv4]
UseDNS=true
UseTimezone=true

enp5s0 is any of the host(TrueNAS)'s NICs
C0:00:00:20:00:62 is the specified MAC address
You can also set up different virtual NICs for multiple Jails on the same NIC

Thank @Jip-Hop

x-cimo · June 14, 2024, 12:23pm

Hello!

Jailmaker is looking really promising. I am working on isolation for increase security. I got user namespacing working, my next step is network.

I am currently using macvlan, it already prevents the jail to connect to the host which is good for me.

Is there a way to limite what the jail can access on my local network? I’ve tried with iptables but it seems with macvlan packet are dumped on the interface at a lower level than iptables.

Alternatively, perhaps a bridge interface would work instead? This is for a 10gbps NIC so I’m hoping to not hurt performance either.

Thanks

Jip-Hop · June 14, 2024, 3:31pm

Are you trying to firewall the jail by adding iptables rules directly on the SCALE host? In case of bridge networking, the SCALE host is forwarding traffic generated by the processes in the container. But the traffic is generated in it’s own network namespace. The OUTPUT chain doesn’t catch packets coming out of a jail. The FORWARD chain should.

When using macvlan, the packets never pass through the host net namespace. You could add the iptables rules inside the jail… but then the jail root user can wipe them. Something else I didn’t experiment with but may work in this case is to use: IPAddressDeny and IPAddressAllow systemd options.

Jip-Hop · June 14, 2024, 3:45pm

I just created a jail using macvlan networking and started it with IPAddressDeny and IPAddressAllow added: systemd-run --property="IPAddressDeny=any" --property="IPAddressAllow=8.8.8.8" --collect --property=Delegate=yes --property=RestartForceExitStatus=133 .... Seems to work great to control which IP addresses the jail can access.

You’d have to add these flags to the systemd_run_default_args in the config file for now.

Edit: I forgot I already tried something similar before.

x-cimo · June 14, 2024, 5:25pm

What I tried is something similar to what I was doing on the Kubernetes bridge interface (which I know if completely different).

This did not work:
iptables -i macvtap3 -I INPUT 1 -p icmp -d 10.100.0.50 -j ACCEPT
iptables -i macvtap3 -I INPUT 2 -p icmp -d 10.100.0.0/16 -j DROP

Basically I’m hoping to restrict what the jail can access on my network on the TrueScale Host.

I will test your solution as it seem very straight forward!!

x-cimo · June 14, 2024, 8:03pm

So the issue with this approach is that it seems to apply to both egress and ingress
systemd.resource-control (www.freedesktop.org)

I started a netcat to listen in the jail: nc -l -p 8888

and added the following args:
-p IPAddressAllow=10.100.0.1 -p IPAddressDeny=10.100.0.0/16

the allow permit traffic to my gateway (for internet), and block everyting from and to my network. so netcat become unreachable. This might be viable if i tunnel everything from a reverse proxy outside the jail.

Otherwise I might have to go to bridge with FORWARD iptables like you suggested above.

rexzhang · June 15, 2024, 7:28am

Perhaps you can use a MAC address/port based ACL setup on the switch.

In my case, my jail (macvlan) uses a different physical NIC than the Host (TrueNAS) system.Then you can put jails into a special subnet/vlan.

LarsR · June 15, 2024, 9:23am

Does anyone know of a docker-compose file that does the same as the
external-service app from truecharts? So let you access a service on your network via traefik that usually can’t be added because you can’t add ingress to the service itself?

sakcaj · June 15, 2024, 12:22pm

Hi All,

Does anyone know how to passthrough USB device from host via jailmaker? I’ve been struggling for a few days now - trying to pass USB Zigbee stick, it’s on /dev/ttyUSB0 on the host. I’ve been trying so many things now I do not think there’s sense listing them all since nothing works now.

I was able to make it work at some point, did not note what I’ve done as there was a number of things, and looks like it was not a persistent setting as after I’ve started working on another container and had to make few restarts in the meantime my Z2M container stopped working, erroring out that the /dev/ttyUSB0 is not found.

etorix · June 15, 2024, 1:35pm

I’m quite sure you have to pass the whole USB controller through, not the device.

sakcaj · June 15, 2024, 4:36pm

Jeez, do you have any other hints? I’ve been googling and trying different things for a few h now since your post and do not seem to get any further. I can see following in the container when looking for *ttyUSB0:

root@docker:/# find -name *ttyUSB0
./sys/bus/usb-serial/devices/ttyUSB0
./sys/bus/usb-serial/drivers/cp210x/ttyUSB0
./sys/class/tty/ttyUSB0
./sys/devices/pci0000:00/0000:00:14.0/usb1/1-2/1-2:1.0/ttyUSB0
./sys/devices/pci0000:00/0000:00:14.0/usb1/1-2/1-2:1.0/ttyUSB0/tty/ttyUSB0
./run/udev/data/+usb-serial:ttyUSB0

Here’s my jlmkr’s config, commented out various items that I’ve tried so far:

systemd_nspawn_user_args=
        --network-bridge=br01
        --resolv-conf=bind-host
        --system-call-filter='add_key keyctl bpf'
        --capability=all
        --bind-ro=/run/udev
        --bind='/mnt/SSD Storage/docker/data:/mnt/data'
        --bind='/mnt/SSD Storage/docker/stacks:/opt/stacks'
        #--bind='/dev/dri/renderD128:/mnt/dri/renderd128'
        #above is for Frigate, passing iCPU
        --device=/dev/ttyUSB0:/dev/ttyUSB0:rwm
#--bind='/dev/serial/by-id/usb-ITead_Sonoff_Zigbee_3.0_USB_Dongle_Plus_e053af01b919ec1193eb37cc47486eb0-if00-port0:/dev>        #--bind='/dev/zigbee:/mnt/zigbeeDongle'
        #--bind='/run/udev:/mnt/udev'
        --bind='/mnt/HDD/HDD_CAMS_SMB:/mnt/media'
        #--DeviceAllow=/dev/ttyUSB0 rwm
        --device '/dev/bus/usb:/dev/bus/usb'

Here’s my USB device on host, I’ve amended the permissions via udev rule:

root@truenas[~]# ls -l /dev/ttyUSB0
crw-rw-rw- 1 root dialout 188, 0 Jun 15 18:23 /dev/ttyUSB0

Jip-Hop · June 15, 2024, 5:07pm

I’d expect adding only --bind=/dev/serial/by-id/usb-ITead_Sonoff_Zigbee_3.0_USB_Dongle_Plus_e053af01b919ec1193eb37cc47486eb0-if00-port0 should work. Get rid of --device and --DeviceAllow. You’d then use this path instead of /dev/ttyUSB0 in your compose file and other config. Basically what wisekki suggested.

That’s probably how you’d do it with VMs, but not with containers AFAIK.

sakcaj · June 15, 2024, 5:33pm

Hey, thanks for your input. I just saw wisekki’s input on the other thread, I’ve made it exactly as you said but it’s still not working. The full dongle name was somehow cut off while copying from terminal, that’s in regards to what he pointed out.

systemd_nspawn_user_args=
        --network-bridge=br01
        --resolv-conf=bind-host
        --system-call-filter='add_key keyctl bpf'
        --capability=all
        --bind-ro=/run/udev
        --bind='/mnt/SSD Storage/docker/data:/mnt/data'
        --bind='/mnt/SSD Storage/docker/stacks:/opt/stacks'
        --bind='/mnt/HDD/HDD_CAMS_SMB:/mnt/media'
#--bind='/dev/dri/renderD128:/mnt/dri/renderd128'
        #above is for Frigate, passing iCPU
        #--device=/dev/ttyUSB0:/dev/ttyUSB0:rwm
        --bind='/dev/serial/by-id/usb-ITead_Sonoff_Zigbee_3.0_USB_Dongle_Plus_e053af01b919ec1193eb37cc47486eb0-if00-port0
        #--bind='/dev/zigbee:/mnt/zigbeeDongle'
        #--bind='/run/udev:/mnt/udev'
        #--DeviceAllow=/dev/ttyUSB0 rwm
        #--device '/dev/bus/usb:/dev/bus/usb'
# Script to run on the HOST before starting the jail
# Load kernel module and config kernel settings required for docker

obviously I’ve restarted the container,

ls -la from container

root@docker:~# ls -la /dev/serial/by-id/
ls: cannot access '/dev/serial/by-id/': No such file or directory

ls -la from host - what I’ve just noticed is that the permissions and groups are different between ttyUSB0 and in by-id - does that make any difference?

root@truenas[~]# ls -l /dev/ttyUSB0
crw-rw-rw- 1 root dialout 188, 0 Jun 15 18:23 /dev/ttyUSB0

lrwxrwxrwx 1 root root 13 Jun 15 18:23 usb-ITead_Sonoff_Zigbee_3.0_USB_Dongle_Plus_e053af01b919ec1193eb37cc47486eb0-if00-port0 -> ../../ttyUSB0

Jip-Hop · June 15, 2024, 6:00pm

Your config still contains a syntax error. --bind='/dev/se... starts with a ' but doesn’t end with one.

Jip-Hop · June 15, 2024, 6:02pm

P.S. There’s no need to do this manually. You can use gpu_passthrough_intel=1 for that.