Linux Jails (sandboxes / containers) with Jailmaker

Sorry, “odd network behavior” is not an IT term.
If not documented somewhere, you don’t remember the specific odd issues?

I have been running that jail with that config, for 3 months now and I have been very happy with the stability.

I believe that you did observe issues, but without any idea of what kind of issues, so we can either agree/disagree or test/confirm, it’s just words.

Thanks for the work/tool, I just never go for “because” as there has to be a technical reason, one way or another.

Well that would be because I’m a graphic designer and not an IT person :wink:

For more background, my bridges were bound to VLAN interfaces and there were four in total. I then assigned all of these to the container using macvlan (also tried ipvlan) and then had docker networks inside the container using these connections.

By odd I mean dropped packets after intermittent periods of time (hours or sometimes days) or complete loss of connection after heavy network traffic.

I asked Jip-Hop about it at the time and he suggested that combining macvlan/ipvlan was the cause.

1 Like

Thanks. That gives us direction.

I can almost remember some very odd TCP/network issues at the beginning of SCALE.
I still observe some sporadic ping timeouts to the TN SCALE IPs when I upgrade/restart certain apps, which had nothing to do with jails, correct?

But, I believe I have observed certain other sporadic network issues that COULD be explained by my specific setup, as you said.

We are going to get to the bottom of this, cause it is, or it is not.
(but it has NEVER been bad enough that it “suddenly” stopped communicating!).

(I went to this rabbit hole 1, 2 years ago, trying to figure out how, as there was NOT, and STILL, no DEFINITIVE guide for semi-complex, VLAN included setups!).

So i’ve now migrated most of my apps from truecharts to jailmaker and portainer and i must say i’m impressed at how much less resources jailmaker uses. There’s only three apps left for me to migrate, but that will have to wait until my holiday starts. The most tricky for me was how to figure out traefik and ingress for the apps, but as soon as i’ve had it figured out on one app the setup was then pretty easy. I don’t even know if i’ll migrate to the native docker solution or just keep portainer running in the jail.

There’s one question i still have: If i want to add a gpu to more then one app, do i have to increase the count from 1 to 2, 3 ,4 or 5?

1 Like

Are you referring to the config/settings for a jailmaker jail? ‘1’ just means yes/true and isn’t like the GPU allocation setting in TrueNAS Apps. You don’t need to worry about allocating resources for the GPU in the same way.

If Intel GPU/iGPU, then it simply passes /dev/dri whereas NVIDIA it passes different directories (I forget which). In my particular case, I have an Intel iGPU and Intel Arc GPU, so both are available in the jail.

1 Like

If you want to use a specific MAC address in Macvlan mode so that the router can assign a fixed IP address via DHCP, you can refer to this setting

jails/docker/config

systemd_nspawn_user_args=--system-call-filter='add_key keyctl bpf'
    --network-macvlan=enp5s0

in jail /etc/systemd/network/mv-dhcp.network

[Match]
Virtualization=container
Name=mv-*

[Link]
MACAddress=C0:00:00:20:00:62

[Network]
DHCP=yes
LinkLocalAddressing=ipv6

[DHCPv4]
UseDNS=true
UseTimezone=true
  • enp5s0 is any of the host(TrueNAS)'s NICs
  • C0:00:00:20:00:62 is the specified MAC address
  • You can also set up different virtual NICs for multiple Jails on the same NIC

Thank @Jip-Hop

2 Likes

Hello!

Jailmaker is looking really promising. I am working on isolation for increase security. I got user namespacing working, my next step is network.

I am currently using macvlan, it already prevents the jail to connect to the host which is good for me.

Is there a way to limite what the jail can access on my local network? I’ve tried with iptables but it seems with macvlan packet are dumped on the interface at a lower level than iptables.

Alternatively, perhaps a bridge interface would work instead? This is for a 10gbps NIC so I’m hoping to not hurt performance either.

Thanks

Are you trying to firewall the jail by adding iptables rules directly on the SCALE host? In case of bridge networking, the SCALE host is forwarding traffic generated by the processes in the container. But the traffic is generated in it’s own network namespace. The OUTPUT chain doesn’t catch packets coming out of a jail. The FORWARD chain should.

When using macvlan, the packets never pass through the host net namespace. You could add the iptables rules inside the jail… but then the jail root user can wipe them. Something else I didn’t experiment with but may work in this case is to use: IPAddressDeny and IPAddressAllow systemd options.

1 Like

I just created a jail using macvlan networking and started it with IPAddressDeny and IPAddressAllow added: systemd-run --property="IPAddressDeny=any" --property="IPAddressAllow=8.8.8.8" --collect --property=Delegate=yes --property=RestartForceExitStatus=133 .... Seems to work great to control which IP addresses the jail can access.

You’d have to add these flags to the systemd_run_default_args in the config file for now.

Edit: I forgot I already tried something similar before.

3 Likes

What I tried is something similar to what I was doing on the Kubernetes bridge interface (which I know if completely different).

This did not work:
iptables -i macvtap3 -I INPUT 1 -p icmp -d 10.100.0.50 -j ACCEPT
iptables -i macvtap3 -I INPUT 2 -p icmp -d 10.100.0.0/16 -j DROP

Basically I’m hoping to restrict what the jail can access on my network on the TrueScale Host.

I will test your solution as it seem very straight forward!!

1 Like

So the issue with this approach is that it seems to apply to both egress and ingress
systemd.resource-control (www.freedesktop.org)

I started a netcat to listen in the jail: nc -l -p 8888

and added the following args:
-p IPAddressAllow=10.100.0.1 -p IPAddressDeny=10.100.0.0/16

the allow permit traffic to my gateway (for internet), and block everyting from and to my network. so netcat become unreachable. This might be viable if i tunnel everything from a reverse proxy outside the jail.

Otherwise I might have to go to bridge with FORWARD iptables like you suggested above.