I recommend iXsystems to consider podman instead of docker for Electric Eel. It has the potential to make a lot of people (or nobody) happy. It is compatible with compose files, it can deploy helm charts, and it can run Sandboxes. It’s not fully supported by the popular Portainer project though and I’m sure there are other downsides. Too bad docker doesn’t have a --rootfs flag like podman and nerdctl.
1 Like
I suspect that this ship has long sailed.
Users who are worried about root-level docker can run docker, or podman, in a sandbox—and keep doing so with Electric Eel.
2 Likes
I’m after some advice for using multiple VLANs / bridges with Jailmaker / Dockge.
Firstly, I’d like to start by saying that, as a longtime user of FreeNAS & TrueNAS Core, I’d had a look at Scale a while ago, but decided not to make the change due to various issues with the Apps. Jailmaker from @Jip-Hop & the video’s from @Stux have completely changed my mind on that. Jailmaker is amazing & the videos have been great to follow. I’ve got a second bare metal test server that I’m using before I commit to changing over my main server.
My aim is to be able to run multiple docker instances within a jail, but to specify which VLAN each instance uses. I can accomplish that at the moment using Jailmaker by running two docker instances in parallel (docker & docker30), each one connected to a different bridge/VLAN, but it would be great to just have one docker instance running.
I’ve been reading the advanced networking Jailmaker docs & following Stux’s video guides. I’ve got the VLAN bridges working fine. One NIC is the “master”, which is bridged to br0. I use this as the main access the NAS. The second NIC has two VLANs (vlan30 & vlan40), which are each bridged (br30 & br40 respectively). IP addresses are provided by my OPNsense firewall DHCP. Using a VM image I have proved that the VLAN bridges work as expected.
My issue is within Jailmaker (2.0.0) & Dockge.
Testing this by adding in vlan30, the relevant bit of my jlmkr.py file is:
startup=0
gpu_passthrough_intel=0
gpu_passthrough_nvidia=0
# Turning off seccomp filtering improves performance at the expense of security
seccomp=1
# Use macvlan networking to provide an isolated network namespace,
# so docker can manage firewall rules
# Alternatively use --network-macvlan=eno1 instead of --network-bridge
# Ensure to change eno1/br1 to the interface name you want to use
# You may want to add additional options here, e.g. bind mounts
systemd_nspawn_user_args=--network-bridge=br0
--network-veth-extra=ve-docker-1:vee-1
--resolv-conf=bind-host
--system-call-filter='add_key keyctl bpf'
--bind='/mnt/trinity/docker/data:/mnt/data'
--bind='/mnt/trinity/docker/stacks:/opt/stacks'
# Script to run on the HOST before starting the jail
# Load kernel module and config kernel settings required for docker
pre_start_hook=#!/usr/bin/bash
set -euo pipefail
echo 'PRE_START_HOOK'
echo 1 > /proc/sys/net/ipv4/ip_forward
modprobe br_netfilter
echo 1 > /proc/sys/net/bridge/bridge-nf-call-iptables
echo 1 > /proc/sys/net/bridge/bridge-nf-call-ip6tables
# Specify command/script to run on the HOST after starting the jail
# For example to attach to multiple bridge interfaces
post_start_hook=#!/usr/bin/bash
set -euo pipefail
echo 'POST_START_HOOK'
ip link set dev ve-docker-1 master br30
ip link set dev ve-docker-1 up
If I run ip a within the docker shell it looks like vee-1 is indeed being created:
root@docker:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host noprefixroute
valid_lft forever preferred_lft forever
2: host0@if21: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 4e:8a:84:5f:18:a5 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 10.20.20.178/24 metric 1024 brd 10.20.20.255 scope global dynamic host0
valid_lft 5357sec preferred_lft 5357sec
inet6 fe80::4c8a:84ff:fe5f:18a5/64 scope link
valid_lft forever preferred_lft forever
3: vee-1@if22: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether e2:74:2e:84:21:4f brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 10.20.30.134/24 metric 1024 brd 10.20.30.255 scope global dynamic vee-1
valid_lft 5356sec preferred_lft 5356sec
inet6 fe80::e074:2eff:fe84:214f/64 scope link
valid_lft forever preferred_lft forever
4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 02:42:64:06:e7:39 brd ff:ff:ff:ff:ff:ff
inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
valid_lft forever preferred_lft forever
5: br-3fb6a4b3356e: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 02:42:62:1f:a6:05 brd ff:ff:ff:ff:ff:ff
inet 172.18.0.1/16 brd 172.18.255.255 scope global br-3fb6a4b3356e
valid_lft forever preferred_lft forever
7: vethbb2286e@if6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-3fb6a4b3356e state UP group default
link/ether ea:41:0a:ef:81:19 brd ff:ff:ff:ff:ff:ff link-netnsid 1
inet6 fe80::e841:aff:feef:8119/64 scope link
valid_lft forever preferred_lft forever
The problem comes when using dockge. I tried to generate a test nginx instance running on vlan30 using compose.yaml:
version: "3.8"
services:
nginx:
image: nginx:latest
restart: unless-stopped
networks:
- vee-1
ports:
- 8080:80
networks:
vee-1:
external: true
But this fails with the error: output: network vee-1 declared as external, but could not be found. I also tried adding name: vee-1 in the lower networks section.
If I don’t declare vee-1 as external then an additional network nginx-vee-1 is created, which again doesn’t map correctly.
Am I missing something?
1 Like
Or even use sysbox in docker 
I wonder if that is docker’s approach to competing with podman.
Good job on making it this far I think what you are missing is to actually create a network with docker (in the compose file or with the CLI manually). When you specify a network as external in docker compose then this is not a network external to docker, but a network made by docker (but external to the current compose file).
OK, thank you. So in the docker shell would that literally be
docker network create vee-1
Even though network vee-1 is already showing if I type
ip a ?
3: vee-1@if22: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether e2:74:2e:84:21:4f brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 10.20.30.134/24 metric 1024 brd 10.20.30.255 scope global dynamic vee-1
valid_lft 4671sec preferred_lft 4671sec
inet6 fe80::e074:2eff:fe84:214f/64 scope link
valid_lft forever preferred_lft forever
Alternatively, how would I do it within the compose file because if I don’t specify it as external it does create a new network, but prepends it with “nginx-”?
So I tried the docker network create vee-1 command.
The nginx instance starts fine now, however it is available on both bridged interfaces (br0 and br30), so is available at both http://10.20.20.178:8080/ and http://10.20.30.134:8080/, even though I was trying to only make it available on vee-1 (br30 / 10.20.30.134). Do you know if this is possible, or am I going to have to fall back to my original method of having two docker instances running in parallel?
Sorry I don’t know how to create a docker network on top of a specific interface. You may have better luck asking in docker specific forums and/or looking through the docker documentation as this is not ‘jailmaker territory’. What I do know is that you can specifically bind to one of those IP addresses e.g. 10.20.30.134:8080 in your compose file. Maybe that helps.
1 Like
Thanks for the reply @Jip-Hop. More learning required! I’m loving Jailmaker though & you’ve convinced me to change from CORE to SCALE 
Ah! This seems to have done the trick:
version: "3.8"
services:
nginx:
image: nginx:latest
restart: unless-stopped
networks:
- vee-1
ports:
- 10.20.30.134:8080:80
networks:
vee-1:
name: vee-1
external: true
By specifying the IP address it’s only available at the 10.20.30.134:8080 address now!
2 Likes
Thank you so much @Jip-Hop for creating Jailmaker, and @Stux for the awesome videos (and iX for TrueNAS and sandboxes of course )
Just finished migrating from CORE to SCALE and setting up a Syncthing jail as this was the last thing stopping me migrating.
So far so good and all working great using macvlan networking, even some private-users name-spacing…
Question for @Jip-Hop - would you accept a PR on your GitHub for additional template(s)? And do you know if it’s possible to pass in multiple macvlan networks (bound to different interfaces) with systemd_nspawn_user_args?
Also just throwing in my 2c support for Podman on Electric Eel
2 Likes
Multiple macvlan networks?
systemd_nspawn_user_args=–network-macvlan=br1
–network-macvlan=br10
–network-macvlan=br223
–resolv-conf=bind-host
1 Like
Happy to report that with me being an absolute docker noob i started playing around with a docker jail and portainer and got my first few container running with docker compose. Right now i’m learning how to setup homepage with automatic container detection. If i have everything running my last big learning curve will be traefik. I feel like i’m doing baby steps but it’s going better then i expected.
6 Likes
Awesome thanks @truenas-fan - didn’t realise it was so simple (reading the Networking FAQ on GitHub it seemed a lot more complex for bridges, so assumed it may be similar for macvlan - glad it’s not
1 Like
I have seen people reporting issues with this setup. I don’t remember if it was in this thread or on GitHub but I think macvlan should not be created on top of a bridge interface.
Thanks @Jip-Hop - I was planning on setting up multiple macvlan networks on the interfaces directly (enX1, enX2 etc.), not using bridges as I don’t need host ↔ container communication.
Also just regarding my previous question would you be open to receiving PRs for additional template examples on GitHub?
I think @Jip-Hop specifically added support for virtual Ethernet’s for just this case
1 Like
Yes that looks right and the veths are referenced in the Networking FAQ link on the GitHub I posted above.
The veth’s seems to only be needed for multiple bridge networks though - according to the Debian man page for systemd-nspawn,
Note that –network-macvlan= implies –private-network. This option may be used more than once to add multiple network interfaces to the container.
So this should work fine (multiple macvlan’s bound to ethernet interfaces not bridges), which I plan to test soon:
systemd_nspawn_user_args=–network-macvlan=enX0
–network-macvlan=enX1
–network-macvlan=enX2
–resolv-conf=bind-host
I’m open to PRs but I’d like to reserve the templates for the more complex setups (with lots of dependencies or specific config file requirements). Syncthing is a prime example of software which needs literally nothing to run. 
Creating a dedicated jail just for syncthing is overkill IMHO. I used to run it directly with systemd back when the SCALE rootfs wasn’t immutable. Now I’d recommend just starting syncthing with systemd-run from a Post Init script and using the .service file as inspiration for the systemd-run flags to use.
1 Like
First. Thanks for the hard work on this awesome solution.
I’m here to learn so I want to know the best practice.
Now, I never take I think/know/feel/assume for more than that.
Do you have any specific examples of configs, logs or anything at all, that can show everybody, the claims/issues, so hopefully “somebody” can “figure” out, for sure, with proof, arguments and/or code, why do or don’t do anything in this realm of magic?
Other than that, can we then please get better/more examples of all of this “somewhere”?
Thx.
That was me. I don’t recommend using macvlan or ipvlan from a bridge interface. You end up with odd network behaviour and instability. FWIW I tested it out using an Intel i210 NIC and an Intel x540 NIC and experienced the same thing with both.
1 Like