Cooking FreeBSD jails in a BSD Pot

General Discussion
41

Pot is very nice.

You need to set up a bridge (@pmh helped me with this):
/etc/rc.conf

    cloned_interfaces="bridge0"
    
    ifconfig_em0="-rxcsum -txcsum -lro -vlanmtu -vlanhwcsum -vlanhwfilter -vlanhwtag up"
    ifconfig_bridge0="addm em0 up"
    ifconfig_bridge0_alias0="inet 1.2.3.4/24"

/boot/loader.conf

    if_bridge_load="YES"

/etc/sysctl.conf

    net.link.bridge.inherit_mac=1

And remember pot is using PF to create and manage that network so you’ll need to configure PF as well.

The only few issues I was having with Pot was the automatic starting, after a reboot, and the templates–more specifically, the details like the jail template IP and whatnot–so I found myself creating small little scripts to create the jails that listed the IP and whatnot in them. Other than that Pot is a very nicely written tool! I eventually want to start playing with the container/image aspect of Pot but I just don’t have the need at the moment.

I eventually switched jail creating to my script which creates a jail.conf file which allows FreeBSD to do my jail management instead; it was very much simpler and cleaner. My script allows me to keep a “cleaner” directory of template jails with notes, and details I can refer to easier.

42

I did do the bridge. But pot has 4 network options.

Inherit - use same MAC and IP of host
Alias - uses different IP but the same MAC
Public-bridge - docker style with internal network
Private-bridge - same as public-bridge but with more isolation internally

I could not find a way to have a completely individual network stack.

How does your script deal with MAC addresses? Are they static even on reboot?

43

I use public-bridge.

Humm… Not sure about MAC addresses. I’ve never looked into it. But my script doesn’t do much besides create the jail.conf file and load in the configure script; -i.e. my script is just saving me from having to type all those steps for every jail (if/when you try my script, you’d see–very simple thing). …I’ll have to check on the MAC address thing.

44

How do you expose service when using the public-bridge? Doesn’t everything have to go through the host IP and therefore a single IP?

45

No. The jail has its own IP. Create the jail and give it 192.x.x.63 and public-bridge. From your laptop just ping that IP. You should get a response.

46

What does you pot config file look like? Did you specify that range in the EXTIF variable?

47

One second. I’ll go look.

48

My pot.conf EXTIF looks like this:

# The name of the network physical interface, to be used as default gateway
POT_EXTIF=bridge0
49

What about the internal network variables?

50

Here is my entire pot.conf file.

# pot configuration file

# All datasets related to pot use the some zfs dataset as parent
# With this variable, you can choose which dataset has to be used
POT_ZFS_ROOT=zroot/jails

# Location where the root dataset is mounted
POT_FS_ROOT=/usr/jails

# This is the cache used to import/export pots
POT_CACHE=/var/cache/pot

# This is where pot is going to store temporary files
POT_TMP=/tmp

# This is the group owning POT_FS_ROOT
POT_GROUP=pot

# This is the suffix added to temporary files created using mktemp,
# X is a placeholder for a random character, see mktemp(1)
POT_MKTEMP_SUFFIX=.XXXXXXXX

# Define the max length of the hostname inside the pot
POT_HOSTNAME_MAX_LENGTH=64

# Internal Virtual Network configuration

# IPv4 Internal Virtual network
#POT_NETWORK=10.192.0.0/10
POT_NETWORK=192.168.0.1/24

# Internal Virtual Network netmask
#POT_NETMASK=255.192.0.0
POT_NETMASK=255.255.255.0

# The default gateway of the Internal Virtual Network
#POT_GATEWAY=10.192.0.1
POT_GATEWAY=192.168.0.250

# The name of the network physical interface, to be used as default gateway
POT_EXTIF=bridge0
#POT_EXTIF=em0          # To create an `inherit` type of jail.

# IP of the DNS
# POT_DNS_IP=10.192.0.2
POT_DNS_IP=192.168.0.1
51

So you changed the dns and gateway to your actual LAN gateway?

I once had a horrible experience doing that with CBSD. It essentially borked my whole network by taking the place of my actual gateway…

This is mine

# Internal Virtual Network configuration

# IPv4 Internal Virtual network
# POT_NETWORK=10.192.0.0/10

# Internal Virtual Network netmask
# POT_NETMASK=255.192.0.0

# The default gateway of the Internal Virtual Network
# POT_GATEWAY=10.192.0.1

# The name of the network physical interface, to be used as default gateway
POT_EXTIF=bridge0

It uses the default values by default.

52

*blink-blink* Oh?! …well, now I’m going to be changing that.

53

I’m not sure if pot does it the same way though.
CBSD created an interface with that IP, so that’s what the issue was.

Pot probably doesn’t do it that way, I’m not sure.

54

Probably not. But it doesn’t hurt to be safe.

55

But if you don’t have that gateway there, then you won’t be able to access the internet, right?

56

That I do not know. I *ALWAYS* only configure the bare minimum so I should have checked that and made a decision to keep that setting in there, but I do not remember testing that.

My guess is that you may need it to get out, but I could be wrong; maybe PF knows enough from your base system to configure that when it makes the epair. However, even the built-in jail.conf needs a gateway so it is quite possible that value is needed. I’ll have to run a test after work.

57

I just confirmed that setting the gateway will actually create a bridge interface with your gateway IP. This will absolutely not work as devices on your LAN will not know where the actual gateway is.

BastilleBSD does this part right I think.

Pot seems more like a docker type networking. The only thing that will get jails functioning on your LAN with their own IP (or closest alternative) is the alias network. But again, no MAC address.

58

Oh!?

Try a jail using the jail.conf method.
Chapter 17. Jails and Containers | FreeBSD Documentation Portal

EDIT: that’s the setup portion (you need to download the userland)
After a few paragraphs you get the part about setting up a VNET jail.
Chapter 17. Jails and Containers | FreeBSD Documentation Portal

59

Attempting that now.

60

I do not think you will get custom MACs with what is written in the handbook. I just did some googling and I think you have to do a few extra steps. But, I’m still trying to understand if the OP got it working or not.