UPDATE: Seems as though adding em0 to bridge0 worked. At least I can ping the jail from within my network. I really didn’t have much time to play but I got a simple jail with the above script (removed the bridge creation part) to work.
Thank you all for the helpful hints!
I’d recommend not building/configuring the bridge from your jail script. Configure it statically like so:
/etc/rc.conf
cloned_interfaces="bridge0"
ifconfig_em0="-rxcsum -txcsum -lro -vlanmtu -vlanhwcsum -vlanhwfilter -vlanhwtag up"
ifconfig_bridge0="addm em0 up"
ifconfig_bridge0_alias0="inet 1.2.3.4/24"
/boot/loader.conf
if_bridge_load="YES"
/etc/sysctl.conf
net.link.bridge.inherit_mac=1
If you boot your host with that configuration in place it will work just like it did but you can now attach/detach jails at will.
Also, why don’t you use a jail manager like iocage or bastille?
Wow! I mean, I had most of the rc.conf stuff from last night but I didn’t have the either two configs. Thank you! You probably saved me hours of trying/reading/etc when I get to the next part.
I most certainly will be using a jail manager. I wrote that script so I can demonstrate my method to you guys (that’s how I was testing VNET the last few days but in script form). You had said the “attach the IP to the bridge not em0” comment and it was the total opposite of what I was doing so I figured if we got into any discussions/demonstrations/etc I had better provide the vehicle (to make any responses or demonstrations from anyone easier -i.e. save “your time” in helping me). I’m not sure that actually was necessary or even polite but that was my goal.
So far, I’ve tried CBSD, and POT. I’ve built “template scripts” for both of those managers and even gotten one going for Bastille/Rocinette.
Iocage is next. I know iX has dropped the templates (aka “plugins”) but you said you use iocage for your large amount of servers; did you guys improve upon the template (plugin) system for iocage or is the current infrastructure good enough? It looks like iX may have written the plugin infrastructure for iocage (I looked at the plugins briefly the other day).
We don’t use the iocage template system. We generate read-only images of complete software installations that we mount with local nullfs mounts into the jail structure.
I would start with iocage base jails or bastille right away. iocage is de facto abandonware, but it does work. No major issue apart from an arcane syntax and almost no error handling.
So if you say iocage foo you don’t get “unknown command foo” or some such but a nice python exception instead 
If you are interested in our architecture:
Iocage actually has a new maintainer. See freshports…
A new port maintainer, it seems, yet the upstream project is archived, so probably maintenance mode only:
BastilleBSD is my current preferred manager and it just manages my VNET jails painlessly. I also set them all up to just be configured by DHCP from the router and then I refer to them by their DNS names. This way, I never have to worry about remembering IP’s.
From my understanding, a new port maintainer is not the same as a new project maintainer. It just means that someone is maintaining the port and ensuring that it builds and installs properly whenever there are either upstream or FreeBSD version updates.
Personally, another reason why I’m not a fan of iocage is because it requires ZFS. There are circumstances where you may prefer to run UFS in your environment.
For me, the updates will be the biggest issue so if I can get a good template system built then I can destroy and create easier(ly) (another method in the tool box).
I like the rc.d script stuff in Pot so I’ll probably try to replicate that in whatever manager I use.