Linux Jails (sandboxes / containers) with Jailmaker

Converted my 1 app (Plex) from the TrueCharts app to a Docker container within Jailmaker managed by Dockge. Now comes the tougher part, deciding if it’s worth migrating Home Assistant from a HAOS VM to running in Docker. HAOS helps keep the WAF high, so I may not migrate that, as much as I would like to reclaim space and resources.

Highly suggest diving in. I’m confused where WAF comes into play? What would you miss in a docker container that comes with HAOS?

At least, this one is a real word… even though one might have to be Polish to pronounce it. (But wait until there’s great software coming from south-western Africa, named using local click phonetic :stuck_out_tongue: )

“Dockge”, on the other hand, with the idea for it coming from Twich :scream: is just screaming: “This is for Gen Z only”.

3 Likes

WAF comes in because my installation of Home Assistant is working without issue. By shifting, I’d need to spin up the containerized version while keeping the VM running (the easy part) and then migrate and test everything while not interrupting the day-to-day of our household (the hard part).

The main things I’d be losing are Home Assistant Supervisor and access to the Home Assistant App Store. In HAOS, all the apps are just Docker containers, so, I should be able to get them running no problem. I should also be able to duplicate Supervisor’s backup and update features outside of HAOS, so I’m less concerned about it. In the end, I just need to figure out how to accomplish all of this.

1 Like

Well, this elder millennial managed to get Dockge running without issue, though I pronounce it like Stux, Dockage.

Maybe I’m just getting old, but if the developer wished for it to be pronounced a specific way, they should have spelled it in a way that would be more indicative of pronunciation. That said, I know different languages can pronounce the same combination of letters differently (or in the case of English, the same language), but I don’t think “ck” falls into that realm.

1 Like

As per jea001’s post, the lack of being able to restore a backup is why I chose to put HA in a VM rather than as a TrueNAS app or as a container. Using app/containerised HA, I couldn’t find a way to restore a backup from my previous Pi-hosted instance, and really didn’t want to recreate everything from scratch.

1 Like

After updating three of my machines to 24.04.1, the Tailscale jail fails to start automatically, shutting off my remote access (I did confirm this on one of those, via SSH).

After the boot I can manually start the jail via shell, and Tailscale access is restored.

Anyone else experiencing this?

Would be super helpful to know what the failure mode is? Does tailscale give any logs or indication of what went wrong? Could be delays in networking generally being available at launch time.

I did get some logs, hope they help - they include the status after boot, and then after manually starting the jail (I’m not clear about the boundary between them):

May 28 10:57:48 saadadvbak systemd-nspawn[8171]:
May 28 10:57:48 saadadvbak systemd-nspawn[8171]: Debian GNU/Linux 12 Tailscale pts/0
May 28 10:57:48 saadadvbak systemd-nspawn[8171]:
May 28 11:00:58 saadadvbak systemd[1]: Stopping jlmkr-Tailscale.service - My nspawn jail Tailscale [created with jailmaker]…
May 28 11:00:58 saadadvbak systemd-nspawn[8171]: Tailscale login: Trying to halt container. Send SIGTERM again to trigger immediate termination.
May 28 11:00:58 saadadvbak systemd-nspawn[8171]: [ OK ] Removed slice system-getty.slice - Slice /system/getty.
May 28 11:00:58 saadadvbak systemd-nspawn[8171]: [ OK ] Removed slice system-modprobe.slice - Slice /system/modprobe.
May 28 11:00:58 saadadvbak systemd-nspawn[8171]: [ OK ] Stopped target graphical.target - Graphical Interface.
May 28 11:00:58 saadadvbak systemd-nspawn[8171]: [ OK ] Stopped target multi-user.target - Multi-User System.
May 28 11:00:58 saadadvbak systemd-nspawn[8171]: [ OK ] Stopped target getty.target - Login Prompts.
May 28 11:00:58 saadadvbak systemd-nspawn[8171]: [ OK ] Stopped target nss-lookup.target - Host and Network Name Lookups.
May 28 11:00:58 saadadvbak systemd-nspawn[8171]: [ OK ] Stopped target remote-cryptsetup.target - Remote Encrypted Volumes.
May 28 11:00:58 saadadvbak systemd-nspawn[8171]: [ OK ] Stopped target remote-veritysetup.t…- Remote Verity Protected Volumes.
May 28 11:00:58 saadadvbak systemd-nspawn[8171]: [ OK ] Stopped target timers.target - Timer Units.
May 28 11:00:58 saadadvbak systemd-nspawn[8171]: [ OK ] Stopped apt-daily-upgrade.timer - D… apt upgrade and clean activities.
May 28 11:00:58 saadadvbak systemd-nspawn[8171]: [ OK ] Stopped apt-daily.timer - Daily apt download activities.
May 28 11:00:58 saadadvbak systemd-nspawn[8171]: [ OK ] Stopped dpkg-db-backup.timer - Daily dpkg database backup timer.
May 28 11:00:58 saadadvbak systemd-nspawn[8171]: [ OK ] Stopped e2scrub_all.timer - Periodi…etadata Check for All Filesystems.
May 28 11:00:58 saadadvbak systemd-nspawn[8171]: [ OK ] Stopped systemd-tmpfiles-clean.time… Cleanup of Temporary Directories.
May 28 11:00:58 saadadvbak systemd-nspawn[8171]: Stopping console-getty.service - Console Getty…
May 28 11:00:58 saadadvbak systemd-nspawn[8171]: Stopping dbus.service - D-Bus System Message Bus…
May 28 11:00:58 saadadvbak systemd-nspawn[8171]: Stopping systemd-logind.service - User Login Management…
May 28 11:00:58 saadadvbak systemd-nspawn[8171]: Stopping tailscaled.service - Tailscale node agent…
May 28 11:00:58 saadadvbak systemd-nspawn[8171]: [ OK ] Stopped dbus.service - D-Bus System Message Bus.
May 28 11:00:58 saadadvbak systemd-nspawn[8171]: [ OK ] Stopped systemd-logind.service - User Login Management.
May 28 11:00:58 saadadvbak systemd-nspawn[8171]: [ OK ] Stopped console-getty.service - Console Getty.
May 28 11:00:58 saadadvbak systemd-nspawn[8171]: Stopping systemd-user-sessions.service - Permit User Sessions…
May 28 11:00:58 saadadvbak systemd-nspawn[8171]: [ OK ] Stopped systemd-user-sessions.service - Permit User Sessions.
May 28 11:00:58 saadadvbak systemd-nspawn[8171]: [ OK ] Stopped target network.target - Network.
May 28 11:00:58 saadadvbak systemd-nspawn[8171]: [ OK ] Stopped target remote-fs.target - Remote File Systems.
May 28 11:00:58 saadadvbak systemd-nspawn[8171]: Stopping systemd-networkd.service - Network Configuration…
May 28 11:00:58 saadadvbak systemd-nspawn[8171]: [ OK ] Stopped systemd-networkd.service - Network Configuration.
May 28 11:00:59 saadadvbak systemd-nspawn[8171]: [ OK ] Stopped tailscaled.service - Tailscale node agent.
May 28 11:00:59 saadadvbak systemd-nspawn[8171]: [ OK ] Stopped target basic.target - Basic System.
May 28 11:00:59 saadadvbak systemd-nspawn[8171]: [ OK ] Stopped target network-pre.target - Preparation for Network.
May 28 11:00:59 saadadvbak systemd-nspawn[8171]: [ OK ] Stopped target paths.target - Path Units.
May 28 11:00:59 saadadvbak systemd-nspawn[8171]: [ OK ] Stopped target slices.target - Slice Units.
May 28 11:00:59 saadadvbak systemd-nspawn[8171]: [ OK ] Removed slice user.slice - User and Session Slice.
May 28 11:00:59 saadadvbak systemd-nspawn[8171]: [ OK ] Stopped target sockets.target - Socket Units.
May 28 11:00:59 saadadvbak systemd-nspawn[8171]: [ OK ] Closed dbus.socket - D-Bus System Message Bus Socket.
May 28 11:00:59 saadadvbak systemd-nspawn[8171]: [ OK ] Stopped target sysinit.target - System Initialization.
May 28 11:00:59 saadadvbak systemd-nspawn[8171]: [ OK ] Stopped target cryptsetup.target - Local Encrypted Volumes.
May 28 11:00:59 saadadvbak systemd-nspawn[8171]: [ OK ] Stopped systemd-ask-password-consol…quests to Console Directory Watch.
May 28 11:00:59 saadadvbak systemd-nspawn[8171]: [ OK ] Stopped systemd-ask-password-wall.p… Requests to Wall Directory Watch.
May 28 11:00:59 saadadvbak systemd-nspawn[8171]: [ OK ] Stopped target integritysetup.targe…Local Integrity Protected Volumes.
May 28 11:00:59 saadadvbak systemd-nspawn[8171]: [ OK ] Stopped target veritysetup.target - Local Verity Protected Volumes.
May 28 11:00:59 saadadvbak systemd-nspawn[8171]: [ OK ] Closed systemd-networkd.socket - Network Service Netlink Socket.
May 28 11:00:59 saadadvbak systemd-nspawn[8171]: [ OK ] Stopped systemd-network-generator.s…rk units from Kernel command line.
May 28 11:00:59 saadadvbak systemd-nspawn[8171]: Stopping systemd-resolved.service - Network Name Resolution…
May 28 11:00:59 saadadvbak systemd-nspawn[8171]: Stopping systemd-update-utmp.servic…rd System Boot/Shutdown in UTMP…
May 28 11:00:59 saadadvbak systemd-nspawn[8171]: [ OK ] Stopped systemd-resolved.service - Network Name Resolution.
May 28 11:00:59 saadadvbak systemd-nspawn[8171]: [ OK ] Stopped systemd-update-utmp.service…cord System Boot/Shutdown in UTMP.
May 28 11:00:59 saadadvbak systemd-nspawn[8171]: [ OK ] Stopped systemd-tmpfiles-setup.serv…te Volatile Files and Directories.
May 28 11:00:59 saadadvbak systemd-nspawn[8171]: [ OK ] Stopped target local-fs.target - Local File Systems.
May 28 11:00:59 saadadvbak systemd-nspawn[8171]: Unmounting run-credentials-systemd\…ntials/systemd-sysusers.service…
May 28 11:00:59 saadadvbak systemd-nspawn[8171]: Unmounting run-credentials-systemd\…/systemd-tmpfiles-setup.service…
May 28 11:00:59 saadadvbak systemd-nspawn[8171]: Unmounting run-credentials-systemd\…temd-tmpfiles-setup-dev.service…
May 28 11:00:59 saadadvbak systemd-nspawn[8171]: Unmounting tmp.mount - /tmp…
May 28 11:01:00 saadadvbak systemd-nspawn[8171]: [ OK ] Unmounted run-credentials-systemd\x…dentials/systemd-sysusers.service.
May 28 11:01:00 saadadvbak systemd-nspawn[8171]: [ OK ] Unmounted run-credentials-systemd\x…ystemd-tmpfiles-setup-dev.service.
May 28 11:01:00 saadadvbak systemd-nspawn[8171]: [ OK ] Unmounted tmp.mount - /tmp.
May 28 11:01:00 saadadvbak systemd-nspawn[8171]: [ OK ] Stopped target swap.target - Swaps.
May 28 11:01:00 saadadvbak systemd-nspawn[8171]: [ OK ] Unmounted run-credentials-systemd\x…ls/systemd-tmpfiles-setup.service.
May 28 11:01:00 saadadvbak systemd-nspawn[8171]: [ OK ] Stopped target local-fs-pre.target …reparation for Local File Systems.
May 28 11:01:00 saadadvbak systemd-nspawn[8171]: [ OK ] Reached target umount.target - Unmount All Filesystems.
May 28 11:01:00 saadadvbak systemd-nspawn[8171]: [ OK ] Stopped systemd-tmpfiles-setup-dev.…reate Static Device Nodes in /dev.
May 28 11:01:00 saadadvbak systemd-nspawn[8171]: [ OK ] Stopped systemd-sysusers.service - Create System Users.
May 28 11:01:00 saadadvbak systemd-nspawn[8171]: [ OK ] Stopped systemd-remount-fs.service …ount Root and Kernel File Systems.
May 28 11:01:00 saadadvbak systemd-nspawn[8171]: [ OK ] Reached target shutdown.target - System Shutdown.
May 28 11:01:00 saadadvbak systemd-nspawn[8171]: [ OK ] Reached target final.target - Late Shutdown Services.
May 28 11:01:00 saadadvbak systemd-nspawn[8171]: Starting systemd-halt.service - System Halt…
May 28 11:01:00 saadadvbak systemd-nspawn[8171]: Sending SIGTERM to remaining processes…
May 28 11:01:00 saadadvbak systemd-nspawn[8171]: Sending SIGKILL to remaining processes…
May 28 11:01:00 saadadvbak systemd-nspawn[8171]: All filesystems, swaps, loop devices, MD devices and DM devices detached.
May 28 11:01:00 saadadvbak systemd-nspawn[8171]: Halting system.
May 28 11:01:00 saadadvbak systemd[1]: jlmkr-Tailscale.service: Deactivated successfully.
May 28 11:01:00 saadadvbak systemd[1]: Stopped jlmkr-Tailscale.service - My nspawn jail Tailscale [created with jailmaker].
May 28 11:01:00 saadadvbak systemd[1]: jlmkr-Tailscale.service: Consumed 2.459s CPU time.
– Boot db13630e983844709d0a85d2f2b15341 –
May 28 11:07:41 saadadvbak systemd[1]: Starting jlmkr-Tailscale.service - My nspawn jail Tailscale [created with jailmaker]…
May 28 11:07:41 saadadvbak systemd[1]: Started jlmkr-Tailscale.service - My nspawn jail Tailscale [created with jailmaker].
May 28 11:07:42 saadadvbak systemd-nspawn[13346]: systemd 252.22-1~deb12u1 running in system mode (+PAM +AUDIT +SELINUX +APPARMOR +IMA +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +>
May 28 11:07:42 saadadvbak systemd-nspawn[13346]: Detected virtualization systemd-nspawn.
May 28 11:07:42 saadadvbak systemd-nspawn[13346]: Detected architecture x86-64.
May 28 11:07:42 saadadvbak systemd-nspawn[13346]:
May 28 11:07:42 saadadvbak systemd-nspawn[13346]: Welcome to Debian GNU/Linux 12 (bookworm)!
May 28 11:07:42 saadadvbak systemd-nspawn[13346]:
May 28 11:07:43 saadadvbak systemd-nspawn[13346]: Queued start job for default target graphical.target.
May 28 11:07:43 saadadvbak systemd-nspawn[13346]: [ OK ] Created slice system-getty.slice - Slice /system/getty.
May 28 11:07:43 saadadvbak systemd-nspawn[13346]: [ OK ] Created slice system-modprobe.slice - Slice /system/modprobe.
May 28 11:07:43 saadadvbak systemd-nspawn[13346]: [ OK ] Created slice user.slice - User and Session Slice.
May 28 11:07:43 saadadvbak systemd-nspawn[13346]: [ OK ] Started systemd-ask-password-consol…quests to Console Directory Watch.
May 28 11:07:43 saadadvbak systemd-nspawn[13346]: [ OK ] Started systemd-ask-password-wall.p… Requests to Wall Directory Watch.
May 28 11:07:43 saadadvbak systemd-nspawn[13346]: [ OK ] Reached target cryptsetup.target - Local Encrypted Volumes.
May 28 11:07:43 saadadvbak systemd-nspawn[13346]: [ OK ] Reached target integritysetup.targe…Local Integrity Protected Volumes.
May 28 11:07:43 saadadvbak systemd-nspawn[13346]: [ OK ] Reached target paths.target - Path Units.
May 28 11:07:43 saadadvbak systemd-nspawn[13346]: [ OK ] Reached target remote-cryptsetup.target - Remote Encrypted Volumes.
May 28 11:07:43 saadadvbak systemd-nspawn[13346]: [ OK ] Reached target remote-fs.target - Remote File Systems.
May 28 11:07:43 saadadvbak systemd-nspawn[13346]: [ OK ] Reached target remote-veritysetup.t…- Remote Verity Protected Volumes.
May 28 11:07:43 saadadvbak systemd-nspawn[13346]: [ OK ] Reached target slices.target - Slice Units.
May 28 11:07:43 saadadvbak systemd-nspawn[13346]: [ OK ] Reached target swap.target - Swaps.
May 28 11:07:43 saadadvbak systemd-nspawn[13346]: [ OK ] Reached target veritysetup.target - Local Verity Protected Volumes.
May 28 11:07:43 saadadvbak systemd-nspawn[13346]: [ OK ] Listening on systemd-initctl.socket… initctl Compatibility Named Pipe.
May 28 11:07:43 saadadvbak systemd-nspawn[13346]: [ OK ] Listening on systemd-journald-dev-l…ocket - Journal Socket (/dev/log).
May 28 11:07:43 saadadvbak systemd-nspawn[13346]: [ OK ] Listening on systemd-journald.socket - Journal Socket.
May 28 11:07:43 saadadvbak systemd-nspawn[13346]: [ OK ] Listening on systemd-networkd.socket - Network Service Netlink Socket.
May 28 11:07:43 saadadvbak systemd-nspawn[13346]: Mounting dev-hugepages.mount - Huge Pages File System…
May 28 11:07:43 saadadvbak systemd-nspawn[13346]: Mounting sys-fs-fuse-connections.mount - FUSE Control File System…
May 28 11:07:43 saadadvbak systemd-nspawn[13346]: Starting systemd-journald.service - Journal Service…
May 28 11:07:43 saadadvbak systemd-nspawn[13346]: Starting systemd-network-generator.… units from Kernel command line…
May 28 11:07:43 saadadvbak systemd-nspawn[13346]: Starting systemd-remount-fs.service…nt Root and Kernel File Systems…
May 28 11:07:43 saadadvbak systemd-nspawn[13346]: [ OK ] Finished systemd-network-generator.…rk units from Kernel command line.
May 28 11:07:43 saadadvbak systemd-nspawn[13346]: [ OK ] Reached target network-pre.target - Preparation for Network.
May 28 11:07:43 saadadvbak systemd-nspawn[13346]: [ OK ] Finished systemd-remount-fs.service…ount Root and Kernel File Systems.
May 28 11:07:43 saadadvbak systemd-nspawn[13346]: Starting systemd-sysusers.service - Create System Users…
May 28 11:07:43 saadadvbak systemd-nspawn[13346]: [ OK ] Started systemd-journald.service - Journal Service.
May 28 11:07:43 saadadvbak systemd-nspawn[13346]: Starting systemd-journal-flush.serv…h Journal to Persistent Storage…
May 28 11:07:43 saadadvbak systemd-nspawn[13346]: [ OK ] Mounted dev-hugepages.mount - Huge Pages File System.
May 28 11:07:43 saadadvbak systemd-nspawn[13346]: [ OK ] Mounted sys-fs-fuse-connections.mount - FUSE Control File System.
May 28 11:07:43 saadadvbak systemd-nspawn[13346]: [ OK ] Finished systemd-sysusers.service - Create System Users.
May 28 11:07:43 saadadvbak systemd-nspawn[13346]: Starting systemd-tmpfiles-setup-dev…ate Static Device Nodes in /dev…
May 28 11:07:43 saadadvbak systemd-nspawn[13346]: [ OK ] Finished systemd-tmpfiles-setup-dev…reate Static Device Nodes in /dev.
May 28 11:07:43 saadadvbak systemd-nspawn[13346]: [ OK ] Reached target local-fs-pre.target …reparation for Local File Systems.
May 28 11:07:43 saadadvbak systemd-nspawn[13346]: [ OK ] Reached target local-fs.target - Local File Systems.
May 28 11:07:43 saadadvbak systemd-nspawn[13346]: Starting systemd-networkd.service - Network Configuration…
May 28 11:07:44 saadadvbak systemd-nspawn[13346]: [ OK ] Finished systemd-journal-flush.serv…ush Journal to Persistent Storage.
May 28 11:07:44 saadadvbak systemd-nspawn[13346]: Starting systemd-tmpfiles-setup.ser… Volatile Files and Directories…
May 28 11:07:44 saadadvbak systemd-nspawn[13346]: [ OK ] Started systemd-networkd.service - Network Configuration.

For now I have managed to work around the issue with a Post Init command basically telling the system to wait two minutes and then to start the jail manually:

sleep 120s && /mnt/Tank/jailmaker/jlmkr.py start Tailscale

I can confirm that this kludge works - the problem could indeed be a race condition between the general startup of networking, and the jail.

P.S.: of course, this should be modified to account for your particular dataset and jail names. Be sure to add the command before upgrading, if you depend on Tailscale for remote TrueNAS management, or you’ll be locked out. I was…

Also, as a precaution I put a long timeout value (300), to avoid having the system automatically stop the command during the initial 2-minute delay.

I just upgraded to Dragonfish-24.04.1 and my jail, and the docker services inside it, came up without issues as usual post upgrade. But my jail uses bridge networking, which already causes start of the jail to delay. Perhaps that plays a role here xD I don’t see anything that stands out in your log file.

1 Like

It could very well be the case - my Tailscale jails all use host networking.

Unfortunately, my customers who use bridge networking inside their jails have this nasty habit of working during business hours, so I can’t update and reboot their servers before tonight. Will keep you posted. Thanks!

2 Likes

Sorry @Stux but v2.0.0 made the install part of the video outdated already :frowning:

So do we need to update our current jailmaker and if so how?

Check out the release notes of v2.0.0.

I understand but the release notes are ambiguous as to how to upgrade from a current jailmaker install to a 2.0.0 release. Maybe @stux can splain it in a video. In the meantime what I have works like a champ.

I’ve updated the description.

1 Like

With jailmaker 1.5.0, I had no automatic startup of a jail with bridge networking, after upgrading to 24.04.1, same issue as with the Tailscale jail with host networking; manually starting them after boot worked.

With jailmaker 2.0.0, all jails started automatically after upgrading to 24.04.1.

@Jip-Hop, did you change something in 2.0.0 that could account for this?

If you cloned the jailmaker repo, as per my video, then just cd into your jailmaker dataset, then run git pull

That should update your repo with the latest code

(Probably best to stop any jails first)

If you have untracked changes, or have modified things you shouldn’t have, then this SO article should cover it.

Assuming that all works it should just be a matter of restarting your jails, and if you get any errors, then editing the config so that it resembles the appropriate template in the template directory should resolve, as the templates are updated as the config options are deprecated/changed.

Thanks Stux that makes perfect sense.
I have to say Stux thanks for what you do.

1 Like