QNAP TS-877 Truenas Journal

mooglestiltzkin · April 28, 2024, 8:41pm

root@docker:~# useradd testing
root@docker:~# deluser testing
deluser: `/usr/bin/crontab' not executed. Skipping crontab removal. Package `cron' required.
Removing user `testing' ...
Done.
root@docker:~# /etc/init.d/cron stop
bash: /etc/init.d/cron: No such file or directory
root@docker:~# /usr/bin/cron stop
bash: /usr/bin/cron: No such file or directory
root@docker:~# /usr/bin/crontab stop
bash: /usr/bin/crontab: No such file or directory
root@docker:~# /usr/bin/crontab stop

yikes now i got a weird bug about cron that never popped out before when i deleted a user. not sure how to remove it x-x; worse case scenario, delete docker jail redo xd…

x-x; trying find out xd

Stux · April 28, 2024, 10:36pm

You can setup the cron inside the jail.

Remember, @winnielinnie told you how to do it in a FreeBSD jail. Concept is similar in a Linux jail. Command is different.

mooglestiltzkin · April 29, 2024, 1:20am

did more testing

tried delete docker jail then jailmaker dataset, redo from scratch.

tested still got the bug

root@docker:~# useradd testing
root@docker:~# deluser testing
deluser: `/usr/bin/crontab' not executed. Skipping crontab removal. Package `cron' required.
Removing user `testing' ...
Done.
root@docker:~#

before when i deluser it wouldnt mention the cron stuff.

so i browsed /usr/bin and found the crontab but not sure what to do with that x-x; for now just fixing up docker since that’s all i know

mooglestiltzkin · April 29, 2024, 2:21am

after trying to remake docker jail, the 192.168.0.24 no longer worked.

i already edited

github.com

Jip-Hop/jailmaker/blob/main/NETWORKING.md

# Jailmaker

## Advanced Networking

These are notes on advanced networking setup you may want to try. Contributions are welcome!

### Bridge Networking

As an alternative to the default host networking mode, you may want to connect to a bridge interface instead and let the jail obtain its IP address via DHCP (although you may have to be patient for up to 20 seconds after the jail started for networking to work, [assigning the IP address is somehow slow](https://github.com/Jip-Hop/jailmaker/issues/7)).

[This YouTube video](https://www.youtube.com/watch?v=7clQw132w58) may be helpful when setting up the bridge interface. Note: You may lock yourself out... It may take several tries... TrueNAS is a bit picky when switching IP addresses and toggling DHCP. May be helpful to connect a monitor and keyboard to the NAS and use `/etc/netcli` to reset the networking interface. Kept bothering with "Register Default Gateway" warning... I just clicked Cancel.

Add the `--network-bridge=br1 --resolv-conf=bind-host` systemd-nspawn flag when asked for `Additional flags` during jail creation, or set it post-creation by [editing](./README.md#edit-jail-config) the `SYSTEMD_NSPAWN_USER_ARGS` variable inside the `config` file.

The TrueNAS host and the jail will be able to communicate with each other as if the jail was just another device on the LAN. It will use the same DNS servers as the TrueNAS host because the `--resolv-conf=bind-host` option bind mounts the `/etc/resolv.conf` file from the host inside the jail. If you want to use the DNS servers advertised via DHCP, then check [DNS via DHCP](#dns-via-dhcp).

To configure a **static IP** with our bridge interface, we need to edit the `80-container-host0.network` file located in `/etc/systemd/network`. Change the `[Network]` section to look like this:

```ini
[Network]

This file has been truncated. show original

[Network]
DHCP=false
Address=192.168.0.12/24
Gateway=192.168.0.1
LinkLocalAddressing=no
LLDP=yes
EmitLLDP=customer-bridge

and restarted

systemctl restart systemd-networkd
systemctl status systemd-networkd
ifconfig

also restart truenas just in case. but still didn’t work. when i did a
docker ps -a

it said the ip for dockge was 0.0.0.0

i also checked the docker jail config, it does have the correct settings for bridge

systemd_nspawn_user_args=--network-bridge=br1
	--resolv-conf=bind-host

in truenas network i didn’t touch there since i had already setup that bridge way before.

mooglestiltzkin · April 29, 2024, 2:52am

i was also double checking with stux video on the networking portion.

mine doesn’t look right

root@docker:~# nano /etc/systemd/network/80-container-host0.network
root@docker:~# reboot
root@docker:~# 
Connection to machine docker terminated.
root@xxxxx[~]# jlmkr shell docker
Connected to machine docker. Press ^] three times within 1s to exit session.
root@docker:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: host0@if15: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether xxxxx link-netnsid 0
    inet 192.168.0.24/24 brd 192.168.0.255 scope global host0
       valid_lft forever preferred_lft forever
3: br-4fxxxxxb8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 0xxxxx
    inet 172.18.0.1/16 brd 172.18.255.255 scope global br-4fxxxxxb8
       valid_lft forever preferred_lft forever
    inet6 fe80::xxxxx/64 scope link 
       valid_lft forever preferred_lft forever
4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether xxxxx
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
6: vethf2c204a@if5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-xxxxxb8 state UP group default 
    link/ether xxxxx link-netnsid 1
    inet6 xxxxx/64 scope link 
       valid_lft forever preferred_lft forever
root@docker:~#

the 192.168.0.24 is there, but when i docker ps -a the dockge container is only showing 0.0.0.0

last time in a working setup it showed 192.168.0.24

not sure what happened there

root@xxxx[~]# ifconfig
br1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.0.210  netmask 255.255.255.0  broadcast 192.168.0.255
        ether xxxx  xxxx 1000  (Ethernet)
        RX packets 4641  bytes 467638 (456.6 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1643  bytes 48530899 (46.2 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

enp19s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 xxxx  prefixlen 64  scopeid 0x20<link>
        ether xxxx  txqueuelen 1000  (Ethernet)
        RX packets 4841  bytes 624438 (609.8 KiB)
        RX errors 0  dropped 5  overruns 0  frame 0
        TX packets 33866  bytes 50271212 (47.9 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 1381  bytes 344653 (336.5 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1381  bytes 344653 (336.5 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vb-docker: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 xxxx  prefixlen 64  scopeid 0x20<link>
        ether xxxx  txqueuelen 1000  (Ethernet)
        RX packets 39  bytes 2103 (2.0 KiB)
        RX errors 0  dropped 6  overruns 0  frame 0
        TX packets 267  bytes 94226 (92.0 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

root@xxxx[~]#

isn’t vb docker suppose to be 192.168.0.24 ?

*update

root@xxxxx[~]# jlmkr list
NAME   RUNNING STARTUP GPU_INTEL GPU_NVIDIA OS     VERSION ADDRESSES    
docker True    True    False     True       debian 12      192.168.0.24…
root@xxxxx[~]#

ok i see the ip for docker when checking.

but the docker compose isn’t using the ip for some reason?

mooglestiltzkin · April 29, 2024, 3:14am

docker inspect dockge

            "NetworkMode": "dockge_default",
            "PortBindings": {
                "5001/tcp": [
                    {
                        "HostIp": "",
                        "HostPort": "5001"

    },
    "NetworkSettings": {
        "Bridge": "",
        "SandboxID": "xxxxxxxxxxxxxxxx",
        "SandboxKey": "/var/run/docker/netns/xxxxxxxxxxxxxxx",
        "Ports": {
            "5001/tcp": [
                {
                    "HostIp": "0.0.0.0",
                    "HostPort": "5001"
                },
                {
                    "HostIp": "::",
                    "HostPort": "5001"
                }
            ]
        },
        "HairpinMode": false,
        "LinkLocalIPv6Address": "",
        "LinkLocalIPv6PrefixLen": 0,
        "SecondaryIPAddresses": null,
        "SecondaryIPv6Addresses": null,
        "EndpointID": "",
        "Gateway": "",
        "GlobalIPv6Address": "",
        "GlobalIPv6PrefixLen": 0,
        "IPAddress": "",
        "IPPrefixLen": 0,
        "IPv6Gateway": "",
        "MacAddress": "",
        "Networks": {
            "dockge_default": {
                "IPAMConfig": null,
                "Links": null,
                "Aliases": [
                    "dockge",
                    "dockge"
                ],
                "MacAddress": "xxxxxxxxxx",
                "NetworkID": "xxxxxxxxxxxxxxx",
                "EndpointID": "xxxxxxxxxxxxxxxxxxxxxxxxx",
                "Gateway": "172.18.0.1",
                "IPAddress": "172.18.0.2",
                "IPPrefixLen": 16,
                "IPv6Gateway": "",
                "GlobalIPv6Address": "",
                "GlobalIPv6PrefixLen": 0,
                "DriverOpts": null,
                "DNSNames": [
                    "dockge",
                    "xxxxxxxxxxxxxxxx"
                ]
            }

Stux · April 29, 2024, 3:26am

did you try installing cron?

apt install cron

mooglestiltzkin · April 29, 2024, 3:28am

well i got a bigger issue atm. can’t get docker working again. networking issue x-x; figuring that one out first before coming back to cron.

if push comes to shove, i just may have to backup then recover from a new reinstall of truenas. not too keen on that

jlmkr list shows 192.168.0.24…

but when i login to jlmkr shell docker

docker ps -a

it shows for dockge 0.0.0.0:5001

going to http://192.168.0.24:5001 no dockge >_<:

Stux · April 29, 2024, 3:45am

0.0.0.0 is the default used when you don’t specify an actual IP to bind the port forward to, and basically means, forward the 5001 port on all the interfaces to 5001 in the docker.

For example, this is a snippet from my gitlab installation

    ports:
      # bind to specific IP or all IPs... which will most likely fail since SSH already exists.
      - '${GITLAB_IP:-0.0.0.0}:80:80'
      - '${GITLAB_IP:-0.0.0.0}:443:443'
      - '${GITLAB_IP:-0.0.0.0}:22:22'

If I specify GITLAB_IP in the .env file, then gitlab will only bind on that interface. This allows me to have gitlab listen on 22/80/443, even though the host may be using those ports.

This gitlab is not hosted with Jailmaker yet… otherwise I probably wouldn’t do it this way yet

Stux · April 29, 2024, 3:48am

Test you jail’s networking first…

From your jail’s shell:

ping 8.8.8.8, that should pings google dns… which will test if you jail has external internet access via your gateway
ping www.google.com, the same, except now you’re testing DNS.
if you want your jail to be able to communicate with your host, for example, to setup WireGuard, or reverse proxy to TrueNAS, then ping your truenas IP

Before looking at dockge… verify everything above is working. And remember, it can take 30s+ for a jail to acquire a DHCP lease (for some reason) after it starts up.

mooglestiltzkin · April 29, 2024, 3:54am

yeah there is connectivity to google

also managed to apt update and stuff

just no idea why the dockge url doesnt work x-x;

mooglestiltzkin · April 29, 2024, 4:10am

root@docker:/mnt/docker/
2024-04-29xxxxxxx [SERVER] INFO: Welcome to dockge!
2024-04-29xxxxxxx [SERVER] INFO: NODE_ENV: production
2024-04-29xxxxxxx [SERVER] INFO: Server Type: HTTP
2024-04-29xxxxxxx [SERVER] INFO: Data Dir: ./data/
2024-04-29xxxxxxx [DB] INFO: Database Type: sqlite
2024-04-29xxxxxxx [SERVER] INFO: Connected to the database
2024-04-29xxxxxxx [SERVER] INFO: Listening on 5001
2024-04-29xxxxxxx [UPDATE-CHECKER] INFO: Failed to check for new versions
root@docker:/mnt/docker/

Stux · April 29, 2024, 4:23am

Can you paste a copy of your dockge compose.yaml?

mooglestiltzkin · April 29, 2024, 4:26am

really dont think thats the issue because i also tried a differen simple compose for librespeed. same issue.

services:
  dockge:
    image: louislam/dockge:latest
    container_name: dockge
    restart: unless-stopped
    ports:
      # Host Port:Container Port
      - 5001:5001
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /mnt/docker/data/dockge/data:/app/data
        
      # If you want to use private registries, you need to share the auth file with Dockge:
      # - /root/.docker/:/root/.docker

      # Stacks Directory
      # ⚠️ READ IT CAREFULLY. If you did it wrong, your data could end up writing into a WRONG PATH.
      # ⚠️ 1. FULL path only. No relative path (MUST)
      # ⚠️ 2. Left Stacks Path === Right Stacks Path (MUST)
      - /mnt/docker/stacks:/mnt/docker/stacks
    environment:
      # Tell Dockge where is your stacks directory
      - DOCKGE_STACKS_DIR=/mnt/docker/stacks
#networks: {}

i tested with networks: {} and without, no difference. in the working setup i deployed without this.

mooglestiltzkin · April 29, 2024, 4:29am

i was googling how to reset truenas networking, then try again from scratch. but ontop of the cron bug, i think i may as well

backup (i only need to backup the container stuff, it’s really not much after i pruned the junk really)
factory reset
setup from a new state
create data sets
recover data from backup to here via rsync

so rather than spend a week possibly longer troubleshooting a needle in a haystack, i’m at the point i rather launch the nuclear option and get this solved in 2-3 days or less.

i understand the processes now.

in fact i did test destroying docker jail before, and redeploying it without issue. it was only this time around that it didn’t go so well, no idea why.

things went downhill when i tried my best to get cron to work x-x;

that’s the current status report.

on my 2nd rodeo, as long as i stick to the plan, don’t deviate, no more testing, i think it should just work at that point.

i thought i did when i was redoing jailmaker but apparently something else was messed up that got overlooked. just couldn’t find out where exactly. i tried nuking docker jail first, then jailmaker dataset. now the only thing left is truenas

if i had to guess why this happened

i entered a cmdline i shouldn’t have and it affected truenas in some way, and this caused issues for jailmaker. possibly cron or when trying to install jailmaker because i had 1 mishap with it. i doubt any of the commands i entered within the docker jail shell was at fault because it wouldn’t affect truenas wouldn’t it? cept that one incident someone used the capabitilies all in the docker config jail that messed up his networking. but it’s been removed from the docker config and warned against using using such settings so should be alright.
networking issue. i went back and forth between truenas networking and jailmaker but couldn’t spot anything wrong with my config, it simply just didn’t work when trying the url.

mooglestiltzkin · April 29, 2024, 4:50am

i think one of my mishaps, i was trying to install jailmaker but forgot to create the jailmaker dataset first x-x; don’t do that.

Stux · April 29, 2024, 5:17am

I’m sure nothings wrong that can’t be fixed.

Nothing is wrong with your TrueNAS installation right?

mooglestiltzkin · April 29, 2024, 5:28am

well like i said i may have entered commands in cmdline that MAy have done something to truenas.

cauz it wouldnt make sense otherwise since i had nuked docker jail, and even nuked jailmaker data set, then recreated those from scratch yet the issue persists.

2 issues. 1, cronjob shows up in commandline when i am doing something else (e.g. when in jailmaker shell docker, i create a user testing, then delete user testing, out pops out a msg about crontab fail to run so it wont proceed… why did that pop out for no reason when i am doing something else? this never happened before when doing the same thing fyi). 2, docker container no longer works on the static ip (i triple checked i did all the steps but either i missed something or, something got messed up, not sure which)

so i dont think its truenas or jailmakers fault, most likely its the user fault (me). I just can’t recall what exactly i did that caused the failure x-x;

all i know is factory reset will solve it. i wish i knew so i can tell u and others but that is beyond me.

mooglestiltzkin · April 29, 2024, 6:33am

ok so in truenas ui u can factory reset. (obviously i had backed up first)

truenas looks new in UI, but i don’t know whether boot pool got cleaned as well or not.

For one of the pools i wiped it clean. but the other i managed to recover. Even the encryption pass phrase worked.

So i probably should have just done the same for the other, but o well

mooglestiltzkin · April 29, 2024, 7:45am

*update

managed a factory reset
recovered my pool with my datasets with the majority of my data (this saved me a lot of time.) Did not have to pull my drives before factory reset, it just left my pool on the drives alone. It would only delete it if you tell it to if you opt not to recover the pool.
wiped the vm pool/datasets. no problem sinced i backed up and i didn’t have as much data there in comparison to the other. should have just recovered pool to save time.
successfully setup rsync
fixed the acls & user creation (not sure if they are ideal but it’s working)
smb share working
finished setting up the bridge based on the youtube video

getting ready to install jailmaker