Trouble with Nextcloud on TrueNAS Scale 25.04 with separate IP

Apps and Virtualization
1

Hi all,

I am fighting the following issue for quite a while now.

I finally migrated from TrueNAS Core a couple of weeks back and am working now on rebuilding the apps for my home server.

I am running TrueNAS behind an OPNsense firewall, with a couple of vlans, to keep some of the exposed services in DMZs. The services are exposed through HAproxy on OPNsense and behind geoblocking to keep things safe.
For that reason, I have defined two vlans on the secondary NIC and two bridges on TrueNAS for the two DMZs, all of this worked perfectly on Core.

On Scale, there is some inconsistancy. While generally, I was able to also define vlans on the secondary nic, with their corresponding bridges and adding alias IPs to them, it doesnt work for all apps.

Plex (community app) and haugene/transmission-openvpn (custom docker app) in the Media DMZ work perfectly fine, they are reachable on their IP alias and traffic really runs on the secondary NIC.

Onlyoffice (community app) also works in my cloud DMZ, showing that my (temporary, quite open) rules on OPNsense do what they should: pass traffic. I will tighten them later.

However, Nextcloud, not so much.

  • When binding the IP to 0.0.0.0, i can reach Nextcloud on my TrueNAS host IP under the specified port. But I want to separate exposed services from SMB and WebUI, so this is no solution for me.

  • But when binding the IP to one of the bridge alias IP, I get no answer (connection timed out).

Some debugging steps (after a long chat with chatGPT):

  • checking the port, that nextcloud listens to with netstat -tulpn
    ( same with 0.0.0.0 and bridge-alias-IP)

  • curl -v from Truenas host: get connection. From client PC: no answer.

  • tried changing apache2 configs to listen on IPv4 instead of v6 (even though this didnt seem to matter when binding to 0000)

  • tcpdump: SYN, but no ACK

I am really puzzled by this, anybody able to reproduce this, or give any pointers to what I might be doing wrong? Could it also be a bug (maybe in the nextcloud app), since it works perfectly fine for all other services I am using.

Thanks in advance

weingeist

2

Good evening,

I am still trying to do this and would like to give some more details.

Network setup:

  • eno3: secondary NIC, no DHCP, no IP aliases defined
  • vlan50 with parent interface eno3 and vlan tag 50, no DHCP, no IP aliases defined.
  • br50, bridge with member vlan50, learning enabled, and two IP aliases:
    • 192.168.50.130 (Nextcloud)
    • 192.168.50.140 (Onlyoffice)

The same setup is running for vlan 60 also on eno3 NIC for Plex and Transmission, no issues there.

I am comparing three docker services:

  • A: nextcloud on host IP 0.0.0.0 and port 8081
  • B: nextcloud on host IP 192.168.50.130 and port 8085
  • C: onlyoffice on host IP 192.168.50.140 and port 30134

B is the only one I cannot reach.

Here is tcpdump for each:
A (Nextcloud responds under the host IP):

21:04:22.381399 IP 192.168.10.14.59965 > 192.168.30.100.8081: Flags [S], seq 472301906, win 64240, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
21:04:22.381529 IP 192.168.30.100.8081 > 192.168.10.14.59965: Flags [S.], seq 947856733, ack 472301907, win 64240, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
21:04:22.384697 IP 192.168.10.14.59965 > 192.168.30.100.8081: Flags [.], ack 1, win 513, length 0
21:04:22.386534 IP 192.168.10.14.59965 > 192.168.30.100.8081: Flags [P.], seq 1:648, ack 1, win 513, length 647
21:04:22.386638 IP 192.168.30.100.8081 > 192.168.10.14.59965: Flags [.], ack 648, win 497, length 0
21:04:22.546044 IP 192.168.30.100.8081 > 192.168.10.14.59965: Flags [P.], seq 1:4381, ack 648, win 497, length 4380
21:04:22.546061 IP 192.168.30.100.8081 > 192.168.10.14.59965: Flags [P.], seq 4381:8761, ack 648, win 497, length 4380
21:04:22.546074 IP 192.168.30.100.8081 > 192.168.10.14.59965: Flags [P.], seq 8761:9559, ack 648, win 497, length 798
21:04:22.548304 IP 192.168.10.14.59965 > 192.168.30.100.8081: Flags [.], ack 9559, win 513, length 0
21:04:27.552366 IP 192.168.30.100.8081 > 192.168.10.14.59965: Flags [F.], seq 9559, ack 648, win 497, length 0
21:04:27.552657 IP 192.168.10.14.59965 > 192.168.30.100.8081: Flags [F.], seq 648, ack 9559, win 513, length 0
21:04:27.552743 IP 192.168.30.100.8081 > 192.168.10.14.59965: Flags [.], ack 649, win 497, length 0
21:04:27.763558 IP 192.168.30.100.8081 > 192.168.10.14.59965: Flags [F.], seq 9559, ack 649, win 497, length 0
21:04:27.766537 IP 192.168.10.14.59965 > 192.168.30.100.8081: Flags [.], ack 9560, win 0, length 0
^C

B (no response from nextcloud in vlan50):

21:01:17.271260 IP 192.168.10.14.59919 > 192.168.50.130.8085: Flags [S], seq 1199755918, win 64240, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
21:01:17.520827 IP 192.168.10.14.59920 > 192.168.50.130.8085: Flags [S], seq 597157365, win 64240, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
21:01:18.274262 IP 192.168.10.14.59919 > 192.168.50.130.8085: Flags [S], seq 1199755918, win 64240, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
21:01:18.531063 IP 192.168.10.14.59920 > 192.168.50.130.8085: Flags [S], seq 597157365, win 64240, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
21:01:20.280165 IP 192.168.10.14.59919 > 192.168.50.130.8085: Flags [S], seq 1199755918, win 64240, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
21:01:20.536541 IP 192.168.10.14.59920 > 192.168.50.130.8085: Flags [S], seq 597157365, win 64240, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
21:01:24.287728 IP 192.168.10.14.59919 > 192.168.50.130.8085: Flags [S], seq 1199755918, win 64240, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
21:01:24.546206 IP 192.168.10.14.59920 > 192.168.50.130.8085: Flags [S], seq 597157365, win 64240, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0

C (onlyoffice living in vlan50 responds):

21:00:59.947647 IP 192.168.10.14.59914 > 192.168.50.140.30134: Flags [.], ack 3237382707, win 513, length 0
21:00:59.955243 IP 192.168.10.14.59914 > 192.168.50.140.30134: Flags [.], seq 0:1460, ack 1, win 513, length 1460
21:00:59.958420 IP 192.168.10.14.59914 > 192.168.50.140.30134: Flags [P.], seq 1460:1876, ack 1, win 513, length 416
21:00:59.969118 IP 192.168.10.14.59914 > 192.168.50.140.30134: Flags [.], ack 1704, win 513, length 0
21:00:59.977676 IP 192.168.10.14.59914 > 192.168.50.140.30134: Flags [P.], seq 1876:1969, ack 1704, win 513, length 93
21:00:59.982413 IP 192.168.10.14.59914 > 192.168.50.140.30134: Flags [.], ack 2031, win 512, length 0
21:00:59.984142 IP 192.168.10.14.59914 > 192.168.50.140.30134: Flags [P.], seq 1969:2068, ack 2031, win 512, length 99
21:00:59.984142 IP 192.168.10.14.59914 > 192.168.50.140.30134: Flags [P.], seq 2068:2413, ack 2031, win 512, length 345
21:00:59.985699 IP 192.168.10.14.59914 > 192.168.50.140.30134: Flags [P.], seq 2413:2451, ack 2031, win 512, length 38
21:00:59.988260 IP 192.168.10.14.59914 > 192.168.50.140.30134: Flags [.], ack 4528, win 513, length 0
21:01:00.179170 IP 192.168.10.14.59914 > 192.168.50.140.30134: Flags [P.], seq 2451:2584, ack 4528, win 513, length 133
21:01:00.179750 IP 192.168.10.14.59914 > 192.168.50.140.30134: Flags [P.], seq 2584:2626, ack 4528, win 513, length 42
21:01:00.182104 IP 192.168.10.14.59914 > 192.168.50.140.30134: Flags [.], ack 6353, win 513, length 0

However docker B does give the proper response when curl’ing from Truenas host system:

root@hostname[/home/truenas_admin]# curl -v 192.168.50.130:8085
*   Trying 192.168.50.130:8085...
* Connected to 192.168.50.130 (192.168.50.130) port 8085 (#0)
> GET / HTTP/1.1
> Host: 192.168.50.130:8085
> User-Agent: curl/7.88.1
> Accept: */*
>
< HTTP/1.1 200 OK

Here the few things I tested:
Netstat output of each docker container:
A:

root@hostname:/var/www/html# netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.11:42861        0.0.0.0:*               LISTEN      -
tcp6       0      0 :::80                   :::*                    LISTEN      1/apache2
udp        0      0 127.0.0.11:39051        0.0.0.0:*                           -

B:

root@hostname:/var/www/html# netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.11:40803        0.0.0.0:*               LISTEN      -
tcp6       0      0 :::80                   :::*                    LISTEN      1/apache2
udp        0      0 127.0.0.11:59480        0.0.0.0:*                           -

C:

root@hostname:/# netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:25672           0.0.0.0:*               LISTEN      -
tcp        0      0 127.0.0.11:42359        0.0.0.0:*               LISTEN      -
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      831/nginx: master p
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      831/nginx: master p
tcp        0      0 0.0.0.0:4369            0.0.0.0:*               LISTEN      -
tcp6       0      0 :::80                   :::*                    LISTEN      831/nginx: master p
tcp6       0      0 :::443                  :::*                    LISTEN      831/nginx: master p
tcp6       0      0 :::8000                 :::*                    LISTEN      -
tcp6       0      0 :::5672                 :::*                    LISTEN      -
tcp6       0      0 :::4369                 :::*                    LISTEN      -
udp        0      0 127.0.0.11:59230        0.0.0.0:*                           -

I found it weird, that Nextcloud seems to be listening with tcp6 and I still can reach it with IPv4, for case A:. So I rule that out as an issue.
But still, I tried changing the apache2 ports.conf and the conf under enabled-sites to listen on IPv4 as well with no change.

Output of iptables on Truenas:
prerouting:

root@hostname[/home/truenas_admin]# iptables -t nat -vnL PREROUTING
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
 2289  134K DOCKER     0    --  *      *       0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL

DOCKER:

root@hostname[/home/truenas_admin]# iptables -t nat -vnL DOCKER
Chain DOCKER (2 references)
 pkts bytes target     prot opt in     out     source               destination
    4   240 RETURN     0    --  br-1399cc9679bb *       0.0.0.0/0            0.0.0.0/0
    0     0 RETURN     0    --  br-8ecc3856b966 *       0.0.0.0/0            0.0.0.0/0
    0     0 RETURN     0    --  br-4c36b13cce1f *       0.0.0.0/0            0.0.0.0/0
    0     0 RETURN     0    --  docker0 *       0.0.0.0/0            0.0.0.0/0
    2   104 DNAT       6    --  !br-4c36b13cce1f *       0.0.0.0/0            192.168.50.140       tcp dpt:30134 to:172.16.4.4:443
    0     0 DNAT       6    --  !br-8ecc3856b966 *       0.0.0.0/0            192.168.50.130       tcp dpt:8085 to:172.16.3.5:80
   70  3640 DNAT       6    --  !br-1399cc9679bb *       0.0.0.0/0            0.0.0.0/0            tcp dpt:8081 to:172.16.5.4:80

The last three lines are C (Onlyoffice), B (Nextcloud on vlan50) and A (Nextcloud on 0.0.0.0). You can see that Nextcloud on vlan50 has no packets.

So while A and B (Nextcloud on 0.0.0.0 and vlan50) are identical inside the docker and B and C (Nextcloud and Onlyoffice both on vlan50) are identical from a network configuration point of view, how can it be possible that in both comparisons one works and the other doesnt?

I hope that someone might be able to spot an issue in my configuration.

Thanks in advance.