IP forwarding in Docker

Apps and Virtualization
, ,
1

I am running my media stack as a Docker stack via Portainer which runs as an App on TrueNAS Scale EE since it came out yesterday.
Before that the stack was running in an Ubuntu 24.04 VM on the same TrueNAS machine. It’s NIC was bridged to my home network.
I am using port forwarding on my router to expose HTTP(s) ports to the internet.

I am using Traefik as an Ingress-Controller/Reverse proxy and I exposed ports 80 and 443 in host mode for it.
On Ubuntu Traefik forwarded the actual IP of Clients through the X-Forwarded-For and X-Real-Ip headers.
I used this in Jellyfin for bandwidth limitations based on whether the client was coming from outside my LAN.

I did not change anything in my docker compose file migrating from the VM to TrueNAS except for volume paths.

Now the headers mentioned above always show my routers WAN IP-Address, no matter whether the request comes from inside or outside the network.
I changed the port forwarding from the Ubuntu VM to TrueNAS itself and it works otherwise perfectly fine.

I also set up a whoami container and accessed it with a browser from within my LAN, it gave the following output:

Hostname: 123831009e4d
IP: 127.0.0.1
IP: ::1
IP: 172.18.0.6
RemoteAddr: 172.18.0.2:51416
GET / HTTP/1.1
Host: whoami.mydomain.tld
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36
Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/vif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7
Accept-Encoding: gzip, deflate, br, zstd
Accept-Language: de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7
Dnt: 1
Priority: u=0, i
Sec-Ch-Ua: "Chomium";v="130", "Google Chrome";v="130", "Not?A_Brand";v="99"
Sec-Ch-Ua-Mobile: ?0
Sec-Ch-Ua-Platform: "Windows"
Sec-Fetch-Dest: document
Sec-Fetch-Mode: navigate
Sec-Fetch-Site: none
Sec-Fetch-User: ?1
Upgrade-Insecure-Requests: 1
X-Forwarded-For: my.real.wan.ip
X-Forwarded-Host: whoami.mydomain.tld
X-Forwarded-Port: 443
X-Forwarded-Proto: https
X-Forwarded-Server: 642f647339e5
X-Real-Ip: my.real.wan.ip

172.18.0.2 is my Traefik’s IP.

Does anyone here have an idea, why I cannot see actual client IPs anymore and how I can get them again?

2

Okay I am running traefik and I spun up a whoami too take a took at this

my output

Hostname: 9e52b1f95625
IP: 127.0.0.1
IP: ::1
IP: 172.24.1.27
RemoteAddr: 172.24.5.0:37958
GET / HTTP/1.1
Host: redacted
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7
Accept-Encoding: gzip, deflate, br, zstd
Accept-Language: en-US,en;q=0.9
Cookie: authelia_session=fBNPLwWD#jyQc-vBc819^bsH_YyfJZ%F
Priority: u=0, i
Purpose: prefetch
Sec-Ch-Ua: "Chromium";v="130", "Google Chrome";v="130", "Not?A_Brand";v="99"
Sec-Ch-Ua-Mobile: ?0
Sec-Ch-Ua-Platform: "Windows"
Sec-Fetch-Dest: document
Sec-Fetch-Mode: navigate
Sec-Fetch-Site: none
Sec-Fetch-User: ?1
Sec-Purpose: prefetch;prerender
Upgrade-Insecure-Requests: 1
X-Forwarded-For: redacted
X-Forwarded-Host: redacted
X-Forwarded-Port: 443
X-Forwarded-Proto: https
X-Forwarded-Server: traefik
X-Real-Ip: redacted

X-Forwarded-For
and
X-Real-Ip

both came up as my REMOTE WAN IP

Just to check your labels

on the traefik service itself

labels:
      - traefik.enable=true
      - traefik.http.routers.traefik.entrypoints=websecure
      - traefik.http.routers.traefik.rule=Host(`${TRAEFIK_DN}`)
      - traefik.http.routers.traefik.tls=true
      - traefik.http.routers.traefik.tls.certresolver=cloudflare
      - traefik.http.routers.traefik.tls.domains[0].main=${FQDN}
      - traefik.http.routers.traefik.tls.domains[0].sans=${SDN}
      - traefik.http.routers.traefik.service=api@internal
      - traefik.http.services.traefik.loadbalancer.server.port=8080
      - traefik.http.routers.traefik.middlewares=authelia@docker

on whoami

    labels:
      - traefik.enable=true
      - traefik.http.routers.whoami.entrypoints=websecure
      - traefik.http.routers.whoami.rule=Host(`${TEST_DN}`)
      - traefik.http.routers.whoami.tls=true
      - traefik.http.services.whoami.loadbalancer.server.port=80

Now the treafik config

global:
  checkNewVersion: false
  sendAnonymousUsage: false

api:
  dashboard: true
  disableDashboardAd: true
  insecure: true
  debug: true

entryPoints:
  web:
    address: ":80"
    http:
      redirections:
        entryPoint:
          to: websecure
          scheme: https
  websecure:
    address: :443

serversTransport:
  insecureSkipVerify: true

certificatesResolvers:  REDACTED rest of file redacted

this config is passing the REMOTE WAN IP through… I hope that helps… I assume the issue is a miss config

3

If I understand you correctly you’re facing the same issue as me then?
The real and forwarded headers should be, at least when coming from outside of the local network, the WAN IP of the client, not my own WAN IP.
When coming from inside the network it should also be some internal IP, preferrably the internal IP of the client device.

That’s the way I would have expected it to be and that’s also how it was on the Ubuntu VM.

Just FYI, my Traefik command:

      - --api.insecure=true
      - --providers.docker
      - --providers.docker.exposedByDefault=false
      - --entrypoints.web80.address=:80
      - --entrypoints.web80.http.redirections.entrypoint.to=web443
      - --entrypoints.web443.address=:443
      - --entrypoints.web443.http.tls.certresolver=cloudflare
      - --certificatesresolvers.cloudflare.acme.dnschallenge=true
      - --certificatesresolvers.cloudflare.acme.dnschallenge.provider=cloudflare
      - --certificatesresolvers.cloudflare.acme.email=REDACTED
      - --certificatesresolvers.cloudflare.acme.storage=/letsencrypt/acme-cloudflare.json

Traefik’s network and port config:

    networks:
      mediastack:
        ipv4_address: 172.18.0.2
    ports:
      - target: 80
        published: 80
        mode: host
      - target: 443
        published: 443
        mode: host
      - "8080:8080"

My docker network:

networks:
  mediastack:
    ipam:
      driver: default
      config:
        - subnet: 172.18.0.0/16
          gateway: 172.18.0.1

And my whoami config (which is basically the same for my jellyfin which has the same “problem”):

    labels:
      - traefik.enable=true
      - traefik.http.services.whoami.loadbalancer.server.port=80
      - traefik.http.routers.whoami.rule=Host(`whoami.my.tld`)
      - traefik.http.routers.whoami.entrypoints=web443

    networks:
      mediastack:
        ipv4_address: 172.18.0.6

EDIT: Now after a few restarts when I am coming from outside my local network the correct IPs get reported. The only remaining problem then is reporting the correct IPs from inside the network.
EDIT2: When I add my TrueNAS to my hosts file for my domain, the containers can see the correct domain. As this was not necessary before I don’t know how to interpret this yet.

4

EDIT

with both your edits yes that is exactly what I am seeing…

I have correct remote WAN IP from a connection outside the local network but from within the network I am showing a hand off from the proxy which I have always accepted as normal

5

After further investigation…

Remote Users proper WAN IP shows

Local users are showing wrong LAN IP from the Reverse Proxy

The reverse proxy Traefik is forwarding the correct headers to the APP, in my case plex in your jellyfin

In other apps not video apps I am seeing the correct IPs from lan clients so I have to think this is not a Traefik issue but a Plex and Jellyfin issue

I found this
https://www.reddit.com/r/PleX/comments/edxkh0/plex_not_respecting_xforwardedfor_or_xrealip/

I also tried this (it did not) but it should not have mattered as Traefik was already sending plex the correct info

- "traefik.http.middlewares.plex-forwarded-headers.headers.customrequestheaders.X-Forwarded-For:"  # Empty value; Traefik populates it automatically
- "traefik.http.middlewares.plex-forwarded-headers.headers.customrequestheaders.X-Real-IP:"        # Empty value; Traefik populates it automatically
- "traefik.http.routers.plex.middlewares=plex-forwarded-headers"

I think this is an issue with Plex and Jellyfin just saying of we didnt get a public IP post the last seen private one in their code

***If host network is used instead this would likely solve the issue