Inter-app communication in 24.10 Electric Eel

[tjhorner]

Problem/Justification

In TrueNAS SCALE 24.04, it was possible for apps to communicate with each other via cluster.local domain names since they were all on the same network. For example, I use the Nginx Proxy Manager app to reverse proxy to another app on the same TrueNAS machine by using its internal domain name, such as http://immich.ix-immich.svc.cluster.local:30041.

In 24.10, this is no longer possible if using the default apps catalog since each app has its own Docker network that is not shared with other apps. Users may work around this by referencing the TrueNAS host’s IP or hostname in place of the previously-used *.cluster.local name, but this creates a dependency between the app and the host and requires the user to expose the necessary port on the host, which may not always be desirable. Another possible way is by using a custom docker-compose YAML file.

Impact

As it currently stands, apps that were once able to communicate with each other are no longer able to. In addition to the reverse proxy use case above, this type of functionality is useful for apps like Sonarr, Radarr, and Prowlarr which communicate with each other and with download clients. I imagine this is a very popular use case for TrueNAS users.

Allowing apps to easily communicate with each other will ease disruption created by the k8s → Docker migration for users upgrading to 24.10 (myself included).

User Story

I can see two possible implementations of this feature.

Implementation 1: All apps share a network

This is most similar to how it worked in 24.04. All apps could simply share a network and all users need to do is change usages of the *.cluster.local names to their Docker network counterpart. For example, immich.ix-immich.svc.cluster.local would become ix-immich-immich-1.

Implementation 2: User-defined networks

By default, apps will only be part of its own network (as the situation is right now). Users will be able to create their own network from the Apps UI or CLI and configure apps to be part of these networks.

This is a slightly more secure option since it facilitates the principle of least privilege, allowing users to dictate which apps can and can’t connect to one another. But it is also slightly more effort.

---

For a clean K8S → Docker migration, I think it makes sense to work on Implementation 1 as a first pass, then save Implementation 2 for a larger release.

[Clonton]

Thankful that someone else brought this up. I’ve been running into issues with NPM for this exact reason, as I have a second server running and would like to connect the docker services on both devices via an overlay network, but TrueNAS Scale is a little limiting in this regard.

[tjhorner]

I’m kind of surprised nobody brought it up during the release candidate testing!

[Clonton]

I would assume because container orchestration is primarily focused around Kubernetes as opposed to docker swarm, but I wish there was at least the freedom to choose. I will most likely have to either just use K8s or find some solution from the community. Oh well.

[truenas-fan]

Plain Docker can do it. It’s just networking understanding and manipulation.

Learn about NAT, Docker networks, docker compose, network type host, network type macvlan.

[tjhorner]

Yes, it’s very simple to do this with normal unmanaged Docker or with a custom docker-compose YAML app (as I mentioned in the original post), but that requires breaking out of the TrueNAS-managed app catalog. I’d like to stick with the official catalog to keep my TrueNAS host low-maintenance and as appliance-like as possible. So this functionality should be available from the 24.10 Apps UI, especially since it was possible in 24.04 and is very commonly used.

[krojew]

I just stumbled into this. Basically, all inter-app communication is now broken from official apps. This makes apps like NPM useless, since they can’t proxy to anything.

[william]

Why can’t you use the $HOSTIP:port ?

[krojew]

This requires exposing containers to the outside. Also, the ip might change.

[tjhorner]

1. Not all ports are forwarded from container to host. For example, Redis in the app catalog does not forward port 6739 (the port for the actual Redis daemon), and there is no way to define that.
2. It creates an unnecessary dependency on the host network — if the TrueNAS hostname or IP changes (depending on which is used), it will break communication between apps.
3. It’s good security practice to not expose ports on the host unless it is necessary.

[Captain_Morgan]

I assume anyone deploying Apps with Dockge and portainer can overcome this problem?

We’d like to improve networking flexibility in future versions, but supported Dockge and Portainer as both a short-term and a sustainable fix.

[tjhorner]

Sure, if you are starting from scratch… I am on 24.04 and my upgrade path to 24.10 is looking really bumpy at the moment, which is kind of disappointing because I would love to upgrade ASAP. At minimum I would like to see all apps on the same network, which shouldn’t be difficult to implement and wouldn’t be a breaking change.

This raises a few questions in my mind: In 24.10, do apps deployed from the official catalog show up as Compose stacks in Portainer, or only custom ones? And would changes to these stacks persist across updates? (my intuition says no but just want to double check, as it changes the feasibility and sustainability of this workaround)

How long will 24.04 and its app catalog be officially supported?

Thanks for considering this.

[Captain_Morgan]

@tjhorner

This raises a few questions in my mind: In 24.10, do apps deployed from the official catalog show up as Compose stacks in Portainer, or only custom ones?

Only custom apps show up in Portainer if deployed that way.

At this stage, you would need to deploy Portainer and then deploy the Compose stacks you want with networking you want.

The networking option for official Apps are limited in Electric eel and will be improved in Fangtooth. Feel free to stay in Dragonfish if this is an issue.

[Stux]

@tjhorner

In 24.10, do apps deployed from the official catalog show up as Compose stacks in Portainer

They show up, but are not editable. Same in Dockge.

@tjhorner

but that requires breaking out of the TrueNAS-managed app catalog. I’d like to stick with the official catalog

Right now, there is no way to do this with the official catalog.

I think the ability to bind an app port to a specific IP alias may be an app catalog feature soon, but custom networking with app catalog apps will probably not be a feature in EE. Maybe it will be in Fangtooth. It was originally mentioned as an Electric Eel feature.

You can perform IP binding or setup custom networks and macvlan networks using custom yaml apps or raw docker-compose, Dockge or Portainer.

(I am working on a video for this)

No. They won’t appear in the app’s view, unless you use the custom yaml option.

Perhaps vote for these features

Display non-ix managed stacks and containers on apps screen

Electric Eel includes Docker, and Docker Compose is used to power the apps The apps screen displays IX managed stacks. It would be helpful to also view non-IX managed stacks, and non stack containers. They could be displayed with generic icons, and the Details information could show generic information. docker has a property on all stacks which defines their stack, and containers without this property are not stacks. This would allow a user to understand what containers and stacks are runni…

Add "Bind IP Address" to app "port" selection UIs

There is no way to bind an app’s port mapping to a specific IP address in Electric Eel beta. If an IP Alias is added to an interface, then that IP Alias can be used when configuring port mapping in raw docker-compose. eg: ports: - 192.168.0.40:80:80 Will bind port 80 on the host’s 192.168.0.40 IP alias to port 80 in the container. Of course, for the above to work, the TrueNAS UI must first be bound to a specific IP instead of 0.0.0.0 Similarly, docker will bind to 0.0.0.0 by default,…

Allow apps to have their own IP

Problem/Justification Add ability to use separate IPs on apps in Electric Eel. With GUI manageability. Impact It will be similar to a CORE jail. User Story I don’t think this needs much of a story. It’s very practical to have apps using their own network stack.

Although none of those are actually the custom network feature you are requesting.

I think the custom network feature could be implemented with 1) a network editing gui and 2) app config options for assigning networks.

And if you decide that you do want to go the raw compose/Dockge route, a smooth migration can be performed by first migrating to docker compose in a sandbox in Dragonfish

TrueNAS Scale: Setting up Sandboxes with Jailmaker

And then you can migrate from the sandbox to native docker (or not)

TrueNAS Scale: Migrating Sandboxes to Docker in Electric Eel // Compose, Dockge & Jailmaker

Of course, the other option is to wait for Fangtooth to see if that satisfies your requirements.

[tjhorner]

As a stop-gap until this is implemented in Fangtooth, I made a small service that creates a Docker network and automatically attaches all containers managed by middlewared, and adds a DNS alias in the k8s-style format for each service, effectively restoring the inter-app networking functionality from Dragonfish.

I’ve tried it on EE in a VM and it seems to work well. To test, I deployed both pgAdmin and Postgres as separate apps and tried connecting pgAdmin to the Postgres instance using its DNS name:

Here’s a link if anyone else wants to try it out:

GitHub - tjhorner/dragonify: TrueNAS SCALE Dragonfish-style networking for...

Please feel free to open an issue on the repo if needed. I will be upgrading my production TrueNAS box to Electric Eel and performing more in-depth testing of this tool in a week or so.

[Stux]

Very neat

[Nikolas_Kurnia]

Really want to upgrade to Electric Eel, but was blocked by this

[tjhorner]

Would you be willing to test out the tool I made to see if it allows you to migrate seamlessly? I have tested it on an Electric Eel VM, and a friend of mine has tried it out on their EE box and confirmed the app containers are available at svc.cluster.local addresses as before, so I’m pretty confident it will work.

I haven’t had a chance to test it coming from a Dragonfish migration yet, so I would recommend having a backup handy in case you need to roll back. But I would really appreciate if you could try it out and let me know if it works for you.

(The only reason I haven’t upgraded my own machine to EE is because I am leaving for vacation next week and I figure it’s probably not the best idea to perform a major upgrade so close to that )

Here is the link if you’d like to try: GitHub - tjhorner/dragonify: TrueNAS SCALE Dragonfish-style networking for Electric Eel