Unable to run more than 1 app at a time

Apps and Virtualization
,
1

After updating to Fangtooth (time reference, don’t know if related), I am unable to run more than one app at a time. That also includes creating new apps, as it attempts to start it right away.

I tested this theory by stopping my only running app (Jellyfin), then deploying a Portainer. It started up normally

But when I tried to restart Jellyfin, I get the “Failed ‘up’ action” error.

[EFAULT] Failed 'up' action for 'syncthing-josh' app. Please check /var/log/app_lifecycle.log for more details 
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/job.py", line 515, in run
    await self.future
  File "/usr/lib/python3/dist-packages/middlewared/job.py", line 562, in __run_body
    rv = await self.middleware.run_in_thread(self.method, *args)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 622, in run_in_thread
    return await self.run_in_executor(io_thread_pool_executor, method, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 619, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/api/base/decorator.py", line 101, in wrapped
    result = func(*args)
             ^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/apps/app_scale.py", line 62, in start
    compose_action(app_name, app_config['version'], 'up', force_recreate=True, remove_orphans=True)
  File "/usr/lib/python3/dist-packages/middlewared/plugins/apps/compose_utils.py", line 61, in compose_action
    raise CallError(err_msg)
middlewared.service_exception.CallError: [EFAULT] Failed 'up' action for 'jellyfin' app. Please check /var/log/app_lifecycle.log for more details
cat /var/log/app_lifecycle.log 
[2025/06/05 21:21:50] (ERROR) app_lifecycle.compose_action():56 - Failed 'up' action for 'syncthing-josh' app:  Network ix-syncthing-josh_default  Creating\n Network ix-syncthing-josh_default  Error\nfailed to create network ix-syncthing-josh_default: Error response from daemon: all predefined address pools have been fully subnetted\n
[2025/06/13 21:42:04] (ERROR) app_lifecycle.compose_action():56 - Failed 'up' action for 'syncthing-josh' app:  Network ix-syncthing-josh_default  Creating\n Network ix-syncthing-josh_default  Error\nfailed to create network ix-syncthing-josh_default: Error response from daemon: all predefined address pools have been fully subnetted\n
[2025/06/13 21:48:44] (ERROR) app_lifecycle.compose_action():56 - Failed 'up' action for 'portainer' app:  portainer Pulling \n ef4d36ba322a Pulling fs layer \n ef8207bd2730 Pulling fs layer \n a0deae24de9f Pulling fs layer \n feba0a818d42 Pulling fs layer \n d537ce5d8f56 Pulling fs layer \n 61c4aa519df8 Pulling fs layer \n dc27b963d4ea Pulling fs layer \n feba0a818d42 Waiting \n 4f4fb700ef54 Pulling fs layer \n d537ce5d8f56 Waiting \n 61c4aa519df8 Waiting \n dc27b963d4ea Waiting \n 4f4fb700ef54 Waiting \n ef4d36ba322a Downloading 
[>                                                  ]  2.301kB/128.3kB\n ef8207bd2730 Downloading [==================================================>]          93B/93B\n ef8207bd2730 Verifying Checksum \n ef8207bd2730 Download complete \n ef4d36ba322a Verifying Checksum \n ef4d36ba322a Download complete \n ef4d36ba322a Extracting 
[============>                                      ]  32.77kB/128.3kB\n ef4d36ba322a Extracting [==================================================>]  128.3kB/128.3kB\n a0deae24de9f Downloading [>                                                  ]  188.7kB/18.85MB\n ef4d36ba322a Pull complete \n a0deae24de9f Downloading [=============>                         ]  4.938MB/18.85MB\n ef8207bd2730 Extracting [==================================================>]          93B/93B\n ef8207bd2730 Extracting [==================================================>]          93B/93B\n a0deae24de9f Downloading [====================>                              ]  7.703MB/18.85MB\n feba0a818d42 Downloading [==================================================>]        439B/439B\n feba0a818d42 Verifying Checksum \n feba0a818d42 Download complete \n d537ce5d8f56 Downloading 
[>                                                  ]  294.4kB/29.06MB\n a0deae24de9f Downloading 
[===============================>                   ]  11.82MB/18.85MB\n ef8207bd2730 Pull complete \n d537ce5d8f56 Downloading [======>                                ]  3.541MB/29.06MB\n a0deae24de9f Downloading [========================================>          ]  15.17MB/18.85MB\n a0deae24de9f Downloading [==============================================>    ]  17.53MB/18.85MB\n d537ce5d8f56 Downloading [==========>                                        ]  5.901MB/29.06MB\n a0deae24de9f Verifying Checksum \n a0deae24de9f Download complete \n a0deae24de9f Extracting 
[>                                                  ]  196.6kB/18.85MB\n d537ce5d8f56 Downloading [===============>                                   ]  9.145MB/29.06MB\n 61c4aa519df8 Downloading [>                                                  ]  106.8kB/10.38MB\n a0deae24de9f Extracting [==>                                                ]    983kB/18.85MB\n 61c4aa519df8 Downloading [======================>                            ]  4.766MB/10.38MB\n d537ce5d8f56 Downloading [=====================>                             ]  12.68MB/29.06MB\n a0deae24de9f Extracting [========>                                          ]  3.146MB/18.85MB\n 61c4aa519df8 Downloading [================================================>  ]  10.16MB/10.38MB\n d537ce5d8f56 Downloading [==============================>                    ]  17.71MB/29.06MB\n 61c4aa519df8 Downloading [==================================================>]  10.38MB/10.38MB\n 61c4aa519df8 Verifying Checksum \n 61c4aa519df8 Download complete \n a0deae24de9f Extracting 
[==============>                                    ]  5.308MB/18.85MB\n dc27b963d4ea Downloading [==================>                                ]     934B/2.576kB\n dc27b963d4ea Downloading [==================================================>]  2.576kB/2.576kB\n dc27b963d4ea Verifying Checksum \n dc27b963d4ea Download complete \n d537ce5d8f56 Downloading 
[========================================>          ]  23.36MB/29.06MB\n d537ce5d8f56 Downloading [================================================>  ]  28.11MB/29.06MB\n a0deae24de9f Extracting [=================>                                 ]  6.685MB/18.85MB\n d537ce5d8f56 Verifying Checksum \n d537ce5d8f56 Download complete \n 4f4fb700ef54 Downloading 
[==================================================>]          32B/32B\n 4f4fb700ef54 Verifying Checksum \n 4f4fb700ef54 Download complete \n a0deae24de9f Extracting 
[=========================>                         ]  9.634MB/18.85MB\n a0deae24de9f Extracting [=======================================>           ]  14.94MB/18.85MB\n a0deae24de9f Extracting [=================================================> ]  18.48MB/18.85MB\n a0deae24de9f Extracting [==================================================>]  18.85MB/18.85MB\n a0deae24de9f Pull complete \n feba0a818d42 Extracting 
[==================================================>]        439B/439B\n feba0a818d42 Extracting [==================================================>]        439B/439B\n feba0a818d42 Pull complete \n d537ce5d8f56 Extracting 
[>                                                  ]  294.9kB/29.06MB\n d537ce5d8f56 Extracting [====>                                              ]  2.654MB/29.06MB\n d537ce5d8f56 Extracting [=======>                                           ]  4.424MB/29.06MB\n d537ce5d8f56 Extracting [=========>                                         ]  5.603MB/29.06MB\n d537ce5d8f56 Extracting [============>                                      ]  7.078MB/29.06MB\n d537ce5d8f56 Extracting [==============>                                    ]  8.258MB/29.06MB\n d537ce5d8f56 Extracting [================>                                  ]  9.437MB/29.06MB\n d537ce5d8f56 Extracting [==================>                                ]  10.62MB/29.06MB\n d537ce5d8f56 Extracting [====================>                              ]  12.09MB/29.06MB\n d537ce5d8f56 Extracting [========================>                          ]  14.16MB/29.06MB\n d537ce5d8f56 Extracting [===========================>                       ]  15.93MB/29.06MB\n d537ce5d8f56 Extracting [==============================>                    ]  17.69MB/29.06MB\n d537ce5d8f56 Extracting [================================>                  ]  18.87MB/29.06MB\n d537ce5d8f56 Extracting [=================================>                 ]  19.76MB/29.06MB\n d537ce5d8f56 Extracting [==================================>                ]  20.05MB/29.06MB\n d537ce5d8f56 Extracting [===================================>               ]  20.35MB/29.06MB\n d537ce5d8f56 Extracting [=====================================>             ]  21.53MB/29.06MB\n d537ce5d8f56 Extracting [=======================================>           ]     23MB/29.06MB\n d537ce5d8f56 Extracting [==========================================>        ]  24.77MB/29.06MB\n d537ce5d8f56 Extracting [==============================================>    ]  26.84MB/29.06MB\n d537ce5d8f56 Extracting [================================================>  ]  28.31MB/29.06MB\n d537ce5d8f56 Extracting [==================================================>]  29.06MB/29.06MB\n d537ce5d8f56 Pull complete \n 61c4aa519df8 Extracting [>                                                  ]  131.1kB/10.38MB\n 61c4aa519df8 Extracting [===>                                               ]  655.4kB/10.38MB\n 61c4aa519df8 Extracting [======>                                            ]  1.311MB/10.38MB\n 61c4aa519df8 Extracting [========>                                          ]  1.704MB/10.38MB\n 61c4aa519df8 Extracting [==========>                                        ]  2.228MB/10.38MB\n 61c4aa519df8 Extracting [=============>                                     ]  2.884MB/10.38MB\n 61c4aa519df8 Extracting [=====================>                             ]  4.456MB/10.38MB\n 61c4aa519df8 Extracting [============================>                      ]  5.898MB/10.38MB\n 61c4aa519df8 Extracting [====================================>              ]  7.602MB/10.38MB\n 61c4aa519df8 Extracting [=========================================>         ]   8.52MB/10.38MB\n 61c4aa519df8 Extracting [============================================>      ]  9.175MB/10.38MB\n 61c4aa519df8 Extracting [===============================================>   ]   9.83MB/10.38MB\n 61c4aa519df8 Extracting [==================================================>]  10.38MB/10.38MB\n 61c4aa519df8 Pull complete \n dc27b963d4ea Extracting [==================================================>]  2.576kB/2.576kB\n dc27b963d4ea Extracting [==================================================>]  2.576kB/2.576kB\n dc27b963d4ea Pull complete \n 4f4fb700ef54 Extracting [==================================================>]      32B/32B\n 4f4fb700ef54 Extracting [==================================================>]      32B/32B\n 4f4fb700ef54 Pull complete \n portainer Pulled \n Network ix-portainer_default  Creating\n Network ix-portainer_default  Error\nfailed to create network ix-portainer_default: Error response from daemon: all predefined address pools have been fully subnetted\n
[2025/06/13 22:06:30] (ERROR) app_lifecycle.compose_action():56 - Failed 'up' action for 'syncthing-josh' app:  Network ix-syncthing-josh_default  Creating\n Network ix-syncthing-josh_default  Error\nfailed to create network ix-syncthing-josh_default: Error response from daemon: all predefined address pools have been fully subnetted\n
[2025/06/14 10:18:05] (ERROR) app_lifecycle.compose_action():56 - Failed 'up' action for 'syncthing-josh' app:  Network ix-syncthing-josh_default  Creating\n Network ix-syncthing-josh_default  Error\nfailed to create network ix-syncthing-josh_default: Error response from daemon: all predefined address pools have been fully subnetted\n
[2025/06/14 10:19:56] (ERROR) app_lifecycle.compose_action():56 - Failed 'up' action for 'portainer' app:  Network ix-portainer_default  Creating\n Network ix-portainer_default  Error\nfailed to create network ix-portainer_default: Error response from daemon: all predefined address pools have been fully subnetted\n
[2025/06/14 10:23:14] (ERROR) app_lifecycle.compose_action():56 - Failed 'up' action for 'jellyfin' app:  Network ix-jellyfin_default  Creating\n Network ix-jellyfin_default  Error\nfailed to create network ix-jellyfin_default: Error response from daemon: all predefined address pools have been fully subnetted\n
[2025/06/14 10:29:55] (ERROR) app_lifecycle.compose_action():56 - Failed 'up' action for 'jellyfin' app:  Network ix-jellyfin_default  Creating\n Network ix-jellyfin_default  Error\nfailed to create network ix-jellyfin_default: Error response from daemon: all predefined address pools have been fully subnetted\n
[2025/06/14 11:12:09] (ERROR) app_lifecycle.compose_action():56 - Failed 'up' action for 'jellyfin' app:  Network ix-jellyfin_default  Creating\n Network ix-jellyfin_default  Error\nfailed to create network ix-jellyfin_default: Error response from daemon: all predefined address pools have been fully subnetted\n
[2025/06/14 11:12:09] (ERROR) app_lifecycle.compose_action():56 - Failed 'up' action for 'syncthing-josh' app:  Network ix-syncthing-josh_default  Creating\n Network ix-syncthing-josh_default  Error\nfailed to create network ix-syncthing-josh_default: Error response from daemon: all predefined address pools have been fully subnetted\n

app_lifecycle.log says “all predefined address pools have been fully subnetted”, yet I only have 1 app running, and 3 apps created, which is far from the subnet limit.

2

Can you post a screenshot of Apps → Settings?

3

image
image1920×1080 195 KB

4

I tried this “solution” that had worked previously, but it didn’t work this time.

https://forums.truenas.com/t/failed-up-action-for-jellyfin-app-how-tofix/36337/10?u=jossk

5

I see you changed the IP and available size.
What happens if you change the /24 back to /12?

6

image
image1920×1080 212 KB

Had to create a new address pool as I got an error that it overlaps with my existing network.

7

Ok. I still get the error, but now they seem to be stuck on Deploying instead of failing to start outright.

8

You need to remove the top address pool as shown on your last screenshot. That one is going to cause issues, it’s improperly sized.

If you don’t mind me asking, are you picking IPs at random?
These are private networks suitable for internal usage:
10.0.0.0/8
172.16.0.0/12
192.168.0.0/16
Use a chunk of those, not the whole networks.

Using 107.x.x.x, a network owned by an ISP, is an unnecessary risk.
And 127.x.x.x is close to the localhost address, causing needless confusion.

9

I have 107 set as my IoT network, and everything has been running fine, except for the apps.

So it seems AT&T in the US owns that subnet. Shouldn’t be a problem, as I’m not publishing anything externally.

As for the 127.x, I only did that just now as a test.

10

Just deleted that 127.x pool, and now I can’t start anything.

Just going to try rebooting.

11

As long as you have that 107.16.0.0/24 network with a docker network size of 24 you are going to have issues. It gives you room for exactly 1 docker network, and, as far as I can tell, TrueNAS allocates one docker network per app.

Edit: Also, I suspect you are creating extra headache if you have an address range on your LAN that overlaps the one available for internal docker networks.

1 Like
12

Just tried. I can confirm your observations.

Thank you

1 Like