Power Outage Causing Apps to fail

Apps and Virtualization
,
1

Good Morning, yesterday we suffered an extended power outage and this morning I can not get my apps service running. I get the below error when going to the apps page for the first time and then a second error message (further below). I’ve tried the unset/reset pool which throws this same error as the top code block. I’ve gone in and systemctl status docker shows docker is running but when I check docker ps -a there’s 0 containers.

Traceback (most recent call last):
File “/usr/lib/python3/dist-packages/middlewared/api/base/server/ws_handler/rpc.py”, line 323, in process_method_call
result = await method.call(app, params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/api/base/server/method.py”, line 52, in call
result = await self.middleware.call_with_audit(self.name, self.serviceobj, methodobj, params, app)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 911, in call_with_audit
result = await self._call(method, serviceobj, methodobj, params, app=app,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 720, in _call
return await methodobj(*prepared_call.args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/api/base/decorator.py”, line 93, in wrapped
result = await func(*args)
^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/plugins/catalog/apps.py”, line 21, in latest
await self.middleware.call(
File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 1005, in call
return await self._call(
^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 731, in _call
return await self.run_in_executor(prepared_call.executor, methodobj, *prepared_call.args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 624, in run_in_executor
return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3.11/concurrent/futures/thread.py”, line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/api/base/decorator.py”, line 101, in wrapped
result = func(*args)
^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/plugins/catalog/apps.py”, line 39, in available
for app in self.middleware.call_sync(‘app.query’)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 1041, in call_sync
return methodobj(*prepared_call.args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/api/base/decorator.py”, line 101, in wrapped
result = func(*args)
^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/plugins/apps/crud.py”, line 72, in query
apps = list_apps(available_apps_mapping, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/plugins/apps/ix_apps/query.py”, line 140, in list_apps
with os.scandir(get_app_parent_config_path()) as scan:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: ‘/mnt/.ix-apps/app_configs’

Error message beside the configuration drop down menu:

Application(s) have failed to start:
[EFAULT] Unable to determine default interface

I suspect it has something to do with the mapping/pairing of /mnt/.ix-apps/app-configs I can see that it is mounted? linked? to pool-storage/ix-apps/app_configs but not sure how to make the apps panel make those two friends.

Any assistance would be very much appreciated.

- JB

2

Your pool might be damaged from the power bump. You may want to scrub your pool. First, does /mnt/.ix-apps exist?

3

Stupid question - can you ping google.com and ping docker.com? I usually see that when ipv4 fails for whatever reason.

4

It seems that the ping is failing to both Google and Docker. This also happens from a desktop that can get to both. Maybe has something to do with my Pi-Hole? I do get the IP address back for each domain.

5

Sorry for the delayed reply. After Tuesday’s storm cutting power for 11 hours we lost power again yesterday for reasons unknown for another 11 hours. Guess it is a good thing I didn’t get started right away since power cut about an hour after I posted this question.

I’ve initiated the scrub. I’m guessing this is scrub of the storage pool that the apps are sitting on, not a data pool that may be reference by those apps, yes?

/mnt/.ix-apps does exist and contains:

@[~]$ ls /mnt/.ix-apps/
app_configs app_mounts docker metadata.yaml truenas_catalog user_config.yaml

6

Until TrueNAS can reach both it’ll keep failing to figure out the default interface. Once it can reach 'em refresh the catalogue, unset/reset the pool & you should be back in business.