Running bluefin 22.12.1.
Server was being weird (got some alerts that didn’t make sense, see end), so I restarted middlewared and it failed to come up. Rebooted, and on reboot was greeted with middleware is not running. Press Enter to open root shell. (without any authentication at all, completely bypassing the root password(!)…)
Checking /var/log/middlewared.log:
ImportError: cannot import name 'DEFAULT_CIPHERS' from 'urllib3.util.ssl_' (/usr/local/lib/python3.9/dist-packages/urllib3/util/ssl_.py)
Server isn’t reachable on my network (I assume because middlewared didn’t set up my link aggregation and interfaces) so I can’t copy the full stack trace that easily but if someone needs the full one I will. Also none of my ZFS pools are present, assuming also due to middlewared failing.
Happy to do any troubleshooting steps. Was considering updating soon anyway (just been busy, but I guess now my hand might be forced…) as I’m guessing updating might just fix whatever weird python issue, but I’d like to actually find a solution if possible, if nothing else then for other people, plus haven’t upgraded without the webUI before so want to make sure I don’t screw up as this system has production data - is there a reasonable way to from the command line?
Example weird alert:
Failed to check for alert ScrubPaused: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/middlewared/plugins/alert.py", line 776, in __run_source alerts = (await alert_source.check()) or [] File "/usr/lib/python3/dist-packages/middlewared/alert/source/scrub_paused.py", line 18, in check for pool in await self.middleware.call("pool.query"): File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1386, in call return await self._call( File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1335, in _call return await methodobj(*prepared_call.args) File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1186, in nf res = await f(*args, **kwargs) File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1318, in nf return await func(*args, **kwargs) File "/usr/lib/python3/dist-packages/middlewared/service.py", line 909, in query result = await self.middleware.call( File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1386, in call return await self._call( File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1335, in _call return await methodobj(*prepared_call.args) File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1318, in nf return await func(*args, **kwargs) File "/usr/lib/python3/dist-packages/middlewared/plugins/datastore/read.py", line 164, in query result = await self._queryset_serialize( File "/usr/lib/python3/dist-packages/middlewared/plugins/datastore/read.py", line 216, in _queryset_serialize return [ File "/usr/lib/python3/dist-packages/middlewared/plugins/datastore/read.py", line 217, in await self._extend(data, extend, extend_context, extend_context_value, select) File "/usr/lib/python3/dist-packages/middlewared/plugins/datastore/read.py", line 230, in _extend data = await self.middleware.call(extend, data, extend_context_value) File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1386, in call return await self._call( File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1346, in _call return await self.run_in_executor(prepared_call.executor, methodobj, *prepared_call.args) File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1249, in run_in_executor return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs)) File "/usr/lib/python3.9/concurrent/futures/thread.py", line 52, in run result = self.fn(*self.args, **self.kwargs) File "/usr/lib/python3/dist-packages/middlewared/plugins/pool.py", line 605, in pool_extend pool |= self.middleware.call_sync('pool.pool_normalize_info', pool['name']) File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1405, in call_sync return self.run_coroutine(methodobj(*prepared_call.args)) File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1445, in run_coroutine return fut.result() File "/usr/lib/python3.9/concurrent/futures/_base.py", line 433, in result return self.__get_result() File "/usr/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result raise self._exception File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1318, in nf return await func(*args, **kwargs) File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1186, in nf res = await f(*args, **kwargs) File "/usr/lib/python3/dist-packages/middlewared/plugins/pool.py", line 565, in pool_normalize_info if info := await self.middleware.call('zfs.pool.query', [('name', '=', pool_name)]): File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1386, in call return await self._call( File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1343, in _call return await self._call_worker(name, *prepared_call.args) File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1349, in _call_worker return await self.run_in_proc(main_worker, name, args, job) File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1264, in run_in_proc return await self.run_in_executor(self.__procpool, method, *args, **kwargs) File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1249, in run_in_executor return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs)) concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.
Edit: last day with normal alerts was the 28th of april, where it failed between 2am and 5am based on alert emails. middlewared stayed up and running until I restarted it including sending emails.