Please help me understand what I am observing here.
I have a small NAS doing mostly nothing, but looking at the CPU usage and Temps over a week period, I see an increase I can can’t explain.
I am running a few containers and no VMs.
A reboot fixes it, but then it slowly goes back up as it was after a week.
Upgrade to Fangtooth was made on 19th of April.
1 Hour
1 Week
1 Month
This is an easy one to answer… Nothing is wrong.
You are interpreting the charts incorrectly.
1 Week is the history for the one week, 1 month is the entire history for the entire one month. This means that any spikes are the Maximum value and the chart resizes to accommodate the largest value.
To help you out, look at the Mean values. They are the average over the period selected. You also have Max and Min, but the Mean is the one for your given concern.
Cheers!
One thing is to interpret the graphs wrong, the other is the room temperature getting warmer because of the fictitious load or whatever is happening.
This is after today’s reboot:
Same was observed here on 4th May:
Could be an errant process in one of your containers.
Running htop might find the culprit.
This is my number one suspect, that one of the containers is causing this.
I have stopped some of them and will observe what is happening in the following days. Like I said, mostly this NAS is idle.
Any idea how to monitor a specific container? Historic CPU usage etc?
What is middlewared and what is it doing?
I have stopped ALL containers now. I’m the only user on this NAS, logged in in the shell view.
No SMB or anything being used, but there is still some load on the CPU.
middlewared is the process that runs the truenas webui and almost everything truenas related…
It shouldn’t constantly be using 10% - 17% CPU. 
@rumplin, did you also notice your non-ARC memory climbing over time?
That’s ARC. I was curious about non-ARC.
You can look at current middleware seesions midclt call auth.sessions | jq in 25.04 and earlier or simply who in master to see sessions. 10-17% CPU (or more) is certainly possible if there are scripts constantly calling midclt commands or 3rd party integrations spamming REST endpoints.
Just to be clear, this NAS was fine before the upgrade to Fangtooth. CPU was almost at 0% utilization. Unfortunately, I don’t have any screenshots from the CPU timeline with the Electric Eel version.
Interestingly, I have another NAS running Electric Eel with a VM and ~20 containers and the load there is minimal:
This makes me wonder what middlewared is doing on the smaller NAS where nothing is happening?
You can look at /var/log/middlewared.log for errors, and as I mentioned above look for unexpected active sessions.
In theory you can run midclt call core.threads_stacks to see what middleware calls are on-stack.
I didn’t know the CPU usage was seemingly abnormal. When I see small cpu changes, i think background tasks.
Another thing you might try is a reset to factoy of the configuration, after saving a copy of your current config. Then see if the problem still exists. Make sure you import your pool. If all looks good, start to reconfigure to see what causes the problem.
This is the harder way, hopefully you can identify it using the info @awalkerix is providing.
That’s also a good point. Generally some CPU utilization from middlewared is expected since it performs background tasks. You’re also comparing CPU on two different systems so potentially an apples / oranges situation.
Now I tried something. I stopped ALL containers and did a reboot.
Before I stopped the containers after the reboot.
There is a difference on the graph. If I stop the containers after the reboot, there is still some activity going on. But stopping containers and reboot, there is idle status.
I’m not sure I grasp how middlewared forks workers and services connections. It looks like the children aren’t closing the parents’ other connections? And there’s a conspicuous number of threads waiting around to service queues.
Here’s a Fangtooth-from-scratch, with the apps and instances services configured but with no apps or instances added. These recurring groups of sequentially forked processes feel a bit off? I should go check the source code.
$ sudo midclt call auth.sessions | jq -c '.[] | {origin,internal,user:.credentials_data.username}'
{"origin":"UNIX socket (pid=31066 uid=0 gid=0)","internal":true,"user":"root"}
{"origin":"UNIX socket (pid=31085 uid=0 gid=0)","internal":true,"user":"root"}
{"origin":"UNIX socket (pid=31302 uid=0 gid=0)","internal":true,"user":"root"}
{"origin":"UNIX socket (pid=31353 uid=0 gid=0)","internal":true,"user":"root"}
{"origin":"UNIX socket (pid=31506 uid=0 gid=0)","internal":true,"user":"root"}
{"origin":"UNIX socket (pid=32034 uid=0 gid=0)","internal":true,"user":"root"}
$ sudo lsof | grep 'middlewared.*sock'
asyncio_l 912 root 20u unix 0x00000000ef04bc7f 0t0 5342 /var/run/middleware/middlewared-internal.sock type=STREAM (LISTEN)
asyncio_l 912 root 53u unix 0x00000000cf15d90b 0t0 3711 /var/run/middleware/middlewared.sock type=STREAM (LISTEN)
asyncio_l 912 root 58u unix 0x0000000038a952f7 0t0 165223 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 root 59u unix 0x00000000aee7d3f8 0t0 165987 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 root 63u unix 0x00000000b9ceed85 0t0 166336 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 root 64u unix 0x0000000019ce13ab 0t0 165442 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 root 65u unix 0x00000000faf90525 0t0 165028 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 921 middlewar root 20u unix 0x00000000ef04bc7f 0t0 5342 /var/run/middleware/middlewared-internal.sock type=STREAM (LISTEN)
asyncio_l 912 921 middlewar root 53u unix 0x00000000cf15d90b 0t0 3711 /var/run/middleware/middlewared.sock type=STREAM (LISTEN)
asyncio_l 912 921 middlewar root 58u unix 0x0000000038a952f7 0t0 165223 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 921 middlewar root 59u unix 0x00000000aee7d3f8 0t0 165987 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 921 middlewar root 63u unix 0x00000000b9ceed85 0t0 166336 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 921 middlewar root 64u unix 0x0000000019ce13ab 0t0 165442 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 921 middlewar root 65u unix 0x00000000faf90525 0t0 165028 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 922 middlewar root 20u unix 0x00000000ef04bc7f 0t0 5342 /var/run/middleware/middlewared-internal.sock type=STREAM (LISTEN)
asyncio_l 912 922 middlewar root 53u unix 0x00000000cf15d90b 0t0 3711 /var/run/middleware/middlewared.sock type=STREAM (LISTEN)
asyncio_l 912 922 middlewar root 58u unix 0x0000000038a952f7 0t0 165223 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 922 middlewar root 59u unix 0x00000000aee7d3f8 0t0 165987 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 922 middlewar root 63u unix 0x00000000b9ceed85 0t0 166336 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 922 middlewar root 64u unix 0x0000000019ce13ab 0t0 165442 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 922 middlewar root 65u unix 0x00000000faf90525 0t0 165028 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 923 middlewar root 20u unix 0x00000000ef04bc7f 0t0 5342 /var/run/middleware/middlewared-internal.sock type=STREAM (LISTEN)
asyncio_l 912 923 middlewar root 53u unix 0x00000000cf15d90b 0t0 3711 /var/run/middleware/middlewared.sock type=STREAM (LISTEN)
asyncio_l 912 923 middlewar root 58u unix 0x0000000038a952f7 0t0 165223 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 923 middlewar root 59u unix 0x00000000aee7d3f8 0t0 165987 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 923 middlewar root 63u unix 0x00000000b9ceed85 0t0 166336 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 923 middlewar root 64u unix 0x0000000019ce13ab 0t0 165442 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 923 middlewar root 65u unix 0x00000000faf90525 0t0 165028 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 924 middlewar root 20u unix 0x00000000ef04bc7f 0t0 5342 /var/run/middleware/middlewared-internal.sock type=STREAM (LISTEN)
asyncio_l 912 924 middlewar root 53u unix 0x00000000cf15d90b 0t0 3711 /var/run/middleware/middlewared.sock type=STREAM (LISTEN)
asyncio_l 912 924 middlewar root 58u unix 0x0000000038a952f7 0t0 165223 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 924 middlewar root 59u unix 0x00000000aee7d3f8 0t0 165987 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 924 middlewar root 63u unix 0x00000000b9ceed85 0t0 166336 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 924 middlewar root 64u unix 0x0000000019ce13ab 0t0 165442 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 924 middlewar root 65u unix 0x00000000faf90525 0t0 165028 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 925 middlewar root 20u unix 0x00000000ef04bc7f 0t0 5342 /var/run/middleware/middlewared-internal.sock type=STREAM (LISTEN)
asyncio_l 912 925 middlewar root 53u unix 0x00000000cf15d90b 0t0 3711 /var/run/middleware/middlewared.sock type=STREAM (LISTEN)
asyncio_l 912 925 middlewar root 58u unix 0x0000000038a952f7 0t0 165223 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 925 middlewar root 59u unix 0x00000000aee7d3f8 0t0 165987 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 925 middlewar root 63u unix 0x00000000b9ceed85 0t0 166336 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 925 middlewar root 64u unix 0x0000000019ce13ab 0t0 165442 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 925 middlewar root 65u unix 0x00000000faf90525 0t0 165028 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 926 middlewar root 20u unix 0x00000000ef04bc7f 0t0 5342 /var/run/middleware/middlewared-internal.sock type=STREAM (LISTEN)
asyncio_l 912 926 middlewar root 53u unix 0x00000000cf15d90b 0t0 3711 /var/run/middleware/middlewared.sock type=STREAM (LISTEN)
asyncio_l 912 926 middlewar root 58u unix 0x0000000038a952f7 0t0 165223 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 926 middlewar root 59u unix 0x00000000aee7d3f8 0t0 165987 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 926 middlewar root 63u unix 0x00000000b9ceed85 0t0 166336 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 926 middlewar root 64u unix 0x0000000019ce13ab 0t0 165442 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 926 middlewar root 65u unix 0x00000000faf90525 0t0 165028 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 927 middlewar root 20u unix 0x00000000ef04bc7f 0t0 5342 /var/run/middleware/middlewared-internal.sock type=STREAM (LISTEN)
asyncio_l 912 927 middlewar root 53u unix 0x00000000cf15d90b 0t0 3711 /var/run/middleware/middlewared.sock type=STREAM (LISTEN)
asyncio_l 912 927 middlewar root 58u unix 0x0000000038a952f7 0t0 165223 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 927 middlewar root 59u unix 0x00000000aee7d3f8 0t0 165987 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 927 middlewar root 63u unix 0x00000000b9ceed85 0t0 166336 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 927 middlewar root 64u unix 0x0000000019ce13ab 0t0 165442 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 927 middlewar root 65u unix 0x00000000faf90525 0t0 165028 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 928 middlewar root 20u unix 0x00000000ef04bc7f 0t0 5342 /var/run/middleware/middlewared-internal.sock type=STREAM (LISTEN)
asyncio_l 912 928 middlewar root 53u unix 0x00000000cf15d90b 0t0 3711 /var/run/middleware/middlewared.sock type=STREAM (LISTEN)
asyncio_l 912 928 middlewar root 58u unix 0x0000000038a952f7 0t0 165223 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 928 middlewar root 59u unix 0x00000000aee7d3f8 0t0 165987 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 928 middlewar root 63u unix 0x00000000b9ceed85 0t0 166336 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 928 middlewar root 64u unix 0x0000000019ce13ab 0t0 165442 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 928 middlewar root 65u unix 0x00000000faf90525 0t0 165028 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 930 middlewar root 20u unix 0x00000000ef04bc7f 0t0 5342 /var/run/middleware/middlewared-internal.sock type=STREAM (LISTEN)
asyncio_l 912 930 middlewar root 53u unix 0x00000000cf15d90b 0t0 3711 /var/run/middleware/middlewared.sock type=STREAM (LISTEN)
asyncio_l 912 930 middlewar root 58u unix 0x0000000038a952f7 0t0 165223 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 930 middlewar root 59u unix 0x00000000aee7d3f8 0t0 165987 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 930 middlewar root 63u unix 0x00000000b9ceed85 0t0 166336 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 930 middlewar root 64u unix 0x0000000019ce13ab 0t0 165442 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 930 middlewar root 65u unix 0x00000000faf90525 0t0 165028 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 935 loop_moni root 20u unix 0x00000000ef04bc7f 0t0 5342 /var/run/middleware/middlewared-internal.sock type=STREAM (LISTEN)
asyncio_l 912 935 loop_moni root 53u unix 0x00000000cf15d90b 0t0 3711 /var/run/middleware/middlewared.sock type=STREAM (LISTEN)
asyncio_l 912 935 loop_moni root 58u unix 0x0000000038a952f7 0t0 165223 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 935 loop_moni root 59u unix 0x00000000aee7d3f8 0t0 165987 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 935 loop_moni root 63u unix 0x00000000b9ceed85 0t0 166336 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 935 loop_moni root 64u unix 0x0000000019ce13ab 0t0 165442 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 935 loop_moni root 65u unix 0x00000000faf90525 0t0 165028 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 936 IoThread root 20u unix 0x00000000ef04bc7f 0t0 5342 /var/run/middleware/middlewared-internal.sock type=STREAM (LISTEN)
asyncio_l 912 936 IoThread root 53u unix 0x00000000cf15d90b 0t0 3711 /var/run/middleware/middlewared.sock type=STREAM (LISTEN)
asyncio_l 912 936 IoThread root 58u unix 0x0000000038a952f7 0t0 165223 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 936 IoThread root 59u unix 0x00000000aee7d3f8 0t0 165987 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 936 IoThread root 63u unix 0x00000000b9ceed85 0t0 166336 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 936 IoThread root 64u unix 0x0000000019ce13ab 0t0 165442 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 936 IoThread root 65u unix 0x00000000faf90525 0t0 165028 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 937 asyncio_l root 20u unix 0x00000000ef04bc7f 0t0 5342 /var/run/middleware/middlewared-internal.sock type=STREAM (LISTEN)
asyncio_l 912 937 asyncio_l root 53u unix 0x00000000cf15d90b 0t0 3711 /var/run/middleware/middlewared.sock type=STREAM (LISTEN)
asyncio_l 912 937 asyncio_l root 58u unix 0x0000000038a952f7 0t0 165223 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 937 asyncio_l root 59u unix 0x00000000aee7d3f8 0t0 165987 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 937 asyncio_l root 63u unix 0x00000000b9ceed85 0t0 166336 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 937 asyncio_l root 64u unix 0x0000000019ce13ab 0t0 165442 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 937 asyncio_l root 65u unix 0x00000000faf90525 0t0 165028 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 938 asyncio_l root 20u unix 0x00000000ef04bc7f 0t0 5342 /var/run/middleware/middlewared-internal.sock type=STREAM (LISTEN)
asyncio_l 912 938 asyncio_l root 53u unix 0x00000000cf15d90b 0t0 3711 /var/run/middleware/middlewared.sock type=STREAM (LISTEN)
asyncio_l 912 938 asyncio_l root 58u unix 0x0000000038a952f7 0t0 165223 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 938 asyncio_l root 59u unix 0x00000000aee7d3f8 0t0 165987 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 938 asyncio_l root 63u unix 0x00000000b9ceed85 0t0 166336 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 938 asyncio_l root 64u unix 0x0000000019ce13ab 0t0 165442 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 938 asyncio_l root 65u unix 0x00000000faf90525 0t0 165028 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 942 IoThread root 20u unix 0x00000000ef04bc7f 0t0 5342 /var/run/middleware/middlewared-internal.sock type=STREAM (LISTEN)
asyncio_l 912 942 IoThread root 53u unix 0x00000000cf15d90b 0t0 3711 /var/run/middleware/middlewared.sock type=STREAM (LISTEN)
asyncio_l 912 942 IoThread root 58u unix 0x0000000038a952f7 0t0 165223 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 942 IoThread root 59u unix 0x00000000aee7d3f8 0t0 165987 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 942 IoThread root 63u unix 0x00000000b9ceed85 0t0 166336 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 942 IoThread root 64u unix 0x0000000019ce13ab 0t0 165442 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 942 IoThread root 65u unix 0x00000000faf90525 0t0 165028 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 945 IoThread root 20u unix 0x00000000ef04bc7f 0t0 5342 /var/run/middleware/middlewared-internal.sock type=STREAM (LISTEN)
asyncio_l 912 945 IoThread root 53u unix 0x00000000cf15d90b 0t0 3711 /var/run/middleware/middlewared.sock type=STREAM (LISTEN)
asyncio_l 912 945 IoThread root 58u unix 0x0000000038a952f7 0t0 165223 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 945 IoThread root 59u unix 0x00000000aee7d3f8 0t0 165987 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 945 IoThread root 63u unix 0x00000000b9ceed85 0t0 166336 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 945 IoThread root 64u unix 0x0000000019ce13ab 0t0 165442 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 945 IoThread root 65u unix 0x00000000faf90525 0t0 165028 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 946 asyncio_l root 20u unix 0x00000000ef04bc7f 0t0 5342 /var/run/middleware/middlewared-internal.sock type=STREAM (LISTEN)
asyncio_l 912 946 asyncio_l root 53u unix 0x00000000cf15d90b 0t0 3711 /var/run/middleware/middlewared.sock type=STREAM (LISTEN)
asyncio_l 912 946 asyncio_l root 58u unix 0x0000000038a952f7 0t0 165223 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 946 asyncio_l root 59u unix 0x00000000aee7d3f8 0t0 165987 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 946 asyncio_l root 63u unix 0x00000000b9ceed85 0t0 166336 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 946 asyncio_l root 64u unix 0x0000000019ce13ab 0t0 165442 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 946 asyncio_l root 65u unix 0x00000000faf90525 0t0 165028 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 961 IoThread root 20u unix 0x00000000ef04bc7f 0t0 5342 /var/run/middleware/middlewared-internal.sock type=STREAM (LISTEN)
asyncio_l 912 961 IoThread root 53u unix 0x00000000cf15d90b 0t0 3711 /var/run/middleware/middlewared.sock type=STREAM (LISTEN)
asyncio_l 912 961 IoThread root 58u unix 0x0000000038a952f7 0t0 165223 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 961 IoThread root 59u unix 0x00000000aee7d3f8 0t0 165987 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 961 IoThread root 63u unix 0x00000000b9ceed85 0t0 166336 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 961 IoThread root 64u unix 0x0000000019ce13ab 0t0 165442 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 961 IoThread root 65u unix 0x00000000faf90525 0t0 165028 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 977 retrieve_ root 20u unix 0x00000000ef04bc7f 0t0 5342 /var/run/middleware/middlewared-internal.sock type=STREAM (LISTEN)
asyncio_l 912 977 retrieve_ root 53u unix 0x00000000cf15d90b 0t0 3711 /var/run/middleware/middlewared.sock type=STREAM (LISTEN)
asyncio_l 912 977 retrieve_ root 58u unix 0x0000000038a952f7 0t0 165223 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 977 retrieve_ root 59u unix 0x00000000aee7d3f8 0t0 165987 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 977 retrieve_ root 63u unix 0x00000000b9ceed85 0t0 166336 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 977 retrieve_ root 64u unix 0x0000000019ce13ab 0t0 165442 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 977 retrieve_ root 65u unix 0x00000000faf90525 0t0 165028 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 980 IoThread root 20u unix 0x00000000ef04bc7f 0t0 5342 /var/run/middleware/middlewared-internal.sock type=STREAM (LISTEN)
asyncio_l 912 980 IoThread root 53u unix 0x00000000cf15d90b 0t0 3711 /var/run/middleware/middlewared.sock type=STREAM (LISTEN)
asyncio_l 912 980 IoThread root 58u unix 0x0000000038a952f7 0t0 165223 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 980 IoThread root 59u unix 0x00000000aee7d3f8 0t0 165987 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 980 IoThread root 63u unix 0x00000000b9ceed85 0t0 166336 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 980 IoThread root 64u unix 0x0000000019ce13ab 0t0 165442 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 980 IoThread root 65u unix 0x00000000faf90525 0t0 165028 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 981 asyncio_l root 20u unix 0x00000000ef04bc7f 0t0 5342 /var/run/middleware/middlewared-internal.sock type=STREAM (LISTEN)
asyncio_l 912 981 asyncio_l root 53u unix 0x00000000cf15d90b 0t0 3711 /var/run/middleware/middlewared.sock type=STREAM (LISTEN)
asyncio_l 912 981 asyncio_l root 58u unix 0x0000000038a952f7 0t0 165223 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 981 asyncio_l root 59u unix 0x00000000aee7d3f8 0t0 165987 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 981 asyncio_l root 63u unix 0x00000000b9ceed85 0t0 166336 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 981 asyncio_l root 64u unix 0x0000000019ce13ab 0t0 165442 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 981 asyncio_l root 65u unix 0x00000000faf90525 0t0 165028 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 1128 IoThread root 20u unix 0x00000000ef04bc7f 0t0 5342 /var/run/middleware/middlewared-internal.sock type=STREAM (LISTEN)
asyncio_l 912 1128 IoThread root 53u unix 0x00000000cf15d90b 0t0 3711 /var/run/middleware/middlewared.sock type=STREAM (LISTEN)
asyncio_l 912 1128 IoThread root 58u unix 0x0000000038a952f7 0t0 165223 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 1128 IoThread root 59u unix 0x00000000aee7d3f8 0t0 165987 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 1128 IoThread root 63u unix 0x00000000b9ceed85 0t0 166336 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 1128 IoThread root 64u unix 0x0000000019ce13ab 0t0 165442 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 1128 IoThread root 65u unix 0x00000000faf90525 0t0 165028 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 1144 IoThread root 20u unix 0x00000000ef04bc7f 0t0 5342 /var/run/middleware/middlewared-internal.sock type=STREAM (LISTEN)
asyncio_l 912 1144 IoThread root 53u unix 0x00000000cf15d90b 0t0 3711 /var/run/middleware/middlewared.sock type=STREAM (LISTEN)
asyncio_l 912 1144 IoThread root 58u unix 0x0000000038a952f7 0t0 165223 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 1144 IoThread root 59u unix 0x00000000aee7d3f8 0t0 165987 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 1144 IoThread root 63u unix 0x00000000b9ceed85 0t0 166336 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 1144 IoThread root 64u unix 0x0000000019ce13ab 0t0 165442 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 1144 IoThread root 65u unix 0x00000000faf90525 0t0 165028 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 1259 IoThread root 20u unix 0x00000000ef04bc7f 0t0 5342 /var/run/middleware/middlewared-internal.sock type=STREAM (LISTEN)
asyncio_l 912 1259 IoThread root 53u unix 0x00000000cf15d90b 0t0 3711 /var/run/middleware/middlewared.sock type=STREAM (LISTEN)
asyncio_l 912 1259 IoThread root 58u unix 0x0000000038a952f7 0t0 165223 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 1259 IoThread root 59u unix 0x00000000aee7d3f8 0t0 165987 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 1259 IoThread root 63u unix 0x00000000b9ceed85 0t0 166336 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 1259 IoThread root 64u unix 0x0000000019ce13ab 0t0 165442 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 1259 IoThread root 65u unix 0x00000000faf90525 0t0 165028 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 2732 IoThread root 20u unix 0x00000000ef04bc7f 0t0 5342 /var/run/middleware/middlewared-internal.sock type=STREAM (LISTEN)
asyncio_l 912 2732 IoThread root 53u unix 0x00000000cf15d90b 0t0 3711 /var/run/middleware/middlewared.sock type=STREAM (LISTEN)
asyncio_l 912 2732 IoThread root 58u unix 0x0000000038a952f7 0t0 165223 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 2732 IoThread root 59u unix 0x00000000aee7d3f8 0t0 165987 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 2732 IoThread root 63u unix 0x00000000b9ceed85 0t0 166336 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 2732 IoThread root 64u unix 0x0000000019ce13ab 0t0 165442 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 2732 IoThread root 65u unix 0x00000000faf90525 0t0 165028 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 2734 IoThread root 20u unix 0x00000000ef04bc7f 0t0 5342 /var/run/middleware/middlewared-internal.sock type=STREAM (LISTEN)
asyncio_l 912 2734 IoThread root 53u unix 0x00000000cf15d90b 0t0 3711 /var/run/middleware/middlewared.sock type=STREAM (LISTEN)
asyncio_l 912 2734 IoThread root 58u unix 0x0000000038a952f7 0t0 165223 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 2734 IoThread root 59u unix 0x00000000aee7d3f8 0t0 165987 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 2734 IoThread root 63u unix 0x00000000b9ceed85 0t0 166336 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 2734 IoThread root 64u unix 0x0000000019ce13ab 0t0 165442 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 2734 IoThread root 65u unix 0x00000000faf90525 0t0 165028 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 2736 IoThread root 20u unix 0x00000000ef04bc7f 0t0 5342 /var/run/middleware/middlewared-internal.sock type=STREAM (LISTEN)
asyncio_l 912 2736 IoThread root 53u unix 0x00000000cf15d90b 0t0 3711 /var/run/middleware/middlewared.sock type=STREAM (LISTEN)
asyncio_l 912 2736 IoThread root 58u unix 0x0000000038a952f7 0t0 165223 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 2736 IoThread root 59u unix 0x00000000aee7d3f8 0t0 165987 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 2736 IoThread root 63u unix 0x00000000b9ceed85 0t0 166336 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 2736 IoThread root 64u unix 0x0000000019ce13ab 0t0 165442 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 2736 IoThread root 65u unix 0x00000000faf90525 0t0 165028 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 2737 IoThread root 20u unix 0x00000000ef04bc7f 0t0 5342 /var/run/middleware/middlewared-internal.sock type=STREAM (LISTEN)
asyncio_l 912 2737 IoThread root 53u unix 0x00000000cf15d90b 0t0 3711 /var/run/middleware/middlewared.sock type=STREAM (LISTEN)
asyncio_l 912 2737 IoThread root 58u unix 0x0000000038a952f7 0t0 165223 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 2737 IoThread root 59u unix 0x00000000aee7d3f8 0t0 165987 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 2737 IoThread root 63u unix 0x00000000b9ceed85 0t0 166336 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 2737 IoThread root 64u unix 0x0000000019ce13ab 0t0 165442 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 2737 IoThread root 65u unix 0x00000000faf90525 0t0 165028 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 2739 IoThread root 20u unix 0x00000000ef04bc7f 0t0 5342 /var/run/middleware/middlewared-internal.sock type=STREAM (LISTEN)
asyncio_l 912 2739 IoThread root 53u unix 0x00000000cf15d90b 0t0 3711 /var/run/middleware/middlewared.sock type=STREAM (LISTEN)
asyncio_l 912 2739 IoThread root 58u unix 0x0000000038a952f7 0t0 165223 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 2739 IoThread root 59u unix 0x00000000aee7d3f8 0t0 165987 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 2739 IoThread root 63u unix 0x00000000b9ceed85 0t0 166336 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 2739 IoThread root 64u unix 0x0000000019ce13ab 0t0 165442 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 2739 IoThread root 65u unix 0x00000000faf90525 0t0 165028 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 2740 IoThread root 20u unix 0x00000000ef04bc7f 0t0 5342 /var/run/middleware/middlewared-internal.sock type=STREAM (LISTEN)
asyncio_l 912 2740 IoThread root 53u unix 0x00000000cf15d90b 0t0 3711 /var/run/middleware/middlewared.sock type=STREAM (LISTEN)
asyncio_l 912 2740 IoThread root 58u unix 0x0000000038a952f7 0t0 165223 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 2740 IoThread root 59u unix 0x00000000aee7d3f8 0t0 165987 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 2740 IoThread root 63u unix 0x00000000b9ceed85 0t0 166336 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 2740 IoThread root 64u unix 0x0000000019ce13ab 0t0 165442 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 2740 IoThread root 65u unix 0x00000000faf90525 0t0 165028 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 2741 IoThread root 20u unix 0x00000000ef04bc7f 0t0 5342 /var/run/middleware/middlewared-internal.sock type=STREAM (LISTEN)
asyncio_l 912 2741 IoThread root 53u unix 0x00000000cf15d90b 0t0 3711 /var/run/middleware/middlewared.sock type=STREAM (LISTEN)
asyncio_l 912 2741 IoThread root 58u unix 0x0000000038a952f7 0t0 165223 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 2741 IoThread root 59u unix 0x00000000aee7d3f8 0t0 165987 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 2741 IoThread root 63u unix 0x00000000b9ceed85 0t0 166336 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 2741 IoThread root 64u unix 0x0000000019ce13ab 0t0 165442 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 2741 IoThread root 65u unix 0x00000000faf90525 0t0 165028 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 2743 IoThread root 20u unix 0x00000000ef04bc7f 0t0 5342 /var/run/middleware/middlewared-internal.sock type=STREAM (LISTEN)
asyncio_l 912 2743 IoThread root 53u unix 0x00000000cf15d90b 0t0 3711 /var/run/middleware/middlewared.sock type=STREAM (LISTEN)
asyncio_l 912 2743 IoThread root 58u unix 0x0000000038a952f7 0t0 165223 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 2743 IoThread root 59u unix 0x00000000aee7d3f8 0t0 165987 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 2743 IoThread root 63u unix 0x00000000b9ceed85 0t0 166336 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 2743 IoThread root 64u unix 0x0000000019ce13ab 0t0 165442 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 2743 IoThread root 65u unix 0x00000000faf90525 0t0 165028 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 2747 IoThread root 20u unix 0x00000000ef04bc7f 0t0 5342 /var/run/middleware/middlewared-internal.sock type=STREAM (LISTEN)
asyncio_l 912 2747 IoThread root 53u unix 0x00000000cf15d90b 0t0 3711 /var/run/middleware/middlewared.sock type=STREAM (LISTEN)
asyncio_l 912 2747 IoThread root 58u unix 0x0000000038a952f7 0t0 165223 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 2747 IoThread root 59u unix 0x00000000aee7d3f8 0t0 165987 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 2747 IoThread root 63u unix 0x00000000b9ceed85 0t0 166336 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 2747 IoThread root 64u unix 0x0000000019ce13ab 0t0 165442 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 2747 IoThread root 65u unix 0x00000000faf90525 0t0 165028 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 2748 IoThread root 20u unix 0x00000000ef04bc7f 0t0 5342 /var/run/middleware/middlewared-internal.sock type=STREAM (LISTEN)
asyncio_l 912 2748 IoThread root 53u unix 0x00000000cf15d90b 0t0 3711 /var/run/middleware/middlewared.sock type=STREAM (LISTEN)
asyncio_l 912 2748 IoThread root 58u unix 0x0000000038a952f7 0t0 165223 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 2748 IoThread root 59u unix 0x00000000aee7d3f8 0t0 165987 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 2748 IoThread root 63u unix 0x00000000b9ceed85 0t0 166336 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 2748 IoThread root 64u unix 0x0000000019ce13ab 0t0 165442 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 2748 IoThread root 65u unix 0x00000000faf90525 0t0 165028 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 2750 IoThread root 20u unix 0x00000000ef04bc7f 0t0 5342 /var/run/middleware/middlewared-internal.sock type=STREAM (LISTEN)
asyncio_l 912 2750 IoThread root 53u unix 0x00000000cf15d90b 0t0 3711 /var/run/middleware/middlewared.sock type=STREAM (LISTEN)
asyncio_l 912 2750 IoThread root 58u unix 0x0000000038a952f7 0t0 165223 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 2750 IoThread root 59u unix 0x00000000aee7d3f8 0t0 165987 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 2750 IoThread root 63u unix 0x00000000b9ceed85 0t0 166336 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 2750 IoThread root 64u unix 0x0000000019ce13ab 0t0 165442 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 2750 IoThread root 65u unix 0x00000000faf90525 0t0 165028 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 2758 IoThread root 20u unix 0x00000000ef04bc7f 0t0 5342 /var/run/middleware/middlewared-internal.sock type=STREAM (LISTEN)
asyncio_l 912 2758 IoThread root 53u unix 0x00000000cf15d90b 0t0 3711 /var/run/middleware/middlewared.sock type=STREAM (LISTEN)
asyncio_l 912 2758 IoThread root 58u unix 0x0000000038a952f7 0t0 165223 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 2758 IoThread root 59u unix 0x00000000aee7d3f8 0t0 165987 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 2758 IoThread root 63u unix 0x00000000b9ceed85 0t0 166336 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 2758 IoThread root 64u unix 0x0000000019ce13ab 0t0 165442 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 2758 IoThread root 65u unix 0x00000000faf90525 0t0 165028 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 2759 IoThread root 20u unix 0x00000000ef04bc7f 0t0 5342 /var/run/middleware/middlewared-internal.sock type=STREAM (LISTEN)
asyncio_l 912 2759 IoThread root 53u unix 0x00000000cf15d90b 0t0 3711 /var/run/middleware/middlewared.sock type=STREAM (LISTEN)
asyncio_l 912 2759 IoThread root 58u unix 0x0000000038a952f7 0t0 165223 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 2759 IoThread root 59u unix 0x00000000aee7d3f8 0t0 165987 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 2759 IoThread root 63u unix 0x00000000b9ceed85 0t0 166336 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 2759 IoThread root 64u unix 0x0000000019ce13ab 0t0 165442 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 2759 IoThread root 65u unix 0x00000000faf90525 0t0 165028 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 4540 asyncio_l root 20u unix 0x00000000ef04bc7f 0t0 5342 /var/run/middleware/middlewared-internal.sock type=STREAM (LISTEN)
asyncio_l 912 4540 asyncio_l root 53u unix 0x00000000cf15d90b 0t0 3711 /var/run/middleware/middlewared.sock type=STREAM (LISTEN)
asyncio_l 912 4540 asyncio_l root 58u unix 0x0000000038a952f7 0t0 165223 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 4540 asyncio_l root 59u unix 0x00000000aee7d3f8 0t0 165987 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 4540 asyncio_l root 63u unix 0x00000000b9ceed85 0t0 166336 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 4540 asyncio_l root 64u unix 0x0000000019ce13ab 0t0 165442 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 4540 asyncio_l root 65u unix 0x00000000faf90525 0t0 165028 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 13157 asyncio_l root 20u unix 0x00000000ef04bc7f 0t0 5342 /var/run/middleware/middlewared-internal.sock type=STREAM (LISTEN)
asyncio_l 912 13157 asyncio_l root 53u unix 0x00000000cf15d90b 0t0 3711 /var/run/middleware/middlewared.sock type=STREAM (LISTEN)
asyncio_l 912 13157 asyncio_l root 58u unix 0x0000000038a952f7 0t0 165223 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 13157 asyncio_l root 59u unix 0x00000000aee7d3f8 0t0 165987 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 13157 asyncio_l root 63u unix 0x00000000b9ceed85 0t0 166336 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 13157 asyncio_l root 64u unix 0x0000000019ce13ab 0t0 165442 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
asyncio_l 912 13157 asyncio_l root 65u unix 0x00000000faf90525 0t0 165028 /var/run/middleware/middlewared-internal.sock type=STREAM (CONNECTED)
$ sudo midclt call core.threads_stacks | jq
{
"139948279723712": [
" File \"/usr/lib/python3.11/threading.py\", line 1002, in _bootstrap\n self._bootstrap_inner()\n",
" File \"/usr/lib/python3.11/threading.py\", line 1045, in _bootstrap_inner\n self.run()\n",
" File \"/usr/lib/python3.11/threading.py\", line 982, in run\n self._target(*self._args, **self._kwargs)\n",
" File \"/usr/lib/python3.11/concurrent/futures/thread.py\", line 81, in _worker\n work_item = work_queue.get(block=True)\n"
],
"139948367779520": [
" File \"/usr/lib/python3.11/threading.py\", line 1002, in _bootstrap\n self._bootstrap_inner()\n",
" File \"/usr/lib/python3.11/threading.py\", line 1045, in _bootstrap_inner\n self.run()\n",
" File \"/usr/lib/python3.11/threading.py\", line 982, in run\n self._target(*self._args, **self._kwargs)\n",
" File \"/usr/lib/python3.11/concurrent/futures/thread.py\", line 81, in _worker\n work_item = work_queue.get(block=True)\n"
],
"139948380366528": [
" File \"/usr/lib/python3.11/threading.py\", line 1002, in _bootstrap\n self._bootstrap_inner()\n",
" File \"/usr/lib/python3.11/threading.py\", line 1045, in _bootstrap_inner\n self.run()\n",
" File \"/usr/lib/python3.11/threading.py\", line 982, in run\n self._target(*self._args, **self._kwargs)\n",
" File \"/usr/lib/python3.11/concurrent/futures/thread.py\", line 81, in _worker\n work_item = work_queue.get(block=True)\n"
],
"139948388759232": [
" File \"/usr/lib/python3.11/threading.py\", line 1002, in _bootstrap\n self._bootstrap_inner()\n",
" File \"/usr/lib/python3.11/threading.py\", line 1045, in _bootstrap_inner\n self.run()\n",
" File \"/usr/lib/python3.11/threading.py\", line 982, in run\n self._target(*self._args, **self._kwargs)\n",
" File \"/usr/lib/python3.11/concurrent/futures/thread.py\", line 81, in _worker\n work_item = work_queue.get(block=True)\n"
],
"139948397151936": [
" File \"/usr/lib/python3.11/threading.py\", line 1002, in _bootstrap\n self._bootstrap_inner()\n",
" File \"/usr/lib/python3.11/threading.py\", line 1045, in _bootstrap_inner\n self.run()\n",
" File \"/usr/lib/python3.11/threading.py\", line 982, in run\n self._target(*self._args, **self._kwargs)\n",
" File \"/usr/lib/python3.11/concurrent/futures/thread.py\", line 81, in _worker\n work_item = work_queue.get(block=True)\n"
],
"139948405544640": [
" File \"/usr/lib/python3.11/threading.py\", line 1002, in _bootstrap\n self._bootstrap_inner()\n",
" File \"/usr/lib/python3.11/threading.py\", line 1045, in _bootstrap_inner\n self.run()\n",
" File \"/usr/lib/python3.11/threading.py\", line 982, in run\n self._target(*self._args, **self._kwargs)\n",
" File \"/usr/lib/python3.11/concurrent/futures/thread.py\", line 81, in _worker\n work_item = work_queue.get(block=True)\n"
],
"139948413937344": [
" File \"/usr/lib/python3.11/threading.py\", line 1002, in _bootstrap\n self._bootstrap_inner()\n",
" File \"/usr/lib/python3.11/threading.py\", line 1045, in _bootstrap_inner\n self.run()\n",
" File \"/usr/lib/python3.11/threading.py\", line 982, in run\n self._target(*self._args, **self._kwargs)\n",
" File \"/usr/lib/python3.11/concurrent/futures/thread.py\", line 81, in _worker\n work_item = work_queue.get(block=True)\n"
],
"139948422330048": [
" File \"/usr/lib/python3.11/threading.py\", line 1002, in _bootstrap\n self._bootstrap_inner()\n",
" File \"/usr/lib/python3.11/threading.py\", line 1045, in _bootstrap_inner\n self.run()\n",
" File \"/usr/lib/python3.11/threading.py\", line 982, in run\n self._target(*self._args, **self._kwargs)\n",
" File \"/usr/lib/python3.11/concurrent/futures/thread.py\", line 81, in _worker\n work_item = work_queue.get(block=True)\n"
],
"139948902553280": [
" File \"/usr/lib/python3.11/threading.py\", line 1002, in _bootstrap\n self._bootstrap_inner()\n",
" File \"/usr/lib/python3.11/threading.py\", line 1045, in _bootstrap_inner\n self.run()\n",
" File \"/usr/lib/python3.11/threading.py\", line 982, in run\n self._target(*self._args, **self._kwargs)\n",
" File \"/usr/lib/python3.11/concurrent/futures/thread.py\", line 81, in _worker\n work_item = work_queue.get(block=True)\n"
],
"139948910945984": [
" File \"/usr/lib/python3.11/threading.py\", line 1002, in _bootstrap\n self._bootstrap_inner()\n",
" File \"/usr/lib/python3.11/threading.py\", line 1045, in _bootstrap_inner\n self.run()\n",
" File \"/usr/lib/python3.11/threading.py\", line 982, in run\n self._target(*self._args, **self._kwargs)\n",
" File \"/usr/lib/python3.11/concurrent/futures/thread.py\", line 83, in _worker\n work_item.run()\n",
" File \"/usr/lib/python3.11/concurrent/futures/thread.py\", line 58, in run\n result = self.fn(*self.args, **self.kwargs)\n",
" File \"/usr/lib/python3/dist-packages/middlewared/service/core_service.py\", line 715, in threads_stacks\n return get_threads_stacks()\n",
" File \"/usr/lib/python3/dist-packages/middlewared/utils/debug.py\", line 61, in get_threads_stacks\n return {\n"
],
"139948919338688": [
" File \"/usr/lib/python3.11/threading.py\", line 1002, in _bootstrap\n self._bootstrap_inner()\n",
" File \"/usr/lib/python3.11/threading.py\", line 1045, in _bootstrap_inner\n self.run()\n",
" File \"/usr/lib/python3.11/threading.py\", line 982, in run\n self._target(*self._args, **self._kwargs)\n",
" File \"/usr/lib/python3.11/concurrent/futures/thread.py\", line 83, in _worker\n work_item.run()\n",
" File \"/usr/lib/python3.11/concurrent/futures/thread.py\", line 58, in run\n result = self.fn(*self.args, **self.kwargs)\n",
" File \"/usr/lib/python3/dist-packages/middlewared/plugins/docker/events.py\", line 16, in setup\n self.process()\n",
" File \"/usr/lib/python3/dist-packages/middlewared/plugins/docker/events.py\", line 25, in process\n self.process_internal(docker_client)\n",
" File \"/usr/lib/python3/dist-packages/middlewared/plugins/docker/events.py\", line 28, in process_internal\n for container_event in client.events(\n",
" File \"/usr/lib/python3/dist-packages/docker/types/daemon.py\", line 29, in __next__\n return next(self._stream)\n",
" File \"/usr/lib/python3/dist-packages/docker/api/client.py\", line 360, in _stream_helper\n yield from json_stream(self._stream_helper(response, False))\n",
" File \"/usr/lib/python3/dist-packages/docker/utils/json_stream.py\", line 60, in split_buffer\n for data in stream_as_text(stream):\n",
" File \"/usr/lib/python3/dist-packages/docker/utils/json_stream.py\", line 16, in stream_as_text\n for data in stream:\n",
" File \"/usr/lib/python3/dist-packages/docker/api/client.py\", line 365, in _stream_helper\n data = reader.read(1)\n",
" File \"/usr/lib/python3/dist-packages/urllib3/response.py\", line 566, in read\n data = self._fp_read(amt) if not fp_closed else b\"\"\n",
" File \"/usr/lib/python3/dist-packages/urllib3/response.py\", line 532, in _fp_read\n return self._fp.read(amt) if amt is not None else self._fp.read()\n",
" File \"/usr/lib/python3.11/http/client.py\", line 467, in read\n return self._read_chunked(amt)\n",
" File \"/usr/lib/python3.11/http/client.py\", line 590, in _read_chunked\n chunk_left = self._get_chunk_left()\n",
" File \"/usr/lib/python3.11/http/client.py\", line 573, in _get_chunk_left\n chunk_left = self._read_next_chunk_size()\n",
" File \"/usr/lib/python3.11/http/client.py\", line 533, in _read_next_chunk_size\n line = self.fp.readline(_MAXLINE + 1)\n",
" File \"/usr/lib/python3.11/socket.py\", line 706, in readinto\n return self._sock.recv_into(b)\n"
],
"139948932974272": [
" File \"/usr/lib/python3.11/threading.py\", line 1002, in _bootstrap\n self._bootstrap_inner()\n",
" File \"/usr/lib/python3.11/threading.py\", line 1045, in _bootstrap_inner\n self.run()\n",
" File \"/usr/lib/python3.11/threading.py\", line 982, in run\n self._target(*self._args, **self._kwargs)\n",
" File \"/usr/lib/python3.11/concurrent/futures/thread.py\", line 81, in _worker\n work_item = work_queue.get(block=True)\n"
],
"139948941366976": [
" File \"/usr/lib/python3.11/threading.py\", line 1002, in _bootstrap\n self._bootstrap_inner()\n",
" File \"/usr/lib/python3.11/threading.py\", line 1045, in _bootstrap_inner\n self.run()\n",
" File \"/usr/lib/python3.11/threading.py\", line 982, in run\n self._target(*self._args, **self._kwargs)\n",
" File \"/usr/lib/python3.11/concurrent/futures/thread.py\", line 81, in _worker\n work_item = work_queue.get(block=True)\n"
],
"139948950808256": [
" File \"/usr/lib/python3.11/threading.py\", line 1002, in _bootstrap\n self._bootstrap_inner()\n",
" File \"/usr/lib/python3.11/threading.py\", line 1045, in _bootstrap_inner\n self.run()\n",
" File \"/usr/lib/python3.11/threading.py\", line 982, in run\n self._target(*self._args, **self._kwargs)\n",
" File \"/usr/lib/python3.11/concurrent/futures/thread.py\", line 81, in _worker\n work_item = work_queue.get(block=True)\n"
],
"139949496071872": [
" File \"/usr/lib/python3.11/threading.py\", line 1002, in _bootstrap\n self._bootstrap_inner()\n",
" File \"/usr/lib/python3.11/threading.py\", line 1045, in _bootstrap_inner\n self.run()\n",
" File \"/usr/lib/python3.11/threading.py\", line 982, in run\n self._target(*self._args, **self._kwargs)\n",
" File \"/usr/lib/python3.11/concurrent/futures/thread.py\", line 81, in _worker\n work_item = work_queue.get(block=True)\n"
],
"139948959200960": [
" File \"/usr/lib/python3.11/threading.py\", line 1002, in _bootstrap\n self._bootstrap_inner()\n",
" File \"/usr/lib/python3.11/threading.py\", line 1045, in _bootstrap_inner\n self.run()\n",
" File \"/usr/lib/python3.11/threading.py\", line 982, in run\n self._target(*self._args, **self._kwargs)\n",
" File \"/usr/lib/python3.11/concurrent/futures/thread.py\", line 81, in _worker\n work_item = work_queue.get(block=True)\n"
],
"139949443618496": [
" File \"/usr/lib/python3.11/threading.py\", line 1002, in _bootstrap\n self._bootstrap_inner()\n",
" File \"/usr/lib/python3.11/threading.py\", line 1045, in _bootstrap_inner\n self.run()\n",
" File \"/usr/lib/python3.11/threading.py\", line 982, in run\n self._target(*self._args, **self._kwargs)\n",
" File \"/usr/lib/python3.11/concurrent/futures/thread.py\", line 81, in _worker\n work_item = work_queue.get(block=True)\n"
],
"139949452011200": [
" File \"/usr/lib/python3.11/threading.py\", line 1002, in _bootstrap\n self._bootstrap_inner()\n",
" File \"/usr/lib/python3.11/threading.py\", line 1045, in _bootstrap_inner\n self.run()\n",
" File \"/usr/lib/python3.11/threading.py\", line 982, in run\n self._target(*self._args, **self._kwargs)\n",
" File \"/usr/lib/python3.11/concurrent/futures/thread.py\", line 81, in _worker\n work_item = work_queue.get(block=True)\n"
],
"139949461452480": [
" File \"/usr/lib/python3.11/threading.py\", line 1002, in _bootstrap\n self._bootstrap_inner()\n",
" File \"/usr/lib/python3.11/threading.py\", line 1045, in _bootstrap_inner\n self.run()\n",
" File \"/usr/lib/python3.11/threading.py\", line 982, in run\n self._target(*self._args, **self._kwargs)\n",
" File \"/usr/lib/python3/dist-packages/middlewared/plugins/device_/udev_events.py\", line 37, in udev_events\n for device in iter(monitor.poll, None):\n",
" File \"/usr/lib/python3/dist-packages/pyudev/monitor.py\", line 354, in poll\n if eintr_retry_call(poll.Poll.for_events((self, \"r\")).poll, timeout):\n",
" File \"/usr/lib/python3/dist-packages/pyudev/_util.py\", line 152, in eintr_retry_call\n return func(*args, **kwargs)\n",
" File \"/usr/lib/python3/dist-packages/pyudev/_os/poll.py\", line 91, in poll\n return list(self._parse_events(eintr_retry_call(self._notifier.poll, timeout)))\n",
" File \"/usr/lib/python3/dist-packages/pyudev/_util.py\", line 152, in eintr_retry_call\n return func(*args, **kwargs)\n"
],
"139949469845184": [
" File \"/usr/lib/python3.11/threading.py\", line 1002, in _bootstrap\n self._bootstrap_inner()\n",
" File \"/usr/lib/python3.11/threading.py\", line 1045, in _bootstrap_inner\n self.run()\n",
" File \"/usr/lib/python3.11/threading.py\", line 982, in run\n self._target(*self._args, **self._kwargs)\n",
" File \"/usr/lib/python3.11/concurrent/futures/thread.py\", line 81, in _worker\n work_item = work_queue.get(block=True)\n"
],
"139949478237888": [
" File \"/usr/lib/python3.11/threading.py\", line 1002, in _bootstrap\n self._bootstrap_inner()\n",
" File \"/usr/lib/python3.11/threading.py\", line 1045, in _bootstrap_inner\n self.run()\n",
" File \"/usr/lib/python3.11/threading.py\", line 982, in run\n self._target(*self._args, **self._kwargs)\n",
" File \"/usr/lib/python3/dist-packages/middlewared/plugins/zfs_/events.py\", line 30, in setup_zfs_events_process\n middleware.call_hook_sync('zfs.pool.events', data=parent_conn.recv())\n",
" File \"/usr/lib/python3.11/multiprocessing/connection.py\", line 250, in recv\n buf = self._recv_bytes()\n",
" File \"/usr/lib/python3.11/multiprocessing/connection.py\", line 430, in _recv_bytes\n buf = self._recv(4)\n",
" File \"/usr/lib/python3.11/multiprocessing/connection.py\", line 395, in _recv\n chunk = read(handle, remaining)\n"
],
"139949487679168": [
" File \"/usr/lib/python3.11/threading.py\", line 1002, in _bootstrap\n self._bootstrap_inner()\n",
" File \"/usr/lib/python3.11/threading.py\", line 1045, in _bootstrap_inner\n self.run()\n",
" File \"/usr/lib/python3.11/threading.py\", line 982, in run\n self._target(*self._args, **self._kwargs)\n",
" File \"/usr/lib/python3/dist-packages/middlewared/plugins/zettarepl.py\", line 884, in _observer_queue_reader\n message = self.observer_queue.get()\n",
" File \"/usr/lib/python3.11/multiprocessing/queues.py\", line 103, in get\n res = self._recv_bytes()\n",
" File \"/usr/lib/python3.11/multiprocessing/connection.py\", line 216, in recv_bytes\n buf = self._recv_bytes(maxlength)\n",
" File \"/usr/lib/python3.11/multiprocessing/connection.py\", line 430, in _recv_bytes\n buf = self._recv(4)\n",
" File \"/usr/lib/python3.11/multiprocessing/connection.py\", line 395, in _recv\n chunk = read(handle, remaining)\n"
],
"139949909194432": [
" File \"/usr/lib/python3.11/threading.py\", line 1002, in _bootstrap\n self._bootstrap_inner()\n",
" File \"/usr/lib/python3.11/threading.py\", line 1045, in _bootstrap_inner\n self.run()\n",
" File \"/usr/lib/python3.11/threading.py\", line 982, in run\n self._target(*self._args, **self._kwargs)\n",
" File \"/usr/lib/python3.11/multiprocessing/queues.py\", line 231, in _feed\n nwait()\n",
" File \"/usr/lib/python3.11/threading.py\", line 327, in wait\n waiter.acquire()\n"
],
"139949917587136": [
" File \"/usr/lib/python3.11/threading.py\", line 1002, in _bootstrap\n self._bootstrap_inner()\n",
" File \"/usr/lib/python3.11/threading.py\", line 1045, in _bootstrap_inner\n self.run()\n",
" File \"/usr/lib/python3.11/threading.py\", line 982, in run\n self._target(*self._args, **self._kwargs)\n",
" File \"/usr/lib/python3.11/concurrent/futures/thread.py\", line 81, in _worker\n work_item = work_queue.get(block=True)\n"
],
"139949929125568": [
" File \"/usr/lib/python3.11/threading.py\", line 1002, in _bootstrap\n self._bootstrap_inner()\n",
" File \"/usr/lib/python3.11/threading.py\", line 1045, in _bootstrap_inner\n self.run()\n",
" File \"/usr/lib/python3.11/threading.py\", line 982, in run\n self._target(*self._args, **self._kwargs)\n",
" File \"/usr/lib/python3.11/concurrent/futures/thread.py\", line 81, in _worker\n work_item = work_queue.get(block=True)\n"
],
"139949939615424": [
" File \"/usr/lib/python3.11/threading.py\", line 1002, in _bootstrap\n self._bootstrap_inner()\n",
" File \"/usr/lib/python3.11/threading.py\", line 1045, in _bootstrap_inner\n self.run()\n",
" File \"/usr/lib/python3.11/threading.py\", line 982, in run\n self._target(*self._args, **self._kwargs)\n",
" File \"/usr/lib/python3.11/concurrent/futures/thread.py\", line 81, in _worker\n work_item = work_queue.get(block=True)\n"
],
"139949948008128": [
" File \"/usr/lib/python3.11/threading.py\", line 1002, in _bootstrap\n self._bootstrap_inner()\n",
" File \"/usr/lib/python3.11/threading.py\", line 1045, in _bootstrap_inner\n self.run()\n",
" File \"/usr/lib/python3.11/concurrent/futures/process.py\", line 344, in run\n result_item, is_broken, cause = self.wait_result_broken_or_wakeup()\n",
" File \"/usr/lib/python3.11/concurrent/futures/process.py\", line 417, in wait_result_broken_or_wakeup\n ready = mp.connection.wait(readers + worker_sentinels)\n",
" File \"/usr/lib/python3.11/multiprocessing/connection.py\", line 948, in wait\n ready = selector.select(timeout)\n",
" File \"/usr/lib/python3.11/selectors.py\", line 415, in select\n fd_event_list = self._selector.poll(timeout)\n"
],
"139950043313856": [
" File \"/usr/lib/python3.11/threading.py\", line 1002, in _bootstrap\n self._bootstrap_inner()\n",
" File \"/usr/lib/python3.11/threading.py\", line 1045, in _bootstrap_inner\n self.run()\n",
" File \"/usr/lib/python3.11/threading.py\", line 982, in run\n self._target(*self._args, **self._kwargs)\n",
" File \"/usr/lib/python3.11/concurrent/futures/thread.py\", line 81, in _worker\n work_item = work_queue.get(block=True)\n"
],
"139950051706560": [
" File \"/usr/lib/python3.11/threading.py\", line 1002, in _bootstrap\n self._bootstrap_inner()\n",
" File \"/usr/lib/python3.11/threading.py\", line 1045, in _bootstrap_inner\n self.run()\n",
" File \"/usr/lib/python3.11/threading.py\", line 982, in run\n self._target(*self._args, **self._kwargs)\n",
" File \"/usr/lib/python3/dist-packages/middlewared/main.py\", line 1269, in _loop_monitor_thread\n time.sleep(2)\n"
],
"139950418810560": [
" File \"/usr/lib/python3.11/threading.py\", line 1002, in _bootstrap\n self._bootstrap_inner()\n",
" File \"/usr/lib/python3.11/threading.py\", line 1045, in _bootstrap_inner\n self.run()\n",
" File \"/usr/lib/python3.11/threading.py\", line 982, in run\n self._target(*self._args, **self._kwargs)\n",
" File \"/usr/lib/python3.11/logging/handlers.py\", line 1573, in _monitor\n record = self.dequeue(True)\n",
" File \"/usr/lib/python3.11/logging/handlers.py\", line 1522, in dequeue\n return self.queue.get(block)\n",
" File \"/usr/lib/python3.11/queue.py\", line 171, in get\n self.not_empty.wait()\n",
" File \"/usr/lib/python3.11/threading.py\", line 327, in wait\n waiter.acquire()\n"
],
"139950427203264": [
" File \"/usr/lib/python3.11/threading.py\", line 1002, in _bootstrap\n self._bootstrap_inner()\n",
" File \"/usr/lib/python3.11/threading.py\", line 1045, in _bootstrap_inner\n self.run()\n",
" File \"/usr/lib/python3.11/threading.py\", line 982, in run\n self._target(*self._args, **self._kwargs)\n",
" File \"/usr/lib/python3.11/logging/handlers.py\", line 1573, in _monitor\n record = self.dequeue(True)\n",
" File \"/usr/lib/python3.11/logging/handlers.py\", line 1522, in dequeue\n return self.queue.get(block)\n",
" File \"/usr/lib/python3.11/queue.py\", line 171, in get\n self.not_empty.wait()\n",
" File \"/usr/lib/python3.11/threading.py\", line 327, in wait\n waiter.acquire()\n"
],
"139950435595968": [
" File \"/usr/lib/python3.11/threading.py\", line 1002, in _bootstrap\n self._bootstrap_inner()\n",
" File \"/usr/lib/python3.11/threading.py\", line 1045, in _bootstrap_inner\n self.run()\n",
" File \"/usr/lib/python3.11/threading.py\", line 982, in run\n self._target(*self._args, **self._kwargs)\n",
" File \"/usr/lib/python3.11/logging/handlers.py\", line 1573, in _monitor\n record = self.dequeue(True)\n",
" File \"/usr/lib/python3.11/logging/handlers.py\", line 1522, in dequeue\n return self.queue.get(block)\n",
" File \"/usr/lib/python3.11/queue.py\", line 171, in get\n self.not_empty.wait()\n",
" File \"/usr/lib/python3.11/threading.py\", line 327, in wait\n waiter.acquire()\n"
],
"139950783428288": [
" File \"/usr/lib/python3.11/threading.py\", line 1002, in _bootstrap\n self._bootstrap_inner()\n",
" File \"/usr/lib/python3.11/threading.py\", line 1045, in _bootstrap_inner\n self.run()\n",
" File \"/usr/lib/python3.11/threading.py\", line 982, in run\n self._target(*self._args, **self._kwargs)\n",
" File \"/usr/lib/python3.11/logging/handlers.py\", line 1573, in _monitor\n record = self.dequeue(True)\n",
" File \"/usr/lib/python3.11/logging/handlers.py\", line 1522, in dequeue\n return self.queue.get(block)\n",
" File \"/usr/lib/python3.11/queue.py\", line 171, in get\n self.not_empty.wait()\n",
" File \"/usr/lib/python3.11/threading.py\", line 327, in wait\n waiter.acquire()\n"
],
"139950791837376": [
" File \"/usr/lib/python3.11/threading.py\", line 1002, in _bootstrap\n self._bootstrap_inner()\n",
" File \"/usr/lib/python3.11/threading.py\", line 1045, in _bootstrap_inner\n self.run()\n",
" File \"/usr/lib/python3.11/threading.py\", line 982, in run\n self._target(*self._args, **self._kwargs)\n",
" File \"/usr/lib/python3.11/logging/handlers.py\", line 1573, in _monitor\n record = self.dequeue(True)\n",
" File \"/usr/lib/python3.11/logging/handlers.py\", line 1522, in dequeue\n return self.queue.get(block)\n",
" File \"/usr/lib/python3.11/queue.py\", line 171, in get\n self.not_empty.wait()\n",
" File \"/usr/lib/python3.11/threading.py\", line 327, in wait\n waiter.acquire()\n"
],
"139950800246464": [
" File \"/usr/lib/python3.11/threading.py\", line 1002, in _bootstrap\n self._bootstrap_inner()\n",
" File \"/usr/lib/python3.11/threading.py\", line 1045, in _bootstrap_inner\n self.run()\n",
" File \"/usr/lib/python3.11/threading.py\", line 982, in run\n self._target(*self._args, **self._kwargs)\n",
" File \"/usr/lib/python3.11/logging/handlers.py\", line 1573, in _monitor\n record = self.dequeue(True)\n",
" File \"/usr/lib/python3.11/logging/handlers.py\", line 1522, in dequeue\n return self.queue.get(block)\n",
" File \"/usr/lib/python3.11/queue.py\", line 171, in get\n self.not_empty.wait()\n",
" File \"/usr/lib/python3.11/threading.py\", line 327, in wait\n waiter.acquire()\n"
],
"139950808655552": [
" File \"/usr/lib/python3.11/threading.py\", line 1002, in _bootstrap\n self._bootstrap_inner()\n",
" File \"/usr/lib/python3.11/threading.py\", line 1045, in _bootstrap_inner\n self.run()\n",
" File \"/usr/lib/python3.11/threading.py\", line 982, in run\n self._target(*self._args, **self._kwargs)\n",
" File \"/usr/lib/python3.11/logging/handlers.py\", line 1573, in _monitor\n record = self.dequeue(True)\n",
" File \"/usr/lib/python3.11/logging/handlers.py\", line 1522, in dequeue\n return self.queue.get(block)\n",
" File \"/usr/lib/python3.11/queue.py\", line 171, in get\n self.not_empty.wait()\n",
" File \"/usr/lib/python3.11/threading.py\", line 327, in wait\n waiter.acquire()\n"
],
"139950817048256": [
" File \"/usr/lib/python3.11/threading.py\", line 1002, in _bootstrap\n self._bootstrap_inner()\n",
" File \"/usr/lib/python3.11/threading.py\", line 1045, in _bootstrap_inner\n self.run()\n",
" File \"/usr/lib/python3.11/threading.py\", line 982, in run\n self._target(*self._args, **self._kwargs)\n",
" File \"/usr/lib/python3.11/logging/handlers.py\", line 1573, in _monitor\n record = self.dequeue(True)\n",
" File \"/usr/lib/python3.11/logging/handlers.py\", line 1522, in dequeue\n return self.queue.get(block)\n",
" File \"/usr/lib/python3.11/queue.py\", line 171, in get\n self.not_empty.wait()\n",
" File \"/usr/lib/python3.11/threading.py\", line 327, in wait\n waiter.acquire()\n"
],
"139950825440960": [
" File \"/usr/lib/python3.11/threading.py\", line 1002, in _bootstrap\n self._bootstrap_inner()\n",
" File \"/usr/lib/python3.11/threading.py\", line 1045, in _bootstrap_inner\n self.run()\n",
" File \"/usr/lib/python3.11/threading.py\", line 982, in run\n self._target(*self._args, **self._kwargs)\n",
" File \"/usr/lib/python3.11/logging/handlers.py\", line 1573, in _monitor\n record = self.dequeue(True)\n",
" File \"/usr/lib/python3.11/logging/handlers.py\", line 1522, in dequeue\n return self.queue.get(block)\n",
" File \"/usr/lib/python3.11/queue.py\", line 171, in get\n self.not_empty.wait()\n",
" File \"/usr/lib/python3.11/threading.py\", line 327, in wait\n waiter.acquire()\n"
],
"139951014817856": [
" File \"/usr/bin/middlewared\", line 33, in <module>\n sys.exit(load_entry_point('middlewared==0.0.0', 'console_scripts', 'middlewared')())\n",
" File \"/usr/lib/python3/dist-packages/middlewared/main.py\", line 1511, in main\n middleware.run()\n",
" File \"/usr/lib/python3/dist-packages/middlewared/main.py\", line 1322, in run\n self.loop.run_forever()\n",
" File \"/usr/lib/python3.11/asyncio/base_events.py\", line 608, in run_forever\n self._run_once()\n",
" File \"/usr/lib/python3.11/asyncio/base_events.py\", line 1898, in _run_once\n event_list = self._selector.select(timeout)\n",
" File \"/usr/lib/python3.11/selectors.py\", line 468, in select\n fd_event_list = self._selector.poll(timeout, max_ev)\n"
]
}