OSError: [Errno 5] Input/output error
[2024/04/23 20:10:55] (DEBUG) middlewared.__plugins_setup():990 - All plugins loaded
[2024/04/23 20:10:55] (DEBUG) middlewared.__initialize():1864 - Accepting connections
[2024/04/23 20:10:55] (DEBUG) middlewared._console_write():1096 - Failed to write to console
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1092, in _console_write
self.__console_io.flush()
OSError: [Errno 5] Input/output error
[2024/04/23 20:10:55] (INFO) DiskService.log_disk_info():75 - Found disks: {'nvme0n1': {'serial': 'S463NF0M524905Y', 'lunid': '0025385591b3e6d7'}, 'sdd': {'serial': '4ZG500YV', 'lunid': '5000cca2a6c245f8'}, 'sdc': {'serial': '4ZG4XHKV', 'lunid': '5000cca2a6c23c7a'}, 'sde': {'serial': '4BHLWKRH', 'lunid': '5000cca2b6d6adaf'}, 'sda': {'serial': 'ZDH7ZDX2', 'lunid': '5000c500c3647fba'}, 'sdg': {'serial': 'ZDH7YGF1', 'lunid': '5000c500c363d147'}, 'sdh': {'serial': 'ZDH7YTTK', 'lunid': '5000c500c3631d1d'}, 'sdf': {'serial': 'ZDH7YRMH', 'lunid': '5000c500c3640b5d'}, 'sdb': {'serial': 'ZGY5YXPE', 'lunid': '5000c500c363b983'}, 'nvme1n1': {'serial': 'S466NX0K828006E', 'lunid': '0025385881b2bf7b'}, 'nvme2n1': {'serial': '50026B7686965837', 'lunid': '00000000000000000026b76869658375'}}
[2024/04/23 20:10:55] (DEBUG) PoolService.import_on_boot():344 - Creating '/data/zfs' (if it doesnt already exist)
[2024/04/23 20:10:55] (DEBUG) PoolService.import_on_boot():351 - Creating '/data/zfs/zpool.cache' (if it doesnt already exist)
[2024/04/23 20:10:55] (DEBUG) PoolService.import_on_boot_impl():223 - Importing 'pool2' with guid: '18035793806657409897'
.......
[2024/04/23 20:11:12] (DEBUG) DiskService.sync_zfs_guid():79 - Pool 'apps' vdev '14785881892275095562' disk is None
[2024/04/23 20:11:12] (DEBUG) EtcService.generate():433 - No new changes for /etc/ssh/sshd_config
[2024/04/23 20:11:12] (DEBUG) DiskService.sync_zfs_guid():79 - Pool 'apps' vdev '14785881892275095562' disk is None
[2024/04/23 20:11:13] (DEBUG) PoolService.import_on_boot_impl():235 - SUCCESS importing 'apps' with guid: '17164552839078562837'
[2024/04/23 20:11:16] (DEBUG) PoolService.unlock_on_boot_impl():281 - SUCCESS unlocking encrypted dataset(s) (if any) for 'apps'
[2024/04/23 20:11:16] (DEBUG) PoolService.import_on_boot():379 - Calling pool.post_import
[2024/04/23 20:11:16] (WARNING) SystemDatasetService.sysdataset_path():112 - /var/db/system: mountpoint not found
[2024/04/23 20:11:19] (DEBUG) EtcService.generate():433 - No new changes for /etc/glusterfs/glusterd.vol
[2024/04/23 20:11:19] (DEBUG) EtcService.generate():433 - No new changes for /etc/smb4.conf
......
[2024/04/23 20:11:22] (DEBUG) EtcService.generate():433 - No new changes for /etc/scst.env
[2024/04/23 20:11:22] (DEBUG) EtcService.generate():433 - No new changes for /etc/initiators.allow
[2024/04/23 20:11:22] (DEBUG) EtcService.generate():433 - No new changes for /etc/initiators.deny
[2024/04/23 20:11:22] (DEBUG) EtcService.generate():433 - No new changes for /etc/glusterfs/glusterd.vol
[2024/04/23 20:11:22] (DEBUG) EtcService.generate():433 - No new changes for /etc/proftpd/proftpd.conf
[2024/04/23 20:11:23] (DEBUG) PoolService.import_on_boot():381 - Finished calling pool.post_import
[2024/04/23 20:11:24] (DEBUG) EtcService.generate():433 - No new changes for /etc/avahi/avahi-daemon.conf
[2024/04/23 20:11:24] (DEBUG) EtcService.generate():433 - No new changes for /etc/nfs.conf.d/local.conf
[2024/04/23 20:11:24] (DEBUG) EtcService.generate():433 - No new changes for /etc/default/rpcbind
[2024/04/23 20:11:24] (DEBUG) EtcService.generate():433 - No new changes for /etc/idmapd.conf
[2024/04/23 20:11:24] (DEBUG) EtcService.generate():433 - No new changes for /etc/exports
[2024/04/23 20:11:25] (DEBUG) middlewared._setup_periodic_tasks():1003 - Setting up periodic task alert.flush_alerts to run every 3600 seconds
[2024/04/23 20:11:25] (DEBUG) middlewared._setup_periodic_tasks():1003 - Setting up periodic task alert.process_alerts to run every 60 seconds
[2024/04/23 20:11:25] (DEBUG) middlewared._setup_periodic_tasks():1003 - Setting up periodic task auth.twofactor.remove_expired_secrets to run every 86400 seconds
[2024/04/23 20:11:25] (DEBUG) middlewared._setup_periodic_tasks():1003 - Setting up periodic task certificate.renew_certs to run every 86400 seconds
[2024/04/23 20:11:25] (DEBUG) middlewared._setup_periodic_tasks():1003 - Setting up periodic task certificateauthority.crl_generation to run every 86400 seconds
[2024/04/23 20:11:25] (DEBUG) middlewared._setup_periodic_tasks():1003 - Setting up periodic task chart.release.periodic_chart_releases_update_checks to run every 86400 seconds
....
[2024/04/23 20:28:00] (ERROR) ContainerImagesService.check_update():36 - [EFAULT] Unable to retrieve latest image digest for registry=ghcr.io image=onedr0p/readarr-develop tag=0.3.21.2475: 404, message='Not Found', url=URL('https://ghcr.io/v2/onedr0p/readarr-develop/manifests/0.3.21.2475')
[2024/04/23 20:28:00] (ERROR) ContainerImagesService.check_update():36 - [EFAULT] Unable to retrieve latest image digest for registry=registry.k8s.io image=sig-storage/smbplugin tag=v1.11.0: Timed out waiting for a response
[2024/04/23 20:28:01] (ERROR) ContainerImagesService.check_update():36 - [EFAULT] Unable to retrieve latest image digest for registry=ghcr.io image=onedr0p/readarr-develop tag=0.3.23.2506: 404, message='Not Found', url=URL('https://ghcr.io/v2/onedr0p/readarr-develop/manifests/0.3.23.2506')
[2024/04/23 20:28:02] (ERROR) ContainerImagesService.check_update():36 - [EFAULT] Unable to retrieve latest image digest for registry=ghcr.io image=onedr0p/readarr-develop tag=0.3.19.2437: 404, message='Not Found', url=URL('https://ghcr.io/v2/onedr0p/readarr-develop/manifests/0.3.19.2437')
[2024/04/23 20:28:02] (ERROR) ContainerImagesService.check_update():36 - [EFAULT] Unable to retrieve latest image digest for registry=ghcr.io image=cloudnative-pg/cloudnative-pg tag=1.20.0: 404, message='Not Found', url=URL('https://ghcr.io/v2/cloudnative-pg/cloudnative-pg/manifests/1.20.0')
[2024/04/23 20:28:07] (ERROR) ContainerImagesService.check_update():36 - [EFAULT] Unable to retrieve latest image digest for registry=ghcr.io image=onedr0p/readarr-develop tag=0.3.18.2411: 404, message='Not Found', url=URL('https://ghcr.io/v2/onedr0p/readarr-develop/manifests/0.3.18.2411')
[2024/04/23 20:28:10] (ERROR) ContainerImagesService.check_update():36 - [EFAULT] Unable to retrieve latest image digest for registry=ghcr.io image=onedr0p/readarr-develop tag=0.3.22.2499: 404, message='Not Found', url=URL('https://ghcr.io/v2/onedr0p/readarr-develop/manifests/0.3.22.2499')
[2024/04/23 20:33:05] (ERROR) ContainerImagesService.check_update():36 - [EFAULT] Unable to retrieve latest image digest for registry=registry.k8s.io image=sig-storage/snapshot-controller tag=v6.2.2: Timed out waiting for a response
[2024/04/23 20:33:13] (ERROR) ContainerImagesService.check_update():36 - [EFAULT] Unable to retrieve latest image digest for registry=registry.k8s.io image=sig-storage/smbplugin tag=v1.11.0: Timed out waiting for a response
[2024/04/23 20:33:16] (ERROR) ContainerImagesService.check_update():36 - [EFAULT] Unable to retrieve latest image digest for registry=ghcr.io image=onedr0p/readarr-develop tag=0.3.19.2437: 404, message='Not Found', url=URL('https://ghcr.io/v2/onedr0p/readarr-develop/manifests/0.3.19.2437')
[2024/04/23 20:38:08] (ERROR) ContainerImagesService.check_update():36 - [EFAULT] Unable to retrieve latest image digest for registry=registry.k8s.io image=sig-storage/csi-node-driver-registrar tag=v2.8.0: Timed out waiting for a response
[2024/04/23 20:38:09] (ERROR) ContainerImagesService.check_update():36 - [EFAULT] Unable to retrieve latest image digest for registry=ghcr.io image=onedr0p/readarr-develop tag=0.3.20.2452: 404, message='Not Found', url=URL('https://ghcr.io/v2/onedr0p/readarr-develop/manifests/0.3.20.2452')
[2024/04/23 20:38:12] (ERROR) ContainerImagesService.check_update():36 - [EFAULT] Unable to retrieve latest image digest for registry=ghcr.io image=onedr0p/readarr-develop tag=0.3.21.2475: 404, message='Not Found', url=URL('https://ghcr.io/v2/onedr0p/readarr-develop/manifests/0.3.21.2475')
[2024/04/23 20:38:15] (ERROR) ContainerImagesService.check_update():36 - [EFAULT] Unable to retrieve latest image digest for registry=ghcr.io image=onedr0p/readarr-develop tag=0.3.23.2506: 404, message='Not Found', url=URL('https://ghcr.io/v2/onedr0p/readarr-develop/manifests/0.3.23.2506')
[2024/04/23 20:38:15] (ERROR) ContainerImagesService.check_update():36 - [EFAULT] Unable to retrieve latest image digest for registry=ghcr.io image=cloudnative-pg/cloudnative-pg tag=1.20.0: 404, message='Not Found', url=URL('https://ghcr.io/v2/cloudnative-pg/cloudnative-pg/manifests/1.20.0')
[2024/04/23 20:38:20] (ERROR) ContainerImagesService.check_update():36 - [EFAULT] Unable to retrieve latest image digest for registry=ghcr.io image=onedr0p/readarr-develop tag=0.3.18.2411: 404, message='Not Found', url=URL('https://ghcr.io/v2/onedr0p/readarr-develop/manifests/0.3.18.2411')
[2024/04/23 20:48:20] (WARNING) middlewared._loop_monitor_thread():1766 - Task seems blocked:
File "/usr/lib/python3/dist-packages/aiohttp/connector.py", line 792, in close
return super().close()
File "/usr/lib/python3/dist-packages/aiohttp/connector.py", line 409, in close
self._close()
File "/usr/lib/python3/dist-packages/aiohttp/connector.py", line 432, in _close
proto.close()
File "/usr/lib/python3/dist-packages/aiohttp/client_proto.py", line 63, in close
transport.close()
File "/usr/lib/python3.11/asyncio/sslproto.py", line 112, in close
self._ssl_protocol._start_shutdown()
File "/usr/lib/python3.11/asyncio/sslproto.py", line 623, in _start_shutdown
self._do_flush()
File "/usr/lib/python3.11/asyncio/sslproto.py", line 636, in _do_flush
self._do_read()
File "/usr/lib/python3.11/asyncio/sslproto.py", line 732, in _do_read
self._do_read__copied()
File "/usr/lib/python3.11/asyncio/sslproto.py", line 777, in _do_read__copied
chunk = self._sslobj.read(self.max_size)
File "/usr/lib/python3.11/ssl.py", line 921, in read
v = self._sslobj.read(len)
[2024/04/23 20:49:11] (WARNING) middlewared._loop_monitor_thread():1766 - Task seems blocked:
File "/usr/lib/python3/dist-packages/aiohttp/connector.py", line 540, in connect
proto = await self._create_connection(req, traces, timeout)
File "/usr/lib/python3/dist-packages/aiohttp/connector.py", line 901, in _create_connection
_, proto = await self._create_direct_connection(req, traces, timeout)
File "/usr/lib/python3/dist-packages/aiohttp/connector.py", line 1175, in _create_direct_connection
transp, proto = await self._wrap_create_connection(
File "/usr/lib/python3/dist-packages/aiohttp/connector.py", line 980, in _wrap_create_connection
return await self._loop.create_connection(*args, **kwargs) # type: ignore[return-value] # noqa
File "/usr/lib/python3.11/asyncio/base_events.py", line 1069, in create_connection
sock = await self._connect_sock(
File "/usr/lib/python3.11/asyncio/base_events.py", line 973, in _connect_sock
await self.sock_connect(sock, address)
File "/usr/lib/python3.11/asyncio/selector_events.py", line 632, in sock_connect
self._sock_connect(fut, sock, address)
File "/usr/lib/python3.11/asyncio/selector_events.py", line 649, in _sock_connect
handle = self._add_writer(
File "/usr/lib/python3.11/asyncio/selector_events.py", line 304, in _add_writer
self._selector.register(fd, selectors.EVENT_WRITE,
File "/usr/lib/python3.11/selectors.py", line 359, in register
self._selector.register(key.fd, poller_events)
[2024/04/23 21:27:22] (ERROR) ContainerImagesService.check_update():36 - [EFAULT] Unable to retrieve latest image digest for registry=ghcr.io image=onedr0p/readarr-develop tag=0.3.23.2506: 404, message='Not Found', url=URL('https://ghcr.io/v2/onedr0p/readarr-develop/manifests/0.3.23.2506')
[2024/04/23 21:27:23] (ERROR) ContainerImagesService.check_update():36 - [EFAULT] Unable to retrieve latest image digest for registry=ghcr.io image=cloudnative-pg/cloudnative-pg tag=1.20.0: 404, message='Not Found', url=URL('https://ghcr.io/v2/cloudnative-pg/cloudnative-pg/manifests/1.20.0')
[2024/04/23 21:27:28] (ERROR) ContainerImagesService.check_update():36 - [EFAULT] Unable to retrieve latest image digest for registry=ghcr.io image=onedr0p/readarr-develop tag=0.3.18.2411: 404, message='Not Found', url=URL('https://ghcr.io/v2/onedr0p/readarr-develop/manifests/0.3.18.2411')
[2024/04/23 21:27:31] (ERROR) ContainerImagesService.check_update():36 - [EFAULT] Unable to retrieve latest image digest for registry=ghcr.io image=onedr0p/readarr-develop tag=0.3.22.2499: 404, message='Not Found', url=URL('https://ghcr.io/v2/onedr0p/readarr-develop/manifests/0.3.22.2499')
[2024/04/23 21:28:54] (WARNING) middlewared._loop_monitor_thread():1766 - Task seems blocked:
File "/usr/lib/python3.11/contextlib.py", line 204, in __aenter__
return await anext(self.gen)
File "/usr/lib/python3/dist-packages/middlewared/plugins/kubernetes_linux/k8s/client.py", line 29, in request
async with await getattr(session, mode)(
File "/usr/lib/python3/dist-packages/aiohttp/client.py", line 536, in _request
conn = await self._connector.connect(
File "/usr/lib/python3/dist-packages/aiohttp/connector.py", line 540, in connect
proto = await self._create_connection(req, traces, timeout)
File "/usr/lib/python3/dist-packages/aiohttp/connector.py", line 901, in _create_connection
_, proto = await self._create_direct_connection(req, traces, timeout)
File "/usr/lib/python3/dist-packages/aiohttp/connector.py", line 1175, in _create_direct_connection
transp, proto = await self._wrap_create_connection(
File "/usr/lib/python3/dist-packages/aiohttp/connector.py", line 980, in _wrap_create_connection
return await self._loop.create_connection(*args, **kwargs) # type: ignore[return-value] # noqa
File "/usr/lib/python3.11/asyncio/base_events.py", line 1069, in create_connection
sock = await self._connect_sock(
File "/usr/lib/python3.11/asyncio/base_events.py", line 950, in _connect_sock
sock = socket.socket(family=family, type=type_, proto=proto)
File "/usr/lib/python3.11/socket.py", line 232, in __init__
_socket.socket.__init__(self, family, type, proto, fileno)
[2024/04/23 21:28:54] (ERROR) middlewared.job.run():440 - Job <function CRUDService._get_crud_wrapper_func.<locals>.nf at 0x7fa988768040> failed: CallError('Failed to install App: Error: INSTALLATION FAILED: rendered manifests contain a resource that already exists. Unable to continue with install: ClusterRole "openebs-zfs-provisioner-role" in namespace "" exists and cannot be imported into the current release: invalid ownership metadata; label validation error: missing key "app.kubernetes.io/managed-by": must be set to "Helm"; annotation validation error: missing key "meta.helm.sh/release-name": must be set to "openebs"; annotation validation error: missing key "meta.helm.sh/release-namespace": must be set to "ix-openebs"\n')
[2024/04/23 21:30:59] (WARNING) middlewared.download_impl_scale():39 - Invalid update file checksum 'd1ee088e68037163324e0448800d6e4878ed1ab06c531e5e370dccaba0d5b07b', re-downloading
[2024/04/23 19:36:18] (INFO) middlewared.__init__():898 - Starting TrueNAS-SCALE-24.04.0 middleware
[2024/04/23 19:36:18] (DEBUG) middlewared._console_write():1171 - Failed to write to console: OSError(5, 'Input/output error')
[2024/04/23 19:36:18] (DEBUG) middlewared._console_write():1171 - Failed to write to console: OSError(5, 'Input/output error')
[2024/04/23 19:36:18] (DEBUG) middlewared._console_write():1171 - Failed to write to console: OSError(5, 'Input/output error')
[2024/04/23 21:36:21] (DEBUG) middlewared.setup():41 - Setting timezone to 'Europe/Oslo'
[2024/04/23 21:36:21] (INFO) UserService.sync_builtin():102 - Creating new group 'nobody'
[2024/04/23 21:36:21] (INFO) UserService.sync_builtin():125 - Removing groups ['gluster']
[2024/04/23 21:36:22] (INFO) UserService.sync_builtin():289 - Removing users ['gluster']
[2024/04/23 21:36:22] (DEBUG) middlewared.generate_keys():213 - Generating SSH host keys
[2024/04/23 21:36:32] (INFO) RDMAService._get_link_choices():67 - Fetching RDMA link netdev choices
[2024/04/23 21:36:33] (DEBUG) middlewared.setup():25 - Certificate setup for System complete
[2024/04/23 21:36:33] (DEBUG) middlewared.__plugins_setup():1063 - All plugins loaded
[2024/04/23 21:36:33] (DEBUG) middlewared.__initialize():2033 - Accepting connections
[2024/04/23 21:36:33] (INFO) DiskService.log_disk_info():75 - Found disks: {'nvme0n1': {'serial': 'S463NF0M524905Y', 'lunid': '0025385591b3e6d7'}, 'sda': {'serial': '4ZG500YV', 'lunid': '5000cca2a6c245f8'}, 'sdb': {'serial': '4ZG4XHKV', 'lunid': '5000cca2a6c23c7a'}, 'sdc': {'serial': '4BHLWKRH', 'lunid': '5000cca2b6d6adaf'}, 'sdd': {'serial': 'ZDH7ZDX2', 'lunid': '5000c500c3647fba'}, 'sde': {'serial': 'ZDH7YGF1', 'lunid': '5000c500c363d147'}, 'sdf': {'serial': 'ZDH7YTTK', 'lunid': '5000c500c3631d1d'}, 'sdh': {'serial': 'ZDH7YRMH', 'lunid': '5000c500c3640b5d'}, 'sdg': {'serial': 'ZGY5YXPE', 'lunid': '5000c500c363b983'}, 'nvme1n1': {'serial': 'S466NX0K828006E', 'lunid': '0025385881b2bf7b'}, 'nvme2n1': {'serial': '50026B7686965837', 'lunid': '00000000000000000026b76869658375'}}
[2024/04/23 21:36:33] (DEBUG) PoolService.import_on_boot():396 - Creating '/data/zfs' (if it doesnt already exist)
[2024/04/23 21:36:33] (DEBUG) PoolService.import_on_boot():403 - Creating '/data/zfs/zpool.cache' (if it doesnt already exist)
[2024/04/23 21:36:33] (DEBUG) PoolService.import_on_boot_impl():226 - Importing 'pool2' with guid: '18035793806657409897'
[2024/04/23 21:36:34] (DEBUG) EtcService.generate():429 - No new changes for /etc/shadow
[2024/04/23 21:36:34] (DEBUG) EtcService.generate():429 - No new changes for /etc/netdata/exporting.conf
[2024/04/23 21:36:34] (DEBUG) EtcService.generate():414 - mako:krb5.conf file removed.
[2024/04/23 21:36:43] (DEBUG) PoolService.import_on_boot_impl():270 - SUCCESS importing 'pool2' with guid: '18035793806657409897'
[2024/04/23 21:36:44] (DEBUG) EtcService.generate():429 - No new changes for /etc/keepalived/keepalived.conf
[2024/04/23 21:36:44] (DEBUG) PoolService.unlock_on_boot_impl():316 - SUCCESS unlocking encrypted dataset(s) (if any) for 'pool2'
[2024/04/23 21:36:44] (DEBUG) middlewared.render_body():35 - UPSMON: monpwd field empty, upsmon will fail to start.
[2024/04/23 21:36:45] (DEBUG) PoolService.import_on_boot_impl():226 - Importing 'apps' with guid: '17164552839078562837'
[2024/04/23 21:36:48] (INFO) InterfaceService.configure():21 - Configuring interface 'enp5s0'
[2024/04/23 21:36:48] (DEBUG) InterfaceService.configure():106 - enp5s0: adding 192.168.10.11/255.255.255.0
[2024/04/23 21:36:48] (DEBUG) InterfaceService.configure():106 - enp5s0: adding 192.168.10.10/255.255.255.0
[2024/04/23 21:36:48] (INFO) InterfaceService.sync():1725 - Interfaces in database: enp5s0
[2024/04/23 21:36:48] (DEBUG) EtcService.generate():429 - No new changes for /etc/dhcp/dhclient.conf
[2024/04/23 21:36:48] (INFO) RouteService.sync():83 - Adding IPv4 default route to 192.168.10.1
[2024/04/23 21:36:52] (DEBUG) PoolService.import_on_boot_impl():270 - SUCCESS importing 'apps' with guid: '17164552839078562837'
[2024/04/23 21:36:56] (DEBUG) PoolService.unlock_on_boot_impl():316 - SUCCESS unlocking encrypted dataset(s) (if any) for 'apps'
[2024/04/23 21:36:56] (DEBUG) PoolService.import_on_boot():431 - Calling pool.post_import
[2024/04/23 21:36:56] (WARNING) SystemDatasetService.sysdataset_path():110 - /var/db/system: mountpoint not found
[2024/04/23 21:36:59] (DEBUG) EtcService.generate():429 - No new changes for /etc/smb4.conf
[2024/04/23 21:37:00] (DEBUG) SMBService.reg_globals():89 - Failed to retrieve global share config from registry
[2024/04/23 21:37:00] (ERROR) ServiceService._restart():306 - Service 'idmap' not running after restart
[2024/04/23 21:37:01] (DEBUG) EtcService.generate():429 - No new changes for /etc/group
[2024/04/23 21:37:01] (DEBUG) EtcService.generate():429 - No new changes for /etc/passwd
[2024/04/23 21:37:01] (DEBUG) EtcService.generate():429 - No new changes for /etc/shadow
[2024/04/23 21:37:01] (DEBUG) EtcService.generate():429 - No new changes for /etc/sudoers
[2024/04/23 21:37:01] (DEBUG) EtcService.generate():429 - No new changes for /etc/aliases
[2024/04/23 21:37:01] (DEBUG) EtcService.generate():429 - No new changes for /etc/smb4.conf
[2024/04/23 21:37:03] (DEBUG) middlewared.remove_service_configs():271 - Removing unexpected avahi service file: /etc/avahi/services/nut.service
[2024/04/23 21:37:03] (DEBUG) EtcService.generate():429 - No new changes for /etc/initiators.allow
[2024/04/23 21:37:03] (DEBUG) EtcService.generate():429 - No new changes for /etc/initiators.deny
[2024/04/23 21:37:03] (DEBUG) EtcService.generate():429 - No new changes for /etc/proftpd/proftpd.conf
[2024/04/23 21:37:05] (DEBUG) EtcService.generate():429 - No new changes for /etc/nfs.conf.d/local.conf
[2024/04/23 21:37:05] (DEBUG) EtcService.generate():429 - No new changes for /etc/default/rpcbind
[2024/04/23 21:37:05] (DEBUG) EtcService.generate():429 - No new changes for /etc/idmapd.conf
[2024/04/23 21:37:05] (DEBUG) EtcService.generate():429 - No new changes for /etc/exports
[2024/04/23 21:37:07] (DEBUG) PoolService.import_on_boot():433 - Finished calling pool.post_import
[2024/04/23 21:37:07] (DEBUG) EtcService.generate():429 - No new changes for /etc/avahi/avahi-daemon.conf
[2024/04/23 21:37:09] (DEBUG) middlewared._setup_periodic_tasks():1076 - Setting up periodic task alert.flush_alerts to run every 3600 seconds
[2024/04/23 21:37:09] (DEBUG) middlewared._setup_periodic_tasks():1076 - Setting up periodic task alert.process_alerts to run every 60 seconds
[2024/04/23 21:37:09] (DEBUG) middlewared._setup_periodic_tasks():1076 - Setting up periodic task auditbackend._AuditBackendService__lifecycle_cleanup to run every 86400 seconds
[2024/04/23 21:37:09] (DEBUG) middlewared._setup_periodic_tasks():1076 - Setting up periodic task auth.twofactor.remove_expired_secrets to run every 86400 seconds
[2024/04/23 21:37:09] (DEBUG) middlewared._setup_periodic_tasks():1076 - Setting up periodic task certificate.renew_certs to run every 86400 seconds
[2024/04/23 21:37:09] (DEBUG) middlewared._setup_periodic_tasks():1076 - Setting up periodic task certificateauthority.crl_generation to run every 86400 seconds
[2024/04/23 21:37:09] (DEBUG) middlewared._setup_periodic_tasks():1076 - Setting up periodic task chart.release.periodic_chart_releases_update_checks to run every 86400 seconds
[2024/04/23 21:37:09] (DEBUG) middlewared._setup_periodic_tasks():1076 - Setting up periodic task kerberos.keytab.check_updated_keytab to run every 3600 seconds
[2024/04/23 21:37:09] (DEBUG) middlewared._setup_periodic_tasks():1076 - Setting up periodic task kmip.sync_keys to run every 86400 seconds
[2024/04/23 21:37:09] (DEBUG) middlewared._setup_periodic_tasks():1076 - Setting up periodic task mail.send_mail_queue to run every 600 seconds
[2024/04/23 21:37:09] (DEBUG) middlewared._setup_periodic_tasks():1076 - Setting up periodic task pool.dataset.sync_db_keys to run every 86400 seconds
[2024/04/23 21:37:09] (DEBUG) middlewared._setup_periodic_tasks():1076 - Setting up periodic task reporting.netdataweb_expire to run every 600 seconds
[2024/04/23 21:37:09] (DEBUG) middlewared._setup_periodic_tasks():1076 - Setting up periodic task service.check_deprecated_services to run every 3600 seconds
[2024/04/23 21:37:09] (DEBUG) middlewared._setup_periodic_tasks():1076 - Setting up periodic task smb.sharesec.check_share_info_tdb to run every 3600 seconds
[2024/04/23 21:37:09] (DEBUG) middlewared._setup_periodic_tasks():1076 - Setting up periodic task truecommand.health_check to run every 1800 seconds
[2024/04/23 21:37:09] (DEBUG) middlewared._setup_periodic_tasks():1076 - Setting up periodic task vmware.delete_pending_snapshots to run every 10800.0 seconds
[2024/04/23 21:37:09] (DEBUG) middlewared._setup_periodic_tasks():1076 - Setting up periodic task zettarepl.flush_state to run every 3600 seconds
[2024/04/23 21:37:09] (DEBUG) EtcService.generate():429 - No new changes for /etc/avahi/avahi-daemon.conf
[2024/04/23 21:37:11] (DEBUG) EtcService.generate():429 - No new changes for /etc/nginx/nginx.conf
[2024/04/23 21:37:11] (WARNING) middlewared.plugins.service_.services.base.systemd_unit():171 - nginx reload failed with code 1: 'nginx.service is not active, cannot reload.\n'
[2024/04/23 21:37:13] (DEBUG) EtcService.generate():429 - No new changes for /etc/nginx/nginx.conf
[2024/04/23 21:37:13] (WARNING) middlewared.plugins.service_.services.base.systemd_unit():171 - nginx reload failed with code 1: 'nginx.service is not active, cannot reload.\n'
[2024/04/23 21:37:16] (DEBUG) EtcService.generate():429 - No new changes for /etc/wsdd.conf
[2024/04/23 22:59:05] (INFO) middlewared.terminate():2039 - Terminating
[2024/04/23 22:59:21] (DEBUG) VMService.teardown_guest_vmemory():51 - Giving back guest memory to ARC: 115964116992
[2024/04/23 22:59:21] (DEBUG) VMService.teardown_guest_vmemory():51 - Giving back guest memory to ARC: 124554051584
[2024/04/23 22:59:21] (DEBUG) VMService.teardown_guest_vmemory():51 - Giving back guest memory to ARC: 133143986176
[2024/04/23 22:59:25] (DEBUG) middlewared.__event_system_shutdown():145 - VM(s) stopped successfully
[2024/04/23 22:59:25] (DEBUG) EtcService.generate():429 - No new changes for /etc/nginx/nginx.conf
[2024/04/23 23:00:35] (ERROR) middlewared.__terminate():2065 - Failed to terminate vm
Traceback (most recent call last):
File "/usr/lib/python3.11/asyncio/tasks.py", line 490, in wait_for
return fut.result()
^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/plugins/vm/lifecycle.py", line 103, in terminate
async with SHUTDOWN_LOCK:
File "/usr/lib/python3.11/asyncio/locks.py", line 15, in __aenter__
await self.acquire()
File "/usr/lib/python3.11/asyncio/locks.py", line 114, in acquire
await fut
asyncio.exceptions.CancelledError
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 2063, in __terminate
await asyncio.wait_for(self.create_task(service.terminate()), timeout)
File "/usr/lib/python3.11/asyncio/tasks.py", line 492, in wait_for
raise exceptions.TimeoutError() from exc
TimeoutError
[2024/04/23 23:02:09] (INFO) middlewared.__init__():898 - Starting TrueNAS-SCALE-24.04.0 middleware
[2024/04/23 23:02:09] (DEBUG) middlewared._console_write():1171 - Failed to write to console: OSError(5, 'Input/output error')
[2024/04/23 23:02:09] (DEBUG) middlewared._console_write():1171 - Failed to write to console: OSError(5, 'Input/output error')
[2024/04/23 23:02:10] (DEBUG) middlewared._console_write():1171 - Failed to write to console: OSError(5, 'Input/output error')
[2024/04/23 23:02:14] (DEBUG) middlewared.setup():41 - Setting timezone to 'Europe/Oslo'
[2024/04/23 23:02:14] (DEBUG) middlewared.generate_keys():213 - Generating SSH host keys
[2024/04/23 23:02:24] (INFO) RDMAService._get_link_choices():67 - Fetching RDMA link netdev choices
[2024/04/23 23:02:24] (DEBUG) middlewared.setup():25 - Certificate setup for System complete
[2024/04/23 23:02:24] (DEBUG) middlewared.__plugins_setup():1063 - All plugins loaded
[2024/04/23 23:02:24] (DEBUG) middlewared.__initialize():2033 - Accepting connections
[2024/04/23 23:02:24] (INFO) DiskService.log_disk_info():75 - Found disks: {'nvme0n1': {'serial': 'S463NF0M524905Y', 'lunid': '0025385591b3e6d7'}, 'sde': {'serial': '4ZG500YV', 'lunid': '5000cca2a6c245f8'}, 'sdd': {'serial': '4ZG4XHKV', 'lunid': '5000cca2a6c23c7a'}, 'sda': {'serial': '4BHLWKRH', 'lunid': '5000cca2b6d6adaf'}, 'sdb': {'serial': 'ZDH7ZDX2', 'lunid': '5000c500c3647fba'}, 'sdc': {'serial': 'ZDH7YGF1', 'lunid': '5000c500c363d147'}, 'sdf': {'serial': 'ZDH7YTTK', 'lunid': '5000c500c3631d1d'}, 'sdg': {'serial': 'ZDH7YRMH', 'lunid': '5000c500c3640b5d'}, 'sdh': {'serial': 'ZGY5YXPE', 'lunid': '5000c500c363b983'}, 'nvme1n1': {'serial': 'S466NX0K828006E', 'lunid': '0025385881b2bf7b'}, 'nvme2n1': {'serial': '50026B7686965837', 'lunid': '00000000000000000026b76869658375'}}
[2024/04/23 23:02:24] (DEBUG) PoolService.import_on_boot():396 - Creating '/data/zfs' (if it doesnt already exist)
[2024/04/23 23:02:24] (DEBUG) PoolService.import_on_boot():403 - Creating '/data/zfs/zpool.cache' (if it doesnt already exist)
[2024/04/23 23:02:24] (DEBUG) PoolService.import_on_boot_impl():226 - Importing 'pool2' with guid: '18035793806657409897'
[2024/04/23 23:02:25] (DEBUG) EtcService.generate():429 - No new changes for /etc/shadow
[2024/04/23 23:02:25] (DEBUG) EtcService.generate():429 - No new changes for /etc/group
[2024/04/23 23:02:25] (DEBUG) EtcService.generate():429 - No new changes for /etc/passwd
[2024/04/23 23:02:25] (DEBUG) EtcService.generate():429 - No new changes for /etc/shadow
[2024/04/23 23:02:25] (DEBUG) EtcService.generate():429 - No new changes for /etc/sudoers
[2024/04/23 23:02:25] (DEBUG) EtcService.generate():429 - No new changes for /etc/aliases
[2024/04/23 23:02:25] (DEBUG) EtcService.generate():429 - No new changes for /etc/netdata/exporting.conf
[2024/04/23 23:02:25] (DEBUG) EtcService.generate():429 - No new changes for /etc/netdata/python.d/smart_log.conf
[2024/04/23 23:02:25] (DEBUG) EtcService.generate():429 - No new changes for /etc/fstab
[2024/04/23 23:02:25] (DEBUG) EtcService.generate():429 - No new changes for /etc/default/keyboard
[2024/04/23 23:02:25] (DEBUG) EtcService.generate():429 - No new changes for /etc/vconsole.conf
[2024/04/23 23:02:25] (DEBUG) EtcService.generate():429 - No new changes for /etc/openldap/ldap.conf
[2024/04/23 23:02:25] (DEBUG) EtcService.generate():429 - No new changes for /etc/nslcd.conf
[2024/04/23 23:02:25] (DEBUG) EtcService.generate():429 - No new changes for /etc/dhcp/dhclient.conf
[2024/04/23 23:02:34] (DEBUG) EtcService.generate():429 - No new changes for /etc/nfs.conf.d/local.conf
[2024/04/23 23:02:34] (DEBUG) EtcService.generate():429 - No new changes for /etc/default/rpcbind
[2024/04/23 23:02:34] (DEBUG) EtcService.generate():429 - No new changes for /etc/idmapd.conf
[2024/04/23 23:02:34] (DEBUG) EtcService.generate():429 - No new changes for /etc/pam.d/common-account
[2024/04/23 23:02:34] (DEBUG) EtcService.generate():429 - No new changes for /etc/pam.d/common-auth
[2024/04/23 23:02:34] (DEBUG) EtcService.generate():429 - No new changes for /etc/pam.d/common-password
[2024/04/23 23:02:34] (DEBUG) EtcService.generate():429 - No new changes for /etc/pam.d/common-session-noninteractive
[2024/04/23 23:02:34] (DEBUG) EtcService.generate():429 - No new changes for /etc/pam.d/common-session
[2024/04/23 23:02:34] (DEBUG) EtcService.generate():429 - No new changes for /etc/security/pam_winbind.conf
[2024/04/23 23:02:34] (DEBUG) EtcService.generate():429 - No new changes for /etc/pam.d/middleware
[2024/04/23 23:02:34] (DEBUG) EtcService.generate():429 - No new changes for /etc/proftpd/proftpd.conf
[2024/04/23 23:02:34] (DEBUG) EtcService.generate():429 - No new changes for /etc/default/kdump-tools
[2024/04/23 23:02:34] (DEBUG) EtcService.generate():429 - No new changes for /etc/sysctl.d/tunables.conf
[2024/04/23 23:02:34] (DEBUG) EtcService.generate():429 - No new changes for /etc/default/smartmontools
[2024/04/23 23:02:35] (DEBUG) EtcService.generate():429 - No new changes for /etc/keepalived/keepalived.conf
[2024/04/23 23:02:36] (DEBUG) EtcService.generate():429 - No new changes for /etc/motd
[2024/04/23 23:02:36] (DEBUG) EtcService.generate():429 - No new changes for /etc/nscd.conf
[2024/04/23 23:02:36] (DEBUG) middlewared.render_body():35 - UPSMON: monpwd field empty, upsmon will fail to start.
[2024/04/23 23:02:36] (DEBUG) EtcService.generate():429 - No new changes for /etc/smb4.conf
[2024/04/23 23:02:36] (DEBUG) EtcService.generate():429 - No new changes for /etc/snmp/snmpd.conf
[2024/04/23 23:02:36] (DEBUG) EtcService.generate():429 - No new changes for /etc/syslog-ng/syslog-ng.conf
[2024/04/23 23:02:36] (DEBUG) EtcService.generate():429 - No new changes for /etc/syslog-ng/conf.d/tndestinations.conf
[2024/04/23 23:02:36] (DEBUG) EtcService.generate():429 - No new changes for /etc/syslog-ng/conf.d/tnfilters.conf
[2024/04/23 23:02:36] (DEBUG) EtcService.generate():429 - No new changes for /etc/syslog-ng/conf.d/tnaudit.conf
[2024/04/23 23:02:36] (DEBUG) EtcService.generate():429 - No new changes for /etc/pam.d/sshd
[2024/04/23 23:02:36] (DEBUG) EtcService.generate():429 - No new changes for /etc/chrony/chrony.conf
[2024/04/23 23:02:36] (DEBUG) PoolService.import_on_boot_impl():270 - SUCCESS importing 'pool2' with guid: '18035793806657409897'
[2024/04/23 23:02:36] (DEBUG) EtcService.generate():429 - No new changes for /etc/pykmip/pykmip.conf
[2024/04/23 23:02:37] (DEBUG) EtcService.generate():429 - No new changes for /etc/hosts
[2024/04/23 23:02:37] (INFO) InterfaceService.configure():21 - Configuring interface 'enp5s0'
[2024/04/23 23:02:37] (DEBUG) InterfaceService.configure():106 - enp5s0: adding 192.168.10.10/255.255.255.0
[2024/04/23 23:02:37] (DEBUG) InterfaceService.configure():106 - enp5s0: adding 192.168.10.11/255.255.255.0
[2024/04/23 23:02:37] (DEBUG) PoolService.unlock_on_boot_impl():316 - SUCCESS unlocking encrypted dataset(s) (if any) for 'pool2'
[2024/04/23 23:02:37] (DEBUG) PoolService.import_on_boot_impl():226 - Importing 'apps' with guid: '17164552839078562837'
[2024/04/23 23:02:37] (INFO) InterfaceService.sync():1725 - Interfaces in database: enp5s0
[2024/04/23 23:02:37] (DEBUG) EtcService.generate():429 - No new changes for /etc/dhcp/dhclient.conf
[2024/04/23 23:02:37] (INFO) RouteService.sync():83 - Adding IPv4 default route to 192.168.10.1
[2024/04/23 23:02:40] (DEBUG) EtcService.generate():429 - No new changes for /etc/exports
[2024/04/23 23:02:40] (DEBUG) EtcService.generate():429 - No new changes for /etc/nginx/nginx.conf
[2024/04/23 23:02:41] (DEBUG) EtcService.generate():429 - No new changes for /etc/ssh/sshd_config
[2024/04/23 23:02:43] (DEBUG) PoolService.import_on_boot_impl():270 - SUCCESS importing 'apps' with guid: '17164552839078562837'
[2024/04/23 23:02:47] (DEBUG) PoolService.unlock_on_boot_impl():316 - SUCCESS unlocking encrypted dataset(s) (if any) for 'apps'
[2024/04/23 23:02:47] (DEBUG) PoolService.import_on_boot():431 - Calling pool.post_import
[2024/04/23 23:02:47] (WARNING) SystemDatasetService.sysdataset_path():110 - /var/db/system: mountpoint not found
[2024/04/23 23:02:49] (DEBUG) EtcService.generate():429 - No new changes for /etc/smb4.conf
[2024/04/23 23:02:51] (DEBUG) EtcService.generate():429 - No new changes for /etc/netdata/netdata.conf
[2024/04/23 23:02:51] (DEBUG) EtcService.generate():429 - No new changes for /etc/netdata/charts.d/exclude_netdata.conf
[2024/04/23 23:02:51] (DEBUG) SMBService.reg_globals():89 - Failed to retrieve global share config from registry
[2024/04/23 23:02:51] (ERROR) ServiceService._restart():306 - Service 'idmap' not running after restart
[2024/04/23 23:02:51] (DEBUG) EtcService.generate():429 - No new changes for /etc/group
[2024/04/23 23:02:51] (DEBUG) EtcService.generate():429 - No new changes for /etc/passwd
[2024/04/23 23:02:51] (DEBUG) EtcService.generate():429 - No new changes for /etc/shadow
[2024/04/23 23:02:51] (DEBUG) EtcService.generate():429 - No new changes for /etc/sudoers
[2024/04/23 23:02:51] (DEBUG) EtcService.generate():429 - No new changes for /etc/aliases
[2024/04/23 23:02:52] (DEBUG) EtcService.generate():429 - No new changes for /etc/smb4.conf
[2024/04/23 23:02:52] (DEBUG) EtcService.generate():429 - No new changes for /etc/avahi/avahi-daemon.conf
[2024/04/23 23:02:53] (DEBUG) EtcService.generate():429 - No new changes for /etc/scst.conf
[2024/04/23 23:02:53] (DEBUG) EtcService.generate():429 - No new changes for /etc/scst.env
[2024/04/23 23:02:53] (DEBUG) EtcService.generate():429 - No new changes for /etc/initiators.allow
[2024/04/23 23:02:53] (DEBUG) EtcService.generate():429 - No new changes for /etc/initiators.deny
[2024/04/23 23:02:54] (DEBUG) EtcService.generate():429 - No new changes for /etc/proftpd/proftpd.conf
[2024/04/23 23:02:55] (DEBUG) EtcService.generate():429 - No new changes for /etc/nfs.conf.d/local.conf
[2024/04/23 23:02:55] (DEBUG) EtcService.generate():429 - No new changes for /etc/default/rpcbind
[2024/04/23 23:02:55] (DEBUG) EtcService.generate():429 - No new changes for /etc/idmapd.conf
[2024/04/23 23:02:55] (DEBUG) EtcService.generate():429 - No new changes for /etc/exports
[2024/04/23 23:02:57] (DEBUG) PoolService.import_on_boot():433 - Finished calling pool.post_import
[2024/04/23 23:02:57] (DEBUG) EtcService.generate():429 - No new changes for /etc/avahi/avahi-daemon.conf
[2024/04/23 23:02:59] (DEBUG) middlewared._setup_periodic_tasks():1076 - Setting up periodic task alert.flush_alerts to run every 3600 seconds
[2024/04/23 23:02:59] (DEBUG) middlewared._setup_periodic_tasks():1076 - Setting up periodic task alert.process_alerts to run every 60 seconds
[2024/04/23 23:02:59] (DEBUG) middlewared._setup_periodic_tasks():1076 - Setting up periodic task auditbackend._AuditBackendService__lifecycle_cleanup to run every 86400 seconds
[2024/04/23 23:02:59] (DEBUG) middlewared._setup_periodic_tasks():1076 - Setting up periodic task auth.twofactor.remove_expired_secrets to run every 86400 seconds
[2024/04/23 23:02:59] (DEBUG) middlewared._setup_periodic_tasks():1076 - Setting up periodic task certificate.renew_certs to run every 86400 seconds
[2024/04/23 23:02:59] (DEBUG) middlewared._setup_periodic_tasks():1076 - Setting up periodic task certificateauthority.crl_generation to run every 86400 seconds
[2024/04/23 23:02:59] (DEBUG) middlewared._setup_periodic_tasks():1076 - Setting up periodic task chart.release.periodic_chart_releases_update_checks to run every 86400 seconds
[2024/04/23 23:02:59] (DEBUG) middlewared._setup_periodic_tasks():1076 - Setting up periodic task kerberos.keytab.check_updated_keytab to run every 3600 seconds
[2024/04/23 23:02:59] (DEBUG) middlewared._setup_periodic_tasks():1076 - Setting up periodic task kmip.sync_keys to run every 86400 seconds
[2024/04/23 23:02:59] (DEBUG) middlewared._setup_periodic_tasks():1076 - Setting up periodic task mail.send_mail_queue to run every 600 seconds
[2024/04/23 23:02:59] (DEBUG) middlewared._setup_periodic_tasks():1076 - Setting up periodic task pool.dataset.sync_db_keys to run every 86400 seconds
[2024/04/23 23:02:59] (DEBUG) middlewared._setup_periodic_tasks():1076 - Setting up periodic task reporting.netdataweb_expire to run every 600 seconds
[2024/04/23 23:02:59] (DEBUG) middlewared._setup_periodic_tasks():1076 - Setting up periodic task service.check_deprecated_services to run every 3600 seconds
[2024/04/23 23:02:59] (DEBUG) middlewared._setup_periodic_tasks():1076 - Setting up periodic task smb.sharesec.check_share_info_tdb to run every 3600 seconds
[2024/04/23 23:02:59] (DEBUG) middlewared._setup_periodic_tasks():1076 - Setting up periodic task truecommand.health_check to run every 1800 seconds
[2024/04/23 23:02:59] (DEBUG) middlewared._setup_periodic_tasks():1076 - Setting up periodic task vmware.delete_pending_snapshots to run every 10800.0 seconds
[2024/04/23 23:02:59] (DEBUG) middlewared._setup_periodic_tasks():1076 - Setting up periodic task zettarepl.flush_state to run every 3600 seconds
[2024/04/23 23:02:59] (DEBUG) EtcService.generate():429 - No new changes for /etc/avahi/avahi-daemon.conf
[2024/04/23 23:02:59] (DEBUG) EtcService.generate():429 - No new changes for /etc/default/libvirt-guests
[2024/04/23 23:03:01] (DEBUG) EtcService.generate():429 - No new changes for /etc/nginx/nginx.conf
[2024/04/23 23:03:01] (WARNING) middlewared.plugins.service_.services.base.systemd_unit():171 - nginx reload failed with code 1: 'nginx.service is not active, cannot reload.\n'
[2024/04/23 23:03:03] (DEBUG) EtcService.generate():429 - No new changes for /etc/nginx/nginx.conf
[2024/04/23 23:03:03] (WARNING) middlewared.plugins.service_.services.base.systemd_unit():171 - nginx reload failed with code 1: 'nginx.service is not active, cannot reload.\n'
[2024/04/23 23:03:06] (DEBUG) EtcService.generate():429 - No new changes for /etc/containerd.env
[2024/04/23 23:03:07] (DEBUG) EtcService.generate():429 - No new changes for /etc/wsdd.conf
[2024/04/23 23:04:20] (ERROR) ContainerImagesService.check_update():36 - [EFAULT] Unable to retrieve latest image digest for registry=ghcr.io image=onedr0p/readarr-develop tag=0.3.22.2499: 404, message='Not Found', url=URL('https://ghcr.io/v2/onedr0p/readarr-develop/manifests/0.3.22.2499')
[2024/04/23 23:04:21] (ERROR) ContainerImagesService.check_update():36 - [EFAULT] Unable to retrieve latest image digest for registry=ghcr.io image=onedr0p/readarr-develop tag=0.3.19.2437: 404, message='Not Found', url=URL('https://ghcr.io/v2/onedr0p/readarr-develop/manifests/0.3.19.2437')
[2024/04/23 23:09:22] (ERROR) ContainerImagesService.check_update():36 - [EFAULT] Unable to retrieve latest image digest for registry=registry.k8s.io image=sig-storage/csi-node-driver-registrar tag=v2.8.0: Timed out waiting for a response
[2024/04/23 23:09:25] (ERROR) ContainerImagesService.check_update():36 - [EFAULT] Unable to retrieve latest image digest for registry=ghcr.io image=autobrr/autobrr tag=v1.39.1: Timed out waiting for a response
[2024/04/23 23:09:25] (ERROR) ContainerImagesService.check_update():36 - [EFAULT] Unable to retrieve latest image digest for registry=ghcr.io image=onedr0p/readarr-develop tag=0.3.19.2437: 404, message='Not Found', url=URL('https://ghcr.io/v2/onedr0p/readarr-develop/manifests/0.3.19.2437')
[2024/04/23 23:09:29] (ERROR) ContainerImagesService.check_update():36 - [EFAULT] Unable to retrieve latest image digest for registry=ghcr.io image=onedr0p/readarr-develop tag=0.3.18.2411: 404, message='Not Found', url=URL('https://ghcr.io/v2/onedr0p/readarr-develop/manifests/0.3.18.2411')
[2024/04/23 23:09:34] (ERROR) ContainerImagesService.check_update():36 - [EFAULT] Unable to retrieve latest image digest for registry=ghcr.io image=cloudnative-pg/cloudnative-pg tag=1.20.0: 404, message='Not Found', url=URL('https://ghcr.io/v2/cloudnative-pg/cloudnative-pg/manifests/1.20.0')
[2024/04/23 23:09:35] (ERROR) ContainerImagesService.check_update():36 - [EFAULT] Unable to retrieve latest image digest for registry=ghcr.io image=onedr0p/readarr-develop tag=0.3.21.2475: 404, message='Not Found', url=URL('https://ghcr.io/v2/onedr0p/readarr-develop/manifests/0.3.21.2475')
[2024/04/23 23:09:36] (ERROR) ContainerImagesService.check_update():36 - [EFAULT] Unable to retrieve latest image digest for registry=ghcr.io image=onedr0p/readarr-develop tag=0.3.23.2506: 404, message='Not Found', url=URL('https://ghcr.io/v2/onedr0p/readarr-develop/manifests/0.3.23.2506')
[2024/04/23 23:14:26] (ERROR) ContainerImagesService.check_update():36 - [EFAULT] Unable to retrieve latest image digest for registry=registry.k8s.io image=sig-storage/csi-provisioner tag=v3.5.0: Timed out waiting for a response
[2024/04/23 23:14:40] (ERROR) ContainerImagesService.check_update():36 - [EFAULT] Unable to retrieve latest image digest for registry=registry.k8s.io image=sig-storage/livenessprobe tag=v2.10.0: Timed out waiting for a response
[2024/04/23 23:14:40] (ERROR) ContainerImagesService.check_update():36 - [EFAULT] Unable to retrieve latest image digest for registry=ghcr.io image=onedr0p/readarr-develop tag=0.3.20.2452: 404, message='Not Found', url=URL('https://ghcr.io/v2/onedr0p/readarr-develop/manifests/0.3.20.2452')
[2024/04/23 23:14:44] (ERROR) ContainerImagesService.check_update():36 - [EFAULT] Unable to retrieve latest image digest for registry=ghcr.io image=onedr0p/readarr-develop tag=0.3.22.2499: 404, message='Not Found', url=URL('https://ghcr.io/v2/onedr0p/readarr-develop/manifests/0.3.22.2499')
[2024/04/23 23:19:30] (ERROR) ContainerImagesService.check_update():36 - [EFAULT] Unable to retrieve latest image digest for registry=ghcr.io image=autobrr/autobrr tag=v1.39.0: Timed out waiting for a response
[2024/04/23 23:19:33] (ERROR) ContainerImagesService.check_update():36 - [EFAULT] Unable to retrieve latest image digest for registry=ghcr.io image=onedr0p/readarr-develop tag=0.3.18.2411: 404, message='Not Found', url=URL('https://ghcr.io/v2/onedr0p/readarr-develop/manifests/0.3.18.2411')
[2024/04/23 23:19:37] (ERROR) ContainerImagesService.check_update():36 - [EFAULT] Unable to retrieve latest image digest for registry=ghcr.io image=onedr0p/readarr-develop tag=0.3.21.2475: 404, message='Not Found', url=URL('https://ghcr.io/v2/onedr0p/readarr-develop/manifests/0.3.21.2475')
[2024/04/23 23:19:38] (ERROR) ContainerImagesService.check_update():36 - [EFAULT] Unable to retrieve latest image digest for registry=ghcr.io image=cloudnative-pg/cloudnative-pg tag=1.20.0: 404, message='Not Found', url=URL('https://ghcr.io/v2/cloudnative-pg/cloudnative-pg/manifests/1.20.0')
[2024/04/23 23:19:39] (ERROR) ContainerImagesService.check_update():36 - [EFAULT] Unable to retrieve latest image digest for registry=ghcr.io image=onedr0p/readarr-develop tag=0.3.23.2506: 404, message='Not Found', url=URL('https://ghcr.io/v2/onedr0p/readarr-develop/manifests/0.3.23.2506')
[2024/04/23 23:19:46] (ERROR) ContainerImagesService.check_update():36 - [EFAULT] Unable to retrieve latest image digest for registry=registry.k8s.io image=sig-storage/csi-node-driver-registrar tag=v2.8.0: Timed out waiting for a response
[2024/04/23 23:24:42] (ERROR) ContainerImagesService.check_update():36 - [EFAULT] Unable to retrieve latest image digest for registry=registry.k8s.io image=sig-storage/livenessprobe tag=v2.10.0: Timed out waiting for a response
[2024/04/23 23:24:42] (ERROR) ContainerImagesService.check_update():36 - [EFAULT] Unable to retrieve latest image digest for registry=ghcr.io image=onedr0p/readarr-develop tag=0.3.20.2452: 404, message='Not Found', url=URL('https://ghcr.io/v2/onedr0p/readarr-develop/manifests/0.3.20.2452')
[2024/04/23 23:29:44] (ERROR) ContainerImagesService.check_update():36 - [EFAULT] Unable to retrieve latest image digest for registry=registry.k8s.io image=sig-storage/smbplugin tag=v1.11.0: Timed out waiting for a response
[2024/04/23 23:34:46] (ERROR) ContainerImagesService.check_update():36 - [EFAULT] Unable to retrieve latest image digest for registry=registry.k8s.io image=sig-storage/nfsplugin tag=v4.4.0: Timed out waiting for a response
[2024/04/23 23:50:16] (DEBUG) UsageService.start():63 - Scheduled next run in 57217 seconds
[2024/04/24 00:08:57] (INFO) middlewared.terminate():2039 - Terminating
[2024/04/24 00:09:12] (DEBUG) VMService.teardown_guest_vmemory():51 - Giving back guest memory to ARC: 115964116992
[2024/04/24 00:09:12] (DEBUG) VMService.teardown_guest_vmemory():51 - Giving back guest memory to ARC: 124554051584
[2024/04/24 00:09:12] (DEBUG) VMService.teardown_guest_vmemory():51 - Giving back guest memory to ARC: 133143986176
[2024/04/24 00:09:17] (DEBUG) middlewared.__event_system_shutdown():145 - VM(s) stopped successfully
[2024/04/24 00:09:17] (DEBUG) EtcService.generate():429 - No new changes for /etc/nginx/nginx.conf
[2024/04/24 00:10:27] (ERROR) middlewared.__terminate():2065 - Failed to terminate vm
Traceback (most recent call last):
File "/usr/lib/python3.11/asyncio/tasks.py", line 490, in wait_for
return fut.result()
^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/plugins/vm/lifecycle.py", line 103, in terminate
async with SHUTDOWN_LOCK:
File "/usr/lib/python3.11/asyncio/locks.py", line 15, in __aenter__
await self.acquire()
File "/usr/lib/python3.11/asyncio/locks.py", line 114, in acquire
await fut
asyncio.exceptions.CancelledError
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 2063, in __terminate
await asyncio.wait_for(self.create_task(service.terminate()), timeout)
File "/usr/lib/python3.11/asyncio/tasks.py", line 492, in wait_for
raise exceptions.TimeoutError() from exc
TimeoutError
[2024/04/24 00:11:56] (INFO) middlewared.__init__():898 - Starting TrueNAS-SCALE-24.04.0 middleware
[2024/04/24 00:11:56] (DEBUG) middlewared._console_write():1171 - Failed to write to console: OSError(5, 'Input/output error')
[2024/04/24 00:11:56] (DEBUG) middlewared._console_write():1171 - Failed to write to console: OSError(5, 'Input/output error')
[2024/04/24 00:11:57] (DEBUG) middlewared._console_write():1171 - Failed to write to console: OSError(5, 'Input/output error')
[2024/04/24 00:12:01] (DEBUG) middlewared.setup():41 - Setting timezone to 'Europe/Oslo'
[2024/04/24 00:12:01] (DEBUG) middlewared.generate_keys():213 - Generating SSH host keys
[2024/04/24 00:12:11] (INFO) RDMAService._get_link_choices():67 - Fetching RDMA link netdev choices
[2024/04/24 00:12:11] (DEBUG) middlewared.setup():25 - Certificate setup for System complete
[2024/04/24 00:12:11] (DEBUG) middlewared.__plugins_setup():1063 - All plugins loaded
[2024/04/24 00:12:11] (DEBUG) middlewared.__initialize():2033 - Accepting connections
[2024/04/24 00:12:11] (INFO) DiskService.log_disk_info():75 - Found disks: {'nvme0n1': {'serial': 'S463NF0M524905Y', 'lunid': '0025385591b3e6d7'}, 'sdb': {'serial': '4ZG500YV', 'lunid': '5000cca2a6c245f8'}, 'sda': {'serial': '4ZG4XHKV', 'lunid': '5000cca2a6c23c7a'}, 'sdd': {'serial': '4BHLWKRH', 'lunid': '5000cca2b6d6adaf'}, 'sdc': {'serial': 'ZDH7ZDX2', 'lunid': '5000c500c3647fba'}, 'sde': {'serial': 'ZDH7YGF1', 'lunid': '5000c500c363d147'}, 'sdf': {'serial': 'ZDH7YTTK', 'lunid': '5000c500c3631d1d'}, 'sdg': {'serial': 'ZDH7YRMH', 'lunid': '5000c500c3640b5d'}, 'sdh': {'serial': 'ZGY5YXPE', 'lunid': '5000c500c363b983'}, 'nvme1n1': {'serial': 'S466NX0K828006E', 'lunid': '0025385881b2bf7b'}, 'nvme2n1': {'serial': '50026B7686965837', 'lunid': '00000000000000000026b76869658375'}}
[2024/04/24 00:12:11] (DEBUG) PoolService.import_on_boot():396 - Creating '/data/zfs' (if it doesnt already exist)
[2024/04/24 00:12:11] (DEBUG) PoolService.import_on_boot():403 - Creating '/data/zfs/zpool.cache' (if it doesnt already exist)
[2024/04/24 00:12:11] (DEBUG) PoolService.import_on_boot_impl():226 - Importing 'pool2' with guid: '18035793806657409897'
[2024/04/24 00:12:12] (DEBUG) EtcService.generate():429 - No new changes for /etc/shadow
[2024/04/24 00:12:12] (DEBUG) EtcService.generate():429 - No new changes for /etc/group
[2024/04/24 00:12:12] (DEBUG) EtcService.generate():429 - No new changes for /etc/passwd
[2024/04/24 00:12:12] (DEBUG) EtcService.generate():429 - No new changes for /etc/shadow
[2024/04/24 00:12:12] (DEBUG) EtcService.generate():429 - No new changes for /etc/sudoers
[2024/04/24 00:12:12] (DEBUG) EtcService.generate():429 - No new changes for /etc/aliases
[2024/04/24 00:12:12] (DEBUG) EtcService.generate():429 - No new changes for /etc/netdata/exporting.conf
[2024/04/24 00:12:12] (DEBUG) EtcService.generate():429 - No new changes for /etc/netdata/python.d/smart_log.conf
[2024/04/24 00:12:12] (DEBUG) EtcService.generate():429 - No new changes for /etc/fstab
[2024/04/24 00:12:12] (DEBUG) EtcService.generate():429 - No new changes for /etc/default/keyboard
[2024/04/24 00:12:12] (DEBUG) EtcService.generate():429 - No new changes for /etc/vconsole.conf
[2024/04/24 00:12:12] (DEBUG) EtcService.generate():429 - No new changes for /etc/openldap/ldap.conf
[2024/04/24 00:12:12] (DEBUG) EtcService.generate():429 - No new changes for /etc/nslcd.conf
[2024/04/24 00:12:12] (DEBUG) EtcService.generate():429 - No new changes for /etc/dhcp/dhclient.conf
[2024/04/24 00:12:21] (DEBUG) EtcService.generate():429 - No new changes for /etc/nfs.conf.d/local.conf
[2024/04/24 00:12:21] (DEBUG) EtcService.generate():429 - No new changes for /etc/default/rpcbind
[2024/04/24 00:12:21] (DEBUG) EtcService.generate():429 - No new changes for /etc/idmapd.conf
[2024/04/24 00:12:21] (DEBUG) EtcService.generate():429 - No new changes for /etc/pam.d/common-account
[2024/04/24 00:12:21] (DEBUG) EtcService.generate():429 - No new changes for /etc/pam.d/common-auth
[2024/04/24 00:12:21] (DEBUG) EtcService.generate():429 - No new changes for /etc/pam.d/common-password
[2024/04/24 00:12:21] (DEBUG) EtcService.generate():429 - No new changes for /etc/pam.d/common-session-noninteractive
[2024/04/24 00:12:21] (DEBUG) EtcService.generate():429 - No new changes for /etc/pam.d/common-session
[2024/04/24 00:12:21] (DEBUG) EtcService.generate():429 - No new changes for /etc/security/pam_winbind.conf
[2024/04/24 00:12:21] (DEBUG) EtcService.generate():429 - No new changes for /etc/pam.d/middleware
[2024/04/24 00:12:21] (DEBUG) EtcService.generate():429 - No new changes for /etc/proftpd/proftpd.conf
[2024/04/24 00:12:21] (DEBUG) EtcService.generate():429 - No new changes for /etc/default/kdump-tools
[2024/04/24 00:12:21] (DEBUG) EtcService.generate():429 - No new changes for /etc/sysctl.d/tunables.conf
[2024/04/24 00:12:21] (DEBUG) EtcService.generate():429 - No new changes for /etc/default/smartmontools
[2024/04/24 00:12:22] (DEBUG) EtcService.generate():429 - No new changes for /etc/keepalived/keepalived.conf
[2024/04/24 00:12:22] (DEBUG) EtcService.generate():429 - No new changes for /etc/motd
[2024/04/24 00:12:22] (DEBUG) EtcService.generate():429 - No new changes for /etc/nscd.conf
[2024/04/24 00:12:22] (DEBUG) middlewared.render_body():35 - UPSMON: monpwd field empty, upsmon will fail to start.
[2024/04/24 00:12:23] (DEBUG) EtcService.generate():429 - No new changes for /etc/smb4.conf
[2024/04/24 00:12:23] (DEBUG) EtcService.generate():429 - No new changes for /etc/snmp/snmpd.conf
[2024/04/24 00:12:23] (DEBUG) EtcService.generate():429 - No new changes for /etc/syslog-ng/syslog-ng.conf
[2024/04/24 00:12:23] (DEBUG) EtcService.generate():429 - No new changes for /etc/syslog-ng/conf.d/tndestinations.conf
[2024/04/24 00:12:23] (DEBUG) EtcService.generate():429 - No new changes for /etc/syslog-ng/conf.d/tnfilters.conf
[2024/04/24 00:12:23] (DEBUG) EtcService.generate():429 - No new changes for /etc/syslog-ng/conf.d/tnaudit.conf
[2024/04/24 00:12:23] (DEBUG) EtcService.generate():429 - No new changes for /etc/pam.d/sshd
[2024/04/24 00:12:23] (DEBUG) EtcService.generate():429 - No new changes for /etc/chrony/chrony.conf
[2024/04/24 00:12:23] (DEBUG) PoolService.import_on_boot_impl():270 - SUCCESS importing 'pool2' with guid: '18035793806657409897'
[2024/04/24 00:12:23] (DEBUG) EtcService.generate():429 - No new changes for /etc/pykmip/pykmip.conf
[2024/04/24 00:12:23] (DEBUG) PoolService.unlock_on_boot_impl():316 - SUCCESS unlocking encrypted dataset(s) (if any) for 'pool2'
[2024/04/24 00:12:23] (DEBUG) PoolService.import_on_boot_impl():226 - Importing 'apps' with guid: '17164552839078562837'
[2024/04/24 00:12:26] (DEBUG) EtcService.generate():429 - No new changes for /etc/hosts
[2024/04/24 00:12:27] (INFO) InterfaceService.configure():21 - Configuring interface 'enp5s0'
[2024/04/24 00:12:27] (DEBUG) InterfaceService.configure():106 - enp5s0: adding 192.168.10.11/255.255.255.0
[2024/04/24 00:12:27] (DEBUG) InterfaceService.configure():106 - enp5s0: adding 192.168.10.10/255.255.255.0
[2024/04/24 00:12:27] (INFO) InterfaceService.sync():1725 - Interfaces in database: enp5s0
[2024/04/24 00:12:27] (DEBUG) EtcService.generate():429 - No new changes for /etc/dhcp/dhclient.conf
[2024/04/24 00:12:27] (INFO) RouteService.sync():83 - Adding IPv4 default route to 192.168.10.1
[2024/04/24 00:12:28] (DEBUG) EtcService.generate():429 - No new changes for /etc/exports
[2024/04/24 00:12:28] (DEBUG) EtcService.generate():429 - No new changes for /etc/nginx/nginx.conf
[2024/04/24 00:12:28] (DEBUG) EtcService.generate():429 - No new changes for /etc/ssh/sshd_config
[2024/04/24 00:12:30] (DEBUG) PoolService.import_on_boot_impl():270 - SUCCESS importing 'apps' with guid: '17164552839078562837'
[2024/04/24 00:12:34] (DEBUG) PoolService.unlock_on_boot_impl():316 - SUCCESS unlocking encrypted dataset(s) (if any) for 'apps'
[2024/04/24 00:12:34] (DEBUG) PoolService.import_on_boot():431 - Calling pool.post_import
[2024/04/24 00:12:34] (WARNING) SystemDatasetService.sysdataset_path():110 - /var/db/system: mountpoint not found
[2024/04/24 00:12:37] (DEBUG) EtcService.generate():429 - No new changes for /etc/smb4.conf
[2024/04/24 00:12:38] (DEBUG) SMBService.reg_globals():89 - Failed to retrieve global share config from registry
[2024/04/24 00:12:38] (DEBUG) EtcService.generate():429 - No new changes for /etc/netdata/netdata.conf
[2024/04/24 00:12:38] (DEBUG) EtcService.generate():429 - No new changes for /etc/netdata/charts.d/exclude_netdata.conf
[2024/04/24 00:12:38] (ERROR) ServiceService._restart():306 - Service 'idmap' not running after restart
[2024/04/24 00:12:38] (DEBUG) EtcService.generate():429 - No new changes for /etc/group
[2024/04/24 00:12:38] (DEBUG) EtcService.generate():429 - No new changes for /etc/passwd
[2024/04/24 00:12:38] (DEBUG) EtcService.generate():429 - No new changes for /etc/shadow
[2024/04/24 00:12:38] (DEBUG) EtcService.generate():429 - No new changes for /etc/sudoers
[2024/04/24 00:12:38] (DEBUG) EtcService.generate():429 - No new changes for /etc/aliases
[2024/04/24 00:12:39] (DEBUG) EtcService.generate():429 - No new changes for /etc/smb4.conf
[2024/04/24 00:12:39] (DEBUG) EtcService.generate():429 - No new changes for /etc/avahi/avahi-daemon.conf
[2024/04/24 00:12:41] (DEBUG) EtcService.generate():429 - No new changes for /etc/scst.conf
[2024/04/24 00:12:41] (DEBUG) EtcService.generate():429 - No new changes for /etc/scst.env
[2024/04/24 00:12:41] (DEBUG) EtcService.generate():429 - No new changes for /etc/initiators.allow
[2024/04/24 00:12:41] (DEBUG) EtcService.generate():429 - No new changes for /etc/initiators.deny
[2024/04/24 00:12:42] (DEBUG) EtcService.generate():429 - No new changes for /etc/proftpd/proftpd.conf
[2024/04/24 00:12:43] (DEBUG) EtcService.generate():429 - No new changes for /etc/nfs.conf.d/local.conf
[2024/04/24 00:12:43] (DEBUG) EtcService.generate():429 - No new changes for /etc/default/rpcbind
[2024/04/24 00:12:43] (DEBUG) EtcService.generate():429 - No new changes for /etc/idmapd.conf
[2024/04/24 00:12:43] (DEBUG) EtcService.generate():429 - No new changes for /etc/exports
[2024/04/24 00:12:45] (DEBUG) PoolService.import_on_boot():433 - Finished calling pool.post_import
[2024/04/24 00:12:45] (DEBUG) EtcService.generate():429 - No new changes for /etc/avahi/avahi-daemon.conf
[2024/04/24 00:12:47] (DEBUG) middlewared._setup_periodic_tasks():1076 - Setting up periodic task alert.flush_alerts to run every 3600 seconds
[2024/04/24 00:12:47] (DEBUG) middlewared._setup_periodic_tasks():1076 - Setting up periodic task alert.process_alerts to run every 60 seconds
[2024/04/24 00:12:47] (DEBUG) middlewared._setup_periodic_tasks():1076 - Setting up periodic task auditbackend._AuditBackendService__lifecycle_cleanup to run every 86400 seconds
[2024/04/24 00:12:47] (DEBUG) middlewared._setup_periodic_tasks():1076 - Setting up periodic task auth.twofactor.remove_expired_secrets to run every 86400 seconds
[2024/04/24 00:12:47] (DEBUG) middlewared._setup_periodic_tasks():1076 - Setting up periodic task certificate.renew_certs to run every 86400 seconds
[2024/04/24 00:12:47] (DEBUG) middlewared._setup_periodic_tasks():1076 - Setting up periodic task certificateauthority.crl_generation to run every 86400 seconds
[2024/04/24 00:12:47] (DEBUG) middlewared._setup_periodic_tasks():1076 - Setting up periodic task chart.release.periodic_chart_releases_update_checks to run every 86400 seconds
[2024/04/24 00:12:47] (DEBUG) middlewared._setup_periodic_tasks():1076 - Setting up periodic task kerberos.keytab.check_updated_keytab to run every 3600 seconds
[2024/04/24 00:12:47] (DEBUG) middlewared._setup_periodic_tasks():1076 - Setting up periodic task kmip.sync_keys to run every 86400 seconds
[2024/04/24 00:12:47] (DEBUG) middlewared._setup_periodic_tasks():1076 - Setting up periodic task mail.send_mail_queue to run every 600 seconds
[2024/04/24 00:12:47] (DEBUG) middlewared._setup_periodic_tasks():1076 - Setting up periodic task pool.dataset.sync_db_keys to run every 86400 seconds
[2024/04/24 00:12:47] (DEBUG) middlewared._setup_periodic_tasks():1076 - Setting up periodic task reporting.netdataweb_expire to run every 600 seconds
[2024/04/24 00:12:47] (DEBUG) middlewared._setup_periodic_tasks():1076 - Setting up periodic task service.check_deprecated_services to run every 3600 seconds
[2024/04/24 00:12:47] (DEBUG) middlewared._setup_periodic_tasks():1076 - Setting up periodic task smb.sharesec.check_share_info_tdb to run every 3600 seconds
[2024/04/24 00:12:47] (DEBUG) middlewared._setup_periodic_tasks():1076 - Setting up periodic task truecommand.health_check to run every 1800 seconds
[2024/04/24 00:12:47] (DEBUG) middlewared._setup_periodic_tasks():1076 - Setting up periodic task vmware.delete_pending_snapshots to run every 10800.0 seconds
[2024/04/24 00:12:47] (DEBUG) middlewared._setup_periodic_tasks():1076 - Setting up periodic task zettarepl.flush_state to run every 3600 seconds
[2024/04/24 00:12:47] (DEBUG) EtcService.generate():429 - No new changes for /etc/avahi/avahi-daemon.conf
[2024/04/24 00:12:47] (DEBUG) EtcService.generate():429 - No new changes for /etc/default/libvirt-guests
[2024/04/24 00:12:49] (DEBUG) EtcService.generate():429 - No new changes for /etc/nginx/nginx.conf
[2024/04/24 00:12:49] (WARNING) middlewared.plugins.service_.services.base.systemd_unit():171 - nginx reload failed with code 1: 'nginx.service is not active, cannot reload.\n'
[2024/04/24 00:12:51] (DEBUG) EtcService.generate():429 - No new changes for /etc/nginx/nginx.conf
[2024/04/24 00:12:51] (WARNING) middlewared.plugins.service_.services.base.systemd_unit():171 - nginx reload failed with code 1: 'nginx.service is not active, cannot reload.\n'
[2024/04/24 00:12:54] (DEBUG) EtcService.generate():429 - No new changes for /etc/wsdd.conf
[2024/04/24 00:12:55] (DEBUG) EtcService.generate():429 - No new changes for /etc/containerd.env
[2024/04/24 00:14:14] (ERROR) ContainerImagesService.check_update():36 - [EFAULT] Unable to retrieve latest image digest for registry=ghcr.io image=onedr0p/readarr-develop tag=0.3.20.2452: 404, message='Not Found', url=URL('https://ghcr.io/v2/onedr0p/readarr-develop/manifests/0.3.20.2452')
[2024/04/24 00:14:17] (ERROR) ContainerImagesService.check_update():36 - [EFAULT] Unable to retrieve latest image digest for registry=ghcr.io image=onedr0p/readarr-develop tag=0.3.23.2506: 404, message='Not Found', url=URL('https://ghcr.io/v2/onedr0p/readarr-develop/manifests/0.3.23.2506')
[2024/04/24 00:14:22] (ERROR) ContainerImagesService.check_update():36 - [EFAULT] Unable to retrieve latest image digest for registry=ghcr.io image=onedr0p/readarr-develop tag=0.3.18.2411: 404, message='Not Found', url=URL('https://ghcr.io/v2/onedr0p/readarr-develop/manifests/0.3.18.2411')
1 Like
I also see nginx is not happy.
I have since booted in to 23.10 but it was something like:
systemctl status nginx
Error bad upstream port 5009
For us to investigate fully we really need a bug report which includes a debug from the 24.04.0 setup. Appreciate the log extract but we need the additional info.
2 Likes
I think that can be fixed!
Is there a writedown somewhere on how to set it up properly for you?
Or is it just setting verbose logging and looking for a specific log file? 
If you don’t have WebUI access but can access through ssh you can generate the debug from the cli:
system debug > debugfilename-date.tgz
The debug command downloads a system debug file to the home directory of the logged in admin user. Specify the file name and extension for the debug file in the command string. Debug files are usually .tgz files but you can use the extension of your choice.
1 Like
Are you running IPv6 in any VMs?
There’s a bug that has been found.
1 Like
Yes sir, confirmed haos uses ipv6… Will just disabling ipv6 on host fix this?
Sorry for not having a debug log yet… Haven’t had any time to take it down 
I made a debug file now, is it safe to put here?
We generally encourage users to safeguard their debug file as it could potentially expose sensitive information. In Jira we offer a private file upload for this reason. In this case the fix has already been made and will be included in a 24.04.1 release, so we don’t need the debug to investigate.
As far as a current workaround fix, what we have in the release notes @Captain_Morgan linked is either to wait for 24.04.1 before upgrading or remove the IPv6 bind addresses in Cobia, before upgrading to 24.04.0. If you’ve reverted to 23.10, then removing the IPv6 addresses from any VMs then upgrading back to 24.10.04 should solve the problem.
1 Like
Oh ok, i tried net.ipv6 .conf.all.disable_ipv6 but it didn’t seem to do much… Disabling ipv6 in haos is apperently a bit of a pain in the butt