We have an HP DL380 Gen9 running TrueNAS CORE 13.0-U6.8, connected externally to an Asus S2044J JBOD via dual SFF-8644 for expansion. We have one ZFS pool (POOL01) composed of 5 RAIDZ2 vdevs comprised of 52 attached disks across the internal chassis and external enclosure.
We use this TN system as a nearline for our production media storage as well as a storage host for our Proxmox VM images. Over the weekend, I believe while running a nearline restore job, TN threw an error:
* Failed to check for alert HasUpdate: Traceback (most recent call last): File
“/usr/local/lib/python3.9/site-packages/middlewared/plugins/alert.py”, line
740, in __run_source alerts = (await alert_source.check()) or [] File
“/usr/local/lib/python3.9/site-packages/middlewared/alert/base.py”, line 212,
in check return await self.middleware.run_in_thread(self.check_sync) File
“/usr/local/lib/python3.9/site-packages/middlewared/main.py”, line 1161, in
run_in_thread return await self.run_in_executor(self.thread_pool_executor,
method, *args, **kwargs) File “/usr/local/lib/python3.9/site-
packages/middlewared/main.py”, line 1158, in run_in_executor return await
loop.run_in_executor(pool, functools.partial(method, *args, **kwargs)) File
“/usr/local/lib/python3.9/concurrent/futures/thread.py”, line 58, in run
result = self.fn(*self.args, **self.kwargs) File
“/usr/local/lib/python3.9/site-packages/middlewared/alert/source/update.py”,
line 67, in check_sync path =
self.middleware.call_sync(“update.get_update_location”) File
“/usr/local/lib/python3.9/site-packages/middlewared/main.py”, line 1301, in
call_sync return self.run_coroutine(methodobj(*prepared_call.args)) File
“/usr/local/lib/python3.9/site-packages/middlewared/main.py”, line 1341, in
run_coroutine return fut.result() File
“/usr/local/lib/python3.9/concurrent/futures/_base.py”, line 439, in result
return self.__get_result() File
“/usr/local/lib/python3.9/concurrent/futures/_base.py”, line 391, in
__get_result raise self._exception File “/usr/local/lib/python3.9/site-
packages/middlewared/plugins/update.py”, line 412, in get_update_location
os.chmod(path, 0o755) OSError: [Errno 6] Device not configured:
‘/var/db/system/update’
When I came in this morning, my VMs were throwing errors because they lost connection to their storage host, so it was clear the pool was offline. TrueNAS was not responding to zpool status -v. I rebooted both chassis, after which TrueNAS booted and showed in the Dashboard that the pool was offline still. I then shut down just the DL380 chassis, pulled power, and inspected the LSI 9300-8e, thinking it was an issue with the SAS connection between the DL380 and S2044J chassis. All looked good physically. I reseated both ends of the SAS cables.
After rebooting again, TrueNAS seemed to import the pool normally, with a few extra checks in the boot process. The Dashboard loaded and showed it was online. After a moment I ran zpool status -v which hanged. Running camcontrol devlist -v confirmed that it was at least seeing all the disks across the two chassis. When I refreshed the Dashboard page, it was blank, but I could still see the TrueNAS sidebar and could navigate elsewhere.
I’m not sure what to do next in troubleshooting, but running gstat -dp in a new terminal window showed what looks like some kind of resilvering activity, with multiple drives being read from and two drives being written to:
dT: 1.005s w: 1.000s
L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps ms/d %busy Name
0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da0
0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da1
0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da2
0 1 1 4 21.0 0 0 0.0 0 0 0.0 2.1| da3
0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da4
1 6 6 24 17.1 0 0 0.0 0 0 0.0 10.2| da5
3 206 206 160915 13.7 0 0 0.0 0 0 0.0 96.6| da6
3 202 202 165942 14.2 0 0 0.0 0 0 0.0 97.3| da7
3 206 206 164633 13.7 0 0 0.0 0 0 0.0 96.5| da8
3 212 212 164135 13.4 0 0 0.0 0 0 0.0 97.0| da9
3 185 185 177196 16.1 0 0 0.0 0 0 0.0 100.2| da10
3 218 218 166977 13.1 0 0 0.0 0 0 0.0 97.3| da11
1 342 0 0 0.0 342 176392 2.5 0 0 0.0 86.6| da12
0 622 0 0 0.0 622 164099 0.8 0 0 0.0 50.3| da13
0 10 10 40 13.0 0 0 0.0 0 0 0.0 12.9| da14
0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da15
0 2 2 8 16.2 0 0 0.0 0 0 0.0 3.2| da16
0 3 3 12 15.7 0 0 0.0 0 0 0.0 4.7| da17
0 9 9 36 11.5 0 0 0.0 0 0 0.0 9.3| da18
0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da19
0 11 11 44 14.2 0 0 0.0 0 0 0.0 14.0| da20
0 6 6 24 12.9 0 0 0.0 0 0 0.0 6.9| da21
0 9 9 36 14.6 0 0 0.0 0 0 0.0 11.5| da22
0 2 2 8 11.8 0 0 0.0 0 0 0.0 2.3| da23
0 6 6 24 12.3 0 0 0.0 0 0 0.0 6.3| da24
0 2 2 8 27.7 0 0 0.0 0 0 0.0 5.5| da25
0 6 6 24 15.3 0 0 0.0 0 0 0.0 9.1| da26
0 3 3 12 17.1 0 0 0.0 0 0 0.0 5.1| da27
0 4 4 16 12.2 0 0 0.0 0 0 0.0 4.9| da28
0 3 3 12 10.7 0 0 0.0 0 0 0.0 3.2| da29
0 4 4 16 17.7 0 0 0.0 0 0 0.0 7.1| da30
0 3 3 12 16.5 0 0 0.0 0 0 0.0 4.9| da31
0 2 2 8 16.6 0 0 0.0 0 0 0.0 3.3| da32
0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da33
0 14 14 68 11.2 0 0 0.0 0 0 0.0 12.4| da34
0 15 15 72 12.9 0 0 0.0 0 0 0.0 17.5| da35
0 6 6 36 11.9 0 0 0.0 0 0 0.0 7.1| da36
0 5 5 20 16.7 0 0 0.0 0 0 0.0 8.7| da37
0 10 10 52 11.0 0 0 0.0 0 0 0.0 10.9| da38
0 14 14 87 7.8 0 0 0.0 0 0 0.0 10.8| da39
0 15 15 91 9.1 0 0 0.0 0 0 0.0 13.5| da40
0 1 1 4 22.6 0 0 0.0 0 0 0.0 2.2| da41
0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da42
0 6 6 24 13.4 0 0 0.0 0 0 0.0 8.0| da43
0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da44
0 1 1 4 9.6 0 0 0.0 0 0 0.0 1.0| da45
0 1 1 4 18.5 0 0 0.0 0 0 0.0 1.8| da46
1 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da47
0 6 6 24 17.2 0 0 0.0 0 0 0.0 10.2| da48
0 11 11 44 10.7 0 0 0.0 0 0 0.0 11.7| da49
0 10 10 40 9.6 0 0 0.0 0 0 0.0 9.5| da50
0 6 6 24 14.9 0 0 0.0 0 0 0.0 8.9| da51
Any help would be greatly appreciated!