Good morning,
I’ve been troubleshooting a sporadic issue where my TrueNAS core system goes out to pasture and hangs. My supermicro box detects this and triggers a WatchDog power cycle of the box and it all comes back online.
Event logs from BMC look like this:
19303 2024/06/15 09:08:17 #0xca Watchdog 2 Timer Interrupt - Assertion
19304 2024/06/15 09:08:52 #0xca Watchdog 2 Power Cycle - Assertion
19305 2024/06/16 01:23:38 #0xca Watchdog 2 Timer Interrupt - Assertion
19306 2024/06/16 01:24:49 #0xca Watchdog 2 Power Cycle - Assertion
19307 2024/06/22 08:04:56 #0xca Watchdog 2 Timer Interrupt - Assertion
19308 2024/06/22 08:05:31 #0xca Watchdog 2 Power Cycle - Assertion
19309 2024/06/23 11:29:53 #0xca Watchdog 2 Timer Interrupt - Assertion
19310 2024/06/23 11:30:28 #0xca Watchdog 2 Power Cycle - Assertion
19311 2024/06/23 23:02:55 #0xca Watchdog 2 Timer Interrupt - Assertion
19312 2024/06/23 23:04:06 #0xca Watchdog 2 Power Cycle - Assertion
19313 2024/06/26 05:11:58 #0xca Watchdog 2 Timer Interrupt - Assertion
19314 2024/06/26 05:13:08 #0xca Watchdog 2 Power Cycle - Assertion
19315 2024/06/26 19:45:43 #0xca Watchdog 2 Timer Interrupt - Assertion
19316 2024/06/26 19:46:18 #0xca Watchdog 2 Power Cycle - Assertion
19317 2024/06/27 05:27:30 #0xca Watchdog 2 Timer Interrupt - Assertion
19318 2024/06/27 05:28:05 #0xca Watchdog 2 Power Cycle - Assertion
19319 2024/06/27 23:53:11 #0xca Watchdog 2 Timer Interrupt - Assertion
19320 2024/06/27 23:54:21 #0xca Watchdog 2 Power Cycle - Assertion
19321 2024/06/28 12:17:16 #0xca Watchdog 2 Timer Interrupt - Assertion
19322 2024/06/28 12:18:27 #0xca Watchdog 2 Power Cycle - Assertion
19323 2024/06/28 21:18:52 #0xca Watchdog 2 Timer Interrupt - Assertion
19324 2024/06/28 21:19:27 #0xca Watchdog 2 Power Cycle - Assertion
19325 2024/06/29 13:58:12 #0xca Watchdog 2 Timer Interrupt - Assertion
19326 2024/06/29 13:58:47 #0xca Watchdog 2 Power Cycle - Assertion
19327 2024/06/30 00:11:24 #0xca Watchdog 2 Timer Interrupt - Assertion
19328 2024/06/30 00:12:35 #0xca Watchdog 2 Power Cycle - Assertion
19329 2024/06/30 17:55:37 #0xca Watchdog 2 Timer Interrupt - Assertion
19330 2024/06/30 17:56:12 #0xca Watchdog 2 Power Cycle - Assertion
19331 2024/07/01 03:59:33 #0xca Watchdog 2 Timer Interrupt - Assertion
19332 2024/07/01 04:00:44 #0xca Watchdog 2 Power Cycle - Assertion
19333 2024/07/02 00:02:18 #0xca Watchdog 2 Timer Interrupt - Assertion
19334 2024/07/02 00:03:28 #0xca Watchdog 2 Power Cycle - Assertion
19335 2024/07/02 16:20:37 #0xca Watchdog 2 Timer Interrupt - Assertion
19336 2024/07/02 16:21:12 #0xca Watchdog 2 Power Cycle - Assertion
19337 2024/08/07 04:14:52 #0xca Watchdog 2 Timer Interrupt - Assertion
19338 2024/08/07 04:15:27 #0xca Watchdog 2 Power Cycle - Assertion
19339 2024/08/07 22:36:06 #0xca Watchdog 2 Timer Interrupt - Assertion
19340 2024/08/07 22:37:17 #0xca Watchdog 2 Power Cycle - Assertion
19341 2024/08/08 11:57:49 #0xca Watchdog 2 Timer Interrupt - Assertion
19342 2024/08/08 11:58:59 #0xca Watchdog 2 Power Cycle - Assertion
19343 2024/08/09 20:21:40 #0xca Watchdog 2 Timer Interrupt - Assertion
19344 2024/08/09 20:22:16 #0xca Watchdog 2 Power Cycle - Assertion
19345 2024/08/10 17:19:25 #0xca Watchdog 2 Timer Interrupt - Assertion
19346 2024/08/10 17:20:00 #0xca Watchdog 2 Power Cycle - Assertion
19347 2024/08/11 19:12:25 #0xca Watchdog 2 Timer Interrupt - Assertion
19348 2024/08/11 19:13:35 #0xca Watchdog 2 Power Cycle - Assertion
19349 2024/08/14 04:03:45 #0xca Watchdog 2 Timer Interrupt - Assertion
19350 2024/08/14 04:04:55 #0xca Watchdog 2 Power Cycle - Assertion
19351 2024/08/18 16:16:47 #0xca Watchdog 2 Timer Interrupt - Assertion
19352 2024/08/18 16:17:22 #0xca Watchdog 2 Power Cycle - Assertion
19353 2024/08/19 17:57:05 #0xca Watchdog 2 Timer Interrupt - Assertion
19354 2024/08/19 17:57:40 #0xca Watchdog 2 Power Cycle - Assertion
19355 2024/08/20 10:53:15 #0xca Watchdog 2 Timer Interrupt - Assertion
19356 2024/08/20 10:54:26 #0xca Watchdog 2 Power Cycle - Assertion
19357 2024/09/26 21:31:07 #0xca Watchdog 2 Timer Interrupt - Assertion
19358 2024/09/26 21:32:17 #0xca Watchdog 2 Power Cycle - Assertion
19359 2024/09/27 06:50:23 #0xca Watchdog 2 Timer Interrupt - Assertion
19360 2024/09/27 06:51:34 #0xca Watchdog 2 Power Cycle - Assertion
19361 2024/09/27 17:16:14 #0xca Watchdog 2 Timer Interrupt - Assertion
19362 2024/09/27 17:16:49 #0xca Watchdog 2 Power Cycle - Assertion
We occasionally detect a drive that is starting to show some SMART errors and replace it and the power cycles seem to become less common, but we are still seeing at least 2-3 crash/reboots a month, sometimes multiple times a day for a few days in a row. We have not detected any RAM or CPU issues on the system and the power supplies seem happy (dual redundant.) System is in a rack with adequate cooling in a data center environment. I’m at a lost, almost ready to toss the whole thing away or to start swapping all hardware (CPU, RAM, Motherboard, Network, PSUs.) Any ideas or thoughts of where else I can look?
Motherboard is a supermicro: X10DRH-C
CPU: Intel Xeon E5-2683 v3 x2
RAM: 128GB
Network: Mellanox ConnectX-4 dual port 100G
Drives:
In server chassis:
30x Seagate Exos 18TB SAS 12G ST18000NM004J
cache 6x Samsung SSD 870 2TB
In external Seagate Exos E 5U84
84x Seagate Exos 16TB SAS 12G ST16000NM002G
zpool status -v
root@truenas[~]# zpool status -v
pool: boot-pool
state: ONLINE
status: Some supported and requested features are not enabled on the pool.
The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
the pool may no longer be accessible by software that does not support
the features. See zpool-features(7) for details.
scan: scrub repaired 0B in 00:00:11 with 0 errors on Tue Sep 24 03:45:11 2024
config:
NAME STATE READ WRITE CKSUM
boot-pool ONLINE 0 0 0
da0p2 ONLINE 0 0 0
errors: No known data errors
pool: sv1-big2-pool
state: ONLINE
status: Some supported and requested features are not enabled on the pool.
The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
the pool may no longer be accessible by software that does not support
the features. See zpool-features(7) for details.
scan: resilvered 15.6T in 8 days 08:12:02 with 0 errors on Fri Aug 16 19:27:59 2024
config:
NAME STATE READ WRITE CKSUM
sv1-big2-pool ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gptid/4a25ecde-c1b5-11ec-a138-0cc47a6ce2b6 ONLINE 0 0 0
gptid/4b0062d8-c1b5-11ec-a138-0cc47a6ce2b6 ONLINE 0 0 0
gptid/4b08a011-c1b5-11ec-a138-0cc47a6ce2b6 ONLINE 0 0 0
gptid/4b5596e5-c1b5-11ec-a138-0cc47a6ce2b6 ONLINE 0 0 0
gptid/3e517807-55b2-11ef-8c65-0cc47a6ce2b6 ONLINE 0 0 0
gptid/4af74cd6-c1b5-11ec-a138-0cc47a6ce2b6 ONLINE 0 0 0
gptid/4c10c652-c1b5-11ec-a138-0cc47a6ce2b6 ONLINE 0 0 0
gptid/4ce24858-c1b5-11ec-a138-0cc47a6ce2b6 ONLINE 0 0 0
gptid/4c07614f-c1b5-11ec-a138-0cc47a6ce2b6 ONLINE 0 0 0
gptid/4c824a00-c1b5-11ec-a138-0cc47a6ce2b6 ONLINE 0 0 0
raidz2-1 ONLINE 0 0 0
gptid/4b4bf3c9-c1b5-11ec-a138-0cc47a6ce2b6 ONLINE 0 0 0
da16p2 ONLINE 0 0 0
gptid/4bb3e307-c1b5-11ec-a138-0cc47a6ce2b6 ONLINE 0 0 0
gptid/4baa1806-c1b5-11ec-a138-0cc47a6ce2b6 ONLINE 0 0 0
gptid/4c411b4d-c1b5-11ec-a138-0cc47a6ce2b6 ONLINE 0 0 0
gptid/4b7c42d3-c1b5-11ec-a138-0cc47a6ce2b6 ONLINE 0 0 0
gptid/4f132521-c1b5-11ec-a138-0cc47a6ce2b6 ONLINE 0 0 0
gptid/4ee2bb98-c1b5-11ec-a138-0cc47a6ce2b6 ONLINE 0 0 0
gptid/5096407d-c1b5-11ec-a138-0cc47a6ce2b6 ONLINE 0 0 0
gptid/516c241c-c1b5-11ec-a138-0cc47a6ce2b6 ONLINE 0 0 0
raidz2-2 ONLINE 0 0 0
gptid/50499944-c1b5-11ec-a138-0cc47a6ce2b6 ONLINE 0 0 0
gptid/4ff6c3e2-c1b5-11ec-a138-0cc47a6ce2b6 ONLINE 0 0 0
gptid/5025741d-c1b5-11ec-a138-0cc47a6ce2b6 ONLINE 0 0 0
gptid/508c8fa0-c1b5-11ec-a138-0cc47a6ce2b6 ONLINE 0 0 0
gptid/d18f2b49-ca38-11ec-8e97-0cc47a6ce2b6 ONLINE 0 0 0
gptid/50fd85e2-c1b5-11ec-a138-0cc47a6ce2b6 ONLINE 0 0 0
gptid/518b5b84-c1b5-11ec-a138-0cc47a6ce2b6 ONLINE 0 0 0
gptid/51118835-c1b5-11ec-a138-0cc47a6ce2b6 ONLINE 0 0 0
gptid/a4ca9f3c-f1f2-11ee-9531-0cc47a6ce2b6 ONLINE 0 0 0
gptid/51ac77cd-c1b5-11ec-a138-0cc47a6ce2b6 ONLINE 0 0 0
raidz2-3 ONLINE 0 0 0
gptid/05d31937-15de-11ee-ab2b-0cc47a6ce2b6 ONLINE 0 0 0
gptid/058fc9b4-15de-11ee-ab2b-0cc47a6ce2b6 ONLINE 0 0 0
gptid/058f1d5f-15de-11ee-ab2b-0cc47a6ce2b6 ONLINE 0 0 0
gptid/058eeb02-15de-11ee-ab2b-0cc47a6ce2b6 ONLINE 0 0 0
gptid/051fb90f-15de-11ee-ab2b-0cc47a6ce2b6 ONLINE 0 0 0
gptid/052626f1-15de-11ee-ab2b-0cc47a6ce2b6 ONLINE 0 0 0
gptid/04a1f458-15de-11ee-ab2b-0cc47a6ce2b6 ONLINE 0 0 0
gptid/05d351cd-15de-11ee-ab2b-0cc47a6ce2b6 ONLINE 0 0 0
gptid/05d2b050-15de-11ee-ab2b-0cc47a6ce2b6 ONLINE 0 0 0
gptid/0616f092-15de-11ee-ab2b-0cc47a6ce2b6 ONLINE 0 0 0
raidz2-4 ONLINE 0 0 0
gptid/06164749-15de-11ee-ab2b-0cc47a6ce2b6 ONLINE 0 0 0
gptid/058f565f-15de-11ee-ab2b-0cc47a6ce2b6 ONLINE 0 0 0
gptid/05d279ec-15de-11ee-ab2b-0cc47a6ce2b6 ONLINE 0 0 0
gptid/05d2dfdb-15de-11ee-ab2b-0cc47a6ce2b6 ONLINE 0 0 0
gptid/0616bb96-15de-11ee-ab2b-0cc47a6ce2b6 ONLINE 0 0 0
gptid/05d38996-15de-11ee-ab2b-0cc47a6ce2b6 ONLINE 0 0 0
gptid/06607a65-15de-11ee-ab2b-0cc47a6ce2b6 ONLINE 0 0 0
gptid/06eaa79f-15de-11ee-ab2b-0cc47a6ce2b6 ONLINE 0 0 0
gptid/06a5bf8f-15de-11ee-ab2b-0cc47a6ce2b6 ONLINE 0 0 0
gptid/085f64df-15de-11ee-ab2b-0cc47a6ce2b6 ONLINE 0 0 0
raidz2-5 ONLINE 0 0 0
gptid/08b27610-15de-11ee-ab2b-0cc47a6ce2b6 ONLINE 0 0 0
gptid/08600d71-15de-11ee-ab2b-0cc47a6ce2b6 ONLINE 0 0 0
gptid/085f2c07-15de-11ee-ab2b-0cc47a6ce2b6 ONLINE 0 0 0
gptid/08b2a935-15de-11ee-ab2b-0cc47a6ce2b6 ONLINE 0 0 0
gptid/085ebc11-15de-11ee-ab2b-0cc47a6ce2b6 ONLINE 0 0 0
gptid/08b83171-15de-11ee-ab2b-0cc47a6ce2b6 ONLINE 0 0 0
gptid/085f9cb5-15de-11ee-ab2b-0cc47a6ce2b6 ONLINE 0 0 0
gptid/085e0a6a-15de-11ee-ab2b-0cc47a6ce2b6 ONLINE 0 0 0
gptid/085ef47a-15de-11ee-ab2b-0cc47a6ce2b6 ONLINE 0 0 0
gptid/085e844a-15de-11ee-ab2b-0cc47a6ce2b6 ONLINE 0 0 0
raidz2-6 ONLINE 0 0 0
gptid/08b2e33f-15de-11ee-ab2b-0cc47a6ce2b6 ONLINE 0 0 0
gptid/08255e90-15de-11ee-ab2b-0cc47a6ce2b6 ONLINE 0 0 0
gptid/085fd6d6-15de-11ee-ab2b-0cc47a6ce2b6 ONLINE 0 0 0
gptid/09388116-15de-11ee-ab2b-0cc47a6ce2b6 ONLINE 0 0 0
gptid/093278bd-15de-11ee-ab2b-0cc47a6ce2b6 ONLINE 0 0 0
gptid/09df0b3d-15de-11ee-ab2b-0cc47a6ce2b6 ONLINE 0 0 0
gptid/0b6ff63b-15de-11ee-ab2b-0cc47a6ce2b6 ONLINE 0 0 0
gptid/0aa8ce8b-15de-11ee-ab2b-0cc47a6ce2b6 ONLINE 0 0 0
gptid/0b75375f-15de-11ee-ab2b-0cc47a6ce2b6 ONLINE 0 0 0
gptid/0b75a660-15de-11ee-ab2b-0cc47a6ce2b6 ONLINE 0 0 0
raidz2-7 ONLINE 0 0 0
gptid/0b51c727-15de-11ee-ab2b-0cc47a6ce2b6 ONLINE 0 0 0
gptid/0b4a8a5d-15de-11ee-ab2b-0cc47a6ce2b6 ONLINE 0 0 0
gptid/0aade3c1-15de-11ee-ab2b-0cc47a6ce2b6 ONLINE 0 0 0
gptid/0b7036d4-15de-11ee-ab2b-0cc47a6ce2b6 ONLINE 0 0 0
gptid/0b5b5cf9-15de-11ee-ab2b-0cc47a6ce2b6 ONLINE 0 0 0
gptid/0b5488cc-15de-11ee-ab2b-0cc47a6ce2b6 ONLINE 0 0 0
gptid/0b60a866-15de-11ee-ab2b-0cc47a6ce2b6 ONLINE 0 0 0
gptid/0b60e3af-15de-11ee-ab2b-0cc47a6ce2b6 ONLINE 0 0 0
gptid/0b756db1-15de-11ee-ab2b-0cc47a6ce2b6 ONLINE 0 0 0
gptid/0b4ac5f1-15de-11ee-ab2b-0cc47a6ce2b6 ONLINE 0 0 0
raidz2-8 ONLINE 0 0 0
gptid/0a724e2a-15de-11ee-ab2b-0cc47a6ce2b6 ONLINE 0 0 0
gptid/0beb88be-15de-11ee-ab2b-0cc47a6ce2b6 ONLINE 0 0 0
gptid/0bfe845b-15de-11ee-ab2b-0cc47a6ce2b6 ONLINE 0 0 0
gptid/0df40ddd-15de-11ee-ab2b-0cc47a6ce2b6 ONLINE 0 0 0
gptid/0df94f7b-15de-11ee-ab2b-0cc47a6ce2b6 ONLINE 0 0 0
gptid/0df988f2-15de-11ee-ab2b-0cc47a6ce2b6 ONLINE 0 0 0
gptid/0df912f9-15de-11ee-ab2b-0cc47a6ce2b6 ONLINE 0 0 0
gptid/0dcb73da-15de-11ee-ab2b-0cc47a6ce2b6 ONLINE 0 0 0
gptid/0dc72a1e-15de-11ee-ab2b-0cc47a6ce2b6 ONLINE 0 0 0
gptid/0df9fd8d-15de-11ee-ab2b-0cc47a6ce2b6 ONLINE 0 0 0
raidz2-9 ONLINE 0 0 0
gptid/0df9c4cb-15de-11ee-ab2b-0cc47a6ce2b6 ONLINE 0 0 0
gptid/0dfa3635-15de-11ee-ab2b-0cc47a6ce2b6 ONLINE 0 0 0
gptid/0df39b1c-15de-11ee-ab2b-0cc47a6ce2b6 ONLINE 0 0 0
gptid/0df3d47b-15de-11ee-ab2b-0cc47a6ce2b6 ONLINE 0 0 0
gptid/0df447a3-15de-11ee-ab2b-0cc47a6ce2b6 ONLINE 0 0 0
gptid/0d08a414-15de-11ee-ab2b-0cc47a6ce2b6 ONLINE 0 0 0
gptid/0dcba70f-15de-11ee-ab2b-0cc47a6ce2b6 ONLINE 0 0 0
gptid/0e3e3d28-15de-11ee-ab2b-0cc47a6ce2b6 ONLINE 0 0 0
gptid/0e3dae9a-15de-11ee-ab2b-0cc47a6ce2b6 ONLINE 0 0 0
gptid/0f0d4a8b-15de-11ee-ab2b-0cc47a6ce2b6 ONLINE 0 0 0
raidz2-10 ONLINE 0 0 0
gptid/1072182a-15de-11ee-ab2b-0cc47a6ce2b6 ONLINE 0 0 0
gptid/0fad67a5-15de-11ee-ab2b-0cc47a6ce2b6 ONLINE 0 0 0
gptid/107253a7-15de-11ee-ab2b-0cc47a6ce2b6 ONLINE 0 0 0
gptid/0ffadd7b-15de-11ee-ab2b-0cc47a6ce2b6 ONLINE 0 0 0
gptid/10730965-15de-11ee-ab2b-0cc47a6ce2b6 ONLINE 0 0 0
gptid/105fec3d-15de-11ee-ab2b-0cc47a6ce2b6 ONLINE 0 0 0
gptid/10718eff-15de-11ee-ab2b-0cc47a6ce2b6 ONLINE 0 0 0
gptid/1072931a-15de-11ee-ab2b-0cc47a6ce2b6 ONLINE 0 0 0
gptid/10596df3-15de-11ee-ab2b-0cc47a6ce2b6 ONLINE 0 0 0
gptid/1072ca30-15de-11ee-ab2b-0cc47a6ce2b6 ONLINE 0 0 0
cache
gptid/4de0c547-c1b5-11ec-a138-0cc47a6ce2b6 ONLINE 0 0 0
gptid/4de740ec-c1b5-11ec-a138-0cc47a6ce2b6 ONLINE 0 0 0
da27p1 ONLINE 0 0 0
gptid/52878259-c1b5-11ec-a138-0cc47a6ce2b6 ONLINE 0 0 0
gptid/52bfe785-c1b5-11ec-a138-0cc47a6ce2b6 ONLINE 0 0 0
gptid/52c788e3-c1b5-11ec-a138-0cc47a6ce2b6 ONLINE 0 0 0
spares
gptid/105faef3-15de-11ee-ab2b-0cc47a6ce2b6 AVAIL
gptid/0fa13d71-15de-11ee-ab2b-0cc47a6ce2b6 AVAIL
gptid/10734a0d-15de-11ee-ab2b-0cc47a6ce2b6 AVAIL
gptid/105f778f-15de-11ee-ab2b-0cc47a6ce2b6 AVAIL
errors: No known data errors
zpool list
root@truenas[~]# zpool list
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
boot-pool 216G 5.11G 211G - - 2% 2% 1.00x ONLINE -
sv1-big2-pool 1.62P 1.13P 500T - - 25% 69% 1.00x ONLINE /mnt
zfs list
root@truenas[~]# zfs list
NAME USED AVAIL REFER MOUNTPOINT
boot-pool 5.11G 204G 24K none
boot-pool/ROOT 5.05G 204G 24K none
boot-pool/ROOT/13.0-U5.1 363K 204G 1.66G /
boot-pool/ROOT/13.0-U6.1 5.05G 204G 1.54G /
boot-pool/ROOT/Initial-Install 1K 204G 1.20G legacy
boot-pool/ROOT/default 190K 204G 2.09G legacy
sv1-big2-pool 880T 381T 457K /mnt/sv1-big2-pool
sv1-big2-pool/.system 1.05G 381T 873M legacy
sv1-big2-pool/.system/configs-a617fea1951f4cf0a2768ef9d432a087 121M 381T 121M legacy
sv1-big2-pool/.system/cores 384K 1024M 384K legacy
sv1-big2-pool/.system/rrd-a617fea1951f4cf0a2768ef9d432a087 64.1M 381T 64.1M legacy
sv1-big2-pool/.system/samba4 3.78M 381T 1.66M legacy
sv1-big2-pool/.system/services 384K 381T 384K legacy
sv1-big2-pool/.system/syslog-a617fea1951f4cf0a2768ef9d432a087 8.58M 381T 8.58M legacy
sv1-big2-pool/.system/webui 384K 381T 384K legacy
sv1-big2-pool/big2 880T 381T 880T /mnt/sv1-big2-pool/big2
sv1-big2-pool/iocage 5.56G 381T 11.6M /mnt/sv1-big2-pool/iocage
sv1-big2-pool/iocage/download 913M 381T 439K /mnt/sv1-big2-pool/iocage/download
sv1-big2-pool/iocage/download/12.3-RELEASE 403M 381T 403M /mnt/sv1-big2-pool/iocage/download/12.3-RELEASE
sv1-big2-pool/iocage/download/12.4-RELEASE 253M 381T 253M /mnt/sv1-big2-pool/iocage/download/12.4-RELEASE
sv1-big2-pool/iocage/download/13.2-RELEASE 257M 381T 257M /mnt/sv1-big2-pool/iocage/download/13.2-RELEASE
sv1-big2-pool/iocage/images 384K 381T 384K /mnt/sv1-big2-pool/iocage/images
sv1-big2-pool/iocage/jails 744M 381T 402K /mnt/sv1-big2-pool/iocage/jails
sv1-big2-pool/iocage/jails/bacula-fd 744M 381T 457K /mnt/sv1-big2-pool/iocage/jails/bacula-fd
sv1-big2-pool/iocage/jails/bacula-fd/root 743M 381T 1.25G /mnt/sv1-big2-pool/iocage/jails/bacula-fd/root
sv1-big2-pool/iocage/log 393K 381T 393K /mnt/sv1-big2-pool/iocage/log
sv1-big2-pool/iocage/releases 3.93G 381T 439K /mnt/sv1-big2-pool/iocage/releases
sv1-big2-pool/iocage/releases/12.3-RELEASE 2.28G 381T 402K /mnt/sv1-big2-pool/iocage/releases/12.3-RELEASE
sv1-big2-pool/iocage/releases/12.3-RELEASE/root 2.28G 381T 2.28G /mnt/sv1-big2-pool/iocage/releases/12.3-RELEASE/root
sv1-big2-pool/iocage/releases/12.4-RELEASE 866M 381T 402K /mnt/sv1-big2-pool/iocage/releases/12.4-RELEASE
sv1-big2-pool/iocage/releases/12.4-RELEASE/root 865M 381T 865M /mnt/sv1-big2-pool/iocage/releases/12.4-RELEASE/root
sv1-big2-pool/iocage/releases/13.2-RELEASE 822M 381T 402K /mnt/sv1-big2-pool/iocage/releases/13.2-RELEASE
sv1-big2-pool/iocage/releases/13.2-RELEASE/root 821M 381T 821M /mnt/sv1-big2-pool/iocage/releases/13.2-RELEASE/root
sv1-big2-pool/iocage/templates 384K 381T 384K /mnt/sv1-big2-pool/iocage/templates