TrueNas server crashing

TrueNAS General
1

Hi All,

I have TrueNAS 25.10.4 - Goldeye, 3*8TB. In another server with Docker I have Immich pointing to the share folders (shared via NSF). All is working fine during the day but then at 3am an Immich process starts and then the TrueNAS Server crashes and reboots (hard reset) like 3-4 times. When I wake up in the morning I see the pool degraded, I ran a scrub and all is back online and no errors. See below the immich process in docker and the reboots in TrueNAS server from Thursday to Sunday. Have you experienced something like that?

Immich Process:

sudo docker logs immich_server --since “2026-07-26T02:45:00” --until “2026-07-26T06:50:00”
[Nest] 7 - 07/26/2026, 3:00:00 AM LOG [Microservices:IntegrityService] Checking for out of date untracked file reports…
[Nest] 7 - 07/26/2026, 3:00:00 AM LOG [Microservices:IntegrityService] Scanning for untracked files…
[Nest] 7 - 07/26/2026, 3:00:09 AM LOG [Microservices:IntegrityService] Queued untracked check of 10000 file(s) (10000 so far)
[Nest] 7 - 07/26/2026, 3:00:15 AM LOG [Microservices:IntegrityService] Queued untracked check of 10000 file(s) (20000 so far)
[Nest] 7 - 07/26/2026, 3:00:19 AM LOG [Microservices:IntegrityService] Queued untracked check of 7927 file(s) (27927 so far)
[Nest] 7 - 07/26/2026, 3:00:43 AM LOG [Microservices:IntegrityService] Queued untracked check of 10000 file(s) (37927 so far)
[Nest] 7 - 07/26/2026, 3:01:19 AM LOG [Microservices:IntegrityService] Queued untracked check of 10000 file(s) (47927 so far)
[Nest] 7 - 07/26/2026, 3:01:58 AM LOG [Microservices:IntegrityService] Queued untracked check of 10000 file(s) (57927 so far)
[Nest] 7 - 07/26/2026, 3:02:36 AM LOG [Microservices:IntegrityService] Queued untracked check of 10000 file(s) (67927 so far)
[Nest] 7 - 07/26/2026, 3:03:08 AM LOG [Microservices:IntegrityService] Queued untracked check of 8683 file(s) (76610 so far)
[Nest] 7 - 07/26/2026, 3:03:08 AM LOG [Microservices:IntegrityService] Checking file checksums… (will run for up to 1.00 hours or until 100.00% of assets are processed)
[Nest] 7 - 07/26/2026, 3:03:08 AM LOG [Microservices:IntegrityService] Processing assets from 2026-07-06T21:01:14.796Z
[Nest] 7 - 07/26/2026, 3:03:12 AM WARN [Microservices:IntegrityService] Failed to process a file: Error: EIO: i/o error, read
[Nest] 7 - 07/26/2026, 3:03:12 AM WARN [Microservices:IntegrityService] Failed to process a file: Error: EIO: i/o error, read
[Nest] 7 - 07/26/2026, 3:03:12 AM WARN [Microservices:IntegrityService] Failed to process a file: Error: EIO: i/o error, read
[Nest] 7 - 07/26/2026, 3:03:12 AM WARN [Microservices:IntegrityService] Failed to process a file: Error: EIO: i/o error, read
[Nest] 7 - 07/26/2026, 3:03:12 AM WARN [Microservices:IntegrityService] Failed to process a file: Error: EIO: i/o error, read
[Nest] 7 - 07/26/2026, 3:03:12 AM WARN [Microservices:IntegrityService] Failed to process a file: Error: EIO: i/o error, read
[Nest] 7 - 07/26/2026, 3:03:12 AM WARN [Microservices:IntegrityService] Failed to process a file: Error: EIO: i/o error, read
[Nest] 7 - 07/26/2026, 3:03:12 AM WARN [Microservices:IntegrityService] Failed to process a file: Error: EIO: i/o error, read
[Nest] 7 - 07/26/2026, 3:03:12 AM WARN [Microservices:IntegrityService] Failed to process a file: Error: EIO: i/o error, read
[Nest] 7 - 07/26/2026, 3:03:18 AM LOG [Microservices:IntegrityService] Processed 100 files so far… (avg. 100.57 ms/asset, 0.40% of all assets)
[Nest] 7 - 07/26/2026, 3:03:19 AM WARN [Microservices:IntegrityService] Failed to process a file: Error: EIO: i/o error, read
[Nest] 7 - 07/26/2026, 3:03:19 AM WARN [Microservices:IntegrityService] Failed to process a file: Error: EIO: i/o error, read
[Nest] 7 - 07/26/2026, 3:03:19 AM WARN [Microservices:IntegrityService] Failed to process a file: Error: EIO: i/o error, read
[Nest] 7 - 07/26/2026, 3:03:19 AM WARN [Microservices:IntegrityService] Failed to process a file: Error: EIO: i/o error, read
[Nest] 7 - 07/26/2026, 3:03:19 AM WARN [Microservices:IntegrityService] Failed to process a file: Error: EIO: i/o error, read
[Nest] 7 - 07/26/2026, 3:03:19 AM WARN [Microservices:IntegrityService] Failed to process a file: Error: EIO: i/o error, read
[Nest] 7 - 07/26/2026, 3:03:19 AM WARN [Microservices:IntegrityService] Failed to process a file: Error: EIO: i/o error, read
[Nest] 7 - 07/26/2026, 3:03:19 AM WARN [Microservices:IntegrityService] Failed to process a file: Error: EIO: i/o error, read
[Nest] 7 - 07/26/2026, 3:03:19 AM WARN [Microservices:IntegrityService] Failed to process a file: Error: EIO: i/o error, read
[Nest] 7 - 07/26/2026, 3:03:19 AM WARN [Microservices:IntegrityService] Failed to process a file: Error: EIO: i/o error, read
[Nest] 7 - 07/26/2026, 3:03:19 AM WARN [Microservices:IntegrityService] Failed to process a file: Error: EIO: i/o error, read
[Nest] 7 - 07/26/2026, 3:03:19 AM WARN [Microservices:IntegrityService] Failed to process a file: Error: EIO: i/o error, read
[Nest] 7 - 07/26/2026, 3:03:20 AM WARN [Microservices:IntegrityService] Failed to process a file: Error: EIO: i/o error, read
[Nest] 7 - 07/26/2026, 3:03:20 AM WARN [Microservices:IntegrityService] Failed to process a file: Error: EIO: i/o error, read
[Nest] 7 - 07/26/2026, 3:03:20 AM WARN [Microservices:IntegrityService] Failed to process a file: Error: EIO: i/o error, read
[Nest] 7 - 07/26/2026, 3:03:27 AM LOG [Microservices:IntegrityService] Processed 200 files so far… (avg. 97.74 ms/asset, 0.79% of all assets)
[Nest] 7 - 07/26/2026, 3:03:42 AM LOG [Microservices:IntegrityService] Processed 300 files so far… (avg. 113.95 ms/asset, 1.19% of all assets)
[Nest] 7 - 07/26/2026, 3:03:45 AM LOG [Microservices:IntegrityService] Processed 400 files so far… (avg. 92.81 ms/asset, 1.58% of all assets)
[Nest] 7 - 07/26/2026, 3:03:50 AM LOG [Microservices:IntegrityService] Processed 500 files so far… (avg. 84.50 ms/asset, 1.98% of all assets)

Reboots:

21 922b1c0d3d5044df9d570afa454d45ad Thu 2026-07-23 03:02:58 EDT Thu 2026-07-23 03:11:10 EDT
-20 4e9d1035039c4f4fa6c1b4c957ee46d9 Thu 2026-07-23 03:12:34 EDT Thu 2026-07-23 03:17:33 EDT
-19 0cb7e291c4d3429ba28d08f2f1894fc0 Thu 2026-07-23 03:19:25 EDT Thu 2026-07-23 03:24:22 EDT
-18 d36f5886f4044da483660d05035578c0 Thu 2026-07-23 03:25:37 EDT Fri 2026-07-24 02:50:00 EDT
-17 ec9dd54892034b61967bfb7337abbe98 Fri 2026-07-24 03:01:13 EDT Fri 2026-07-24 03:10:20 EDT
-16 85a0b17954d8408392c673ea88489b77 Fri 2026-07-24 03:11:46 EDT Fri 2026-07-24 03:16:19 EDT
-15 00a82bd0ba304e59aaf98519c3e50b79 Fri 2026-07-24 03:17:56 EDT Fri 2026-07-24 03:24:01 EDT
-14 e355d71e8928463cae53d1c3e83f2e80 Fri 2026-07-24 03:25:16 EDT Sat 2026-07-25 07:23:50 EDT
-13 89e22bd1e1a8403e8205e14641411cbf Sat 2026-07-25 07:25:46 EDT Sat 2026-07-25 08:31:01 EDT
-12 ae4bf0b10f8f4bdb9b9247ecb4b2ef2a Sat 2026-07-25 09:17:09 EDT Sun 2026-07-26 03:06:43 EDT
-11 cb194eaf16424ee9878c0c6497303a1b Sun 2026-07-26 03:08:40 EDT Sun 2026-07-26 03:12:42 EDT
-10 4f0f2644825c472aaf596bd7bf670276 Sun 2026-07-26 03:14:14 EDT Sun 2026-07-26 03:18:25 EDT
-9 5d804f0901af4a5a981bdf154449eb73 Sun 2026-07-26 03:19:50 EDT Sun 2026-07-26 03:24:37 EDT
-8 c5301a5cc3ac4ceb8ccd8ddec4597118 Sun 2026-07-26 03:25:51 EDT Sun 2026-07-26 03:30:01 EDT
-7 0e8627dba2704e35a4464a2a9057498a Sun 2026-07-26 03:31:37 EDT Sun 2026-07-26 12:17:01 EDT
-6 053c858981a34c62a2b56edac8b6f5d5 Sun 2026-07-26 12:28:58 EDT Sun 2026-07-26 14:50:33 EDT
-5 243d294ff4b14cd19ab92b2346e1c71f Sun 2026-07-26 15:02:21 EDT Mon 2026-07-27 03:03:16 EDT
-4 6c9d3a2c65c64a24af2de690eb7b2e16 Mon 2026-07-27 03:06:30 EDT Mon 2026-07-27 03:07:17 EDT
-3 3ff70375022d4ef78cd8b85f38fbb0ab Mon 2026-07-27 03:11:37 EDT Mon 2026-07-27 03:12:26 EDT
-2 64b144aba00e41b29dbfba8f7d667e23 Mon 2026-07-27 03:17:15 EDT Mon 2026-07-27 03:20:14 EDT
-1 49ce2a49ed274b368ea58f3caaec8902 Mon 2026-07-27 03:24:58 EDT Mon 2026-07-27 03:29:15 EDT
0 60d55f53f6df4eb38fd85054a16a693c Mon 2026-07-27 03:31:07 EDT Wed 2026-07-29 16:03:54 EDT

2

Full hardware details could help. Guessing Raid-Z1? Usual first two checks are RAM and CPU stability. Since you only listed 3 drives, guessing attached to motherboard and not a HBA card involved.

Maybe try changing Immich to do it’s scheduled thing when you can watch servers and see what is happening on TrueNAS at that time.

3

Hi,

AMD Ryzen 5 5600G with Radeon Graphics 32GB RAM. ASUS PRIME B450M A-II. Yes Raid-Z1 connected to the board now. I had connected to the HBA Card because I thought that was the problem but the same issue with the disks connected to the SATA Board.

I started the immich process manually (check sum process. Check Checksums: Verifies that asset checksums match their source files to detect corruption) and now the pool is degraded (it says thousand of files) although no hard reset.

Odd thing is that when I took one of those “corrupted files” I can get a sha256sum from the truenas server and from the ubuntu pc (Same checksum) and I can copy from the pool to the /tmp folder. It is like:

Immich Integrity Check

↓

Reads thousands of files rapidly

↓

Storage path glitches

↓

EIO returned to application

↓

ZFS records checksum/data errors

↓

Files get added to permanent error list

Could it be the ram? I’m planning to run a memory test. Or just I need to recreate the pool from scratch? Because it looks like the files are good but for some reason when that process starts the pool got degraded.

sha256sum file

4

this is the pool v status command after the pool got degraded:

	pool: tankstate: DEGRADEDstatus: One or more devices has experienced an error resulting in datacorruption.  Applications may be affected.action: Restore the file in question if possible.  Otherwise restore theentire pool from backup.see: scan: scrub repaired 0B in 00:48:28 with 0 errors on Mon Jul 27 07:05:46 2026config:

NAME                                      STATE     READ WRITE CKSUM
tank                                      DEGRADED     0     0     0
  raidz1-0                                DEGRADED     0     0     0
    32664cc0-22d6-4b6f-88dc-8cc27a1c22cf  ONLINE       0     0 87.2K
    0ca22c33-9e9b-48fd-a17c-1d5afb07a7b7  ONLINE       0     0 87.3K
    f111ae3c-5084-4298-a6da-5f997f5585f2  DEGRADED     0     0 87.3K  too many errors

errors: Permanent errors have been detected in the following files:

5

Critical

Pool tank state is DEGRADED: One or more devices has experienced an error resulting in data corruption. Applications may be affected.

The following devices are not healthy:

  • Disk WDC_WD80EFPX-68C4ZN0 WD-1F0BBMJU
6

Hi everyone,

Any feedback would be really appreciated it! I will ran a memory test today. I don’t want to destroy the pool and start over again because I think the issue will be back again. Not sure if it is bios or ram or sata controller.

7

Since this is a gaming board, make sure ALL overclocking, including any automatic overclocking, is permanently disabled in the BIOS. It may even be necessary to underclock to achieve stability.

Next, check your RAM with Passmark memtest86 with multiple passes over an extended period.

Last, check your power supply. Consumer power supplies skimp on the 5v rail to prioritize the CPU & GPU 12v rails, and your disks may be short on power.

8

Re memory, extended period means memtest86+ (+ is the FOSS one) for up to 5 days. Some memory errors take longer to show, but 5 days is a good compromise. If this is memory, since it dies so reliably every day, it may show up way earlier in the test.

Testing RAM is a pita and the reason people recommend ECC builds. Not because ZFS requires ECC. Because trying to rule out faulty consumer RAM is tedious.

That said you can absolutely run on your hardware, you “just” need to figure out whether RAM can be trusted.

Also look at cooling you have set up for the board. Some coolers don’t blow “down” and so the VRMs get hot, which annoyingly might not show during a ram test but does show as instability when running a real software load.

9

Hi Everyone,

I did the memory test. it failed within minutes (two sticks and one stick=FAIL test has aborted due to many errors Errors: 10000). Odd thing is with the second stick it does not boot (fans spin, but never reaches the ASUS logo). At this point not sure if it is a problem with the motherboard. Thanks again for the feedback. Maybe I need to think about a new build based on these results. I can try testing the memory in another system.

10

@winnielinnie strikes again :frowning: