Troubleshooting strange behavior TrueNAS Scale

Hello! First time here. Been runnig TrueNas Scale for almody six months and been veey happy so far. I use a HP MicroServer Gen8, Pentium CPU, 12GB Ecc ram, SSD for OS and 4x16TB SAS in ZFS1.

I did a ”press and hold powerbutton” shutdown a couple of weeks ago, not shure if it is releated but I want to mention it. (Was a bit stressed for a massive thunderstorm came out of nowhere and I wanted to unplug the server quickly).

Maybe a week ago, I noticed qbittorrent stating ”missing files” on some torrents. A forced resume in qbittorrent fixed the problem, but mostly tempoary and it kept happening to more and more torrents. One movie was suddenly unable to play in Plex. Some archives was unable to be unrared. Two days ago I could watch any movies at all, and the GUI sign in was unaccessable. Haven’t set up SSH so had to turn of from powerbutton.

After restart the GUI worked and everything seemed to look fine, but Plex still didn’t work, and after a little while the GUI stopped working. Tried this two times and then gave up for the night, planned to continue troubleshooting next day.

Next day everything worked! GUI, resuming torrents and everything looked fine. Plex worked, even the movie that previously couldn’t be played worked. Also unraring.
But later at night, everything still worked but a few torrents started to show missing files. In the GUI everything seems to be working fine.

Don’t really have clue where to start and where to look. Any tips and ideas would be greatly appriciated.

(Qbittorrent is mainlys used for Linux distro and file syc, Plex playes movies I ripped från Blueray and DVD)

Well, for starters ZFS1 is wrong :slight_smile: ZFS is the file system, and you mean RAIDZ1.

Suggested readings are Introduction to ZFS | TrueNAS Community and iX's ZFS Pool Layout White Paper.

I would suggest setting up a SSH terminal since you will have to do some commands… but SCALE apparently has fixed the WebUI’s Shell: up to your preference, though I still suggest SSH.

What’s the output of zpool status? When was the last time you scrubbed? How are you connecting the drives to the motherboard?

Also, I assume you mean 128GB of RAM instead of 12.

1 Like

Thank you very much your reply. Greatly approciated!

Yes of course, RaidZ1. :slight_smile:
Read the documentation when setting it up, think I got the hang of it then, but can take another look.
Will set up SSH and share the output. In the GUI the pool status is OK, which throws me of a bit.

And I meant 12GB and not 128GB… 16GB is max in the G8…

:nerd_face:

Just remembered I missed those questions :nerd_face:

From what I can remeber it is set to auto (by default) and the last scrub was for about a week ago. Will check when I ger back home.

The disks are connected to a ”LSI 6Gbps SAS HBA LSI 9201-8i 9211-8i P20 IT Mode ZFS FreeNAS unRAID”

Here is a trail. Output of sas2flash -list please; anyone betting it’s not flashed in IT mode, gentlemen?

Will come back with the out getting home.

The card states ”IT MODE” when booting.

Is the scrubbing something you have to manually setup? Because right now I can’t remeber doing that.

And once again, thank you for your help.

I know RAIDZ1 isn’t that nice with 16TB drives. I’m looking into building a new system with 8x16TB and runnig RAIDZ2. (But that will hopefully be a lather concern).

Regarding ram, I understand that 12GB ram is on the low end. I use about 7,5TB and have about 35TB free in the pool.

1 Like

Generally it’s something I set up alongside the SMART testing routine, but iirc it has a default task.

Blockquote pool: Pool1
state: ONLINE
scan: scrub repaired 0B in 03:50:17 with 0 errors on Sun Mar 17 03:50:18 2024
config:

    NAME                                      STATE     READ WRITE CKSUM
    Pool1                                     ONLINE       0     0     0
      raidz1-0                                ONLINE       0     0     0
        e9d94262-94cb-4e38-baf8-bfab20455675  ONLINE       0     0     0
        78089819-cbe1-449d-ac10-9e7fe90ca4d8  ONLINE       0     0     0
        3597f5b5-f0e4-4454-a860-cb5e882b3656  ONLINE       0     0     0
        218c3d30-a1bf-45fe-95d6-caf4e61e8e44  ONLINE       0     0     0

errors: No known data errors

pool: boot-pool
state: ONLINE
scan: scrub repaired 0B in 00:01:35 with 0 errors on Tue Apr 16 03:46:36 2024
config:

    NAME        STATE     READ WRITE CKSUM
    boot-pool   ONLINE       0     0     0
      sdd3      ONLINE       0     0     0

errors: No known data errors

sudo sas2flash -list
LSI Corporation SAS2 Flash Utility
Version 20.00.00.00 (2014.09.18)
Copyright (c) 2008-2014 LSI Corporation. All rights reserved

    Adapter Selected is a LSI SAS: SAS2008(B2)

    Controller Number              : 0
    Controller                     : SAS2008(B2)
    PCI Address                    : 00:07:00:00
    SAS Address                    : 500605b-0-044b-93b0
    NVDATA Version (Default)       : 0a.02.00.05
    NVDATA Version (Persistent)    : 0a.02.00.05
    Firmware Product ID            : 0x2213 (IT)
    Firmware Version               : 10.00.03.00
    NVDATA Vendor                  : XIV
    NVDATA Product ID              : SAS9201-8i
    BIOS Version                   : 07.11.00.00
    UEFI BSD Version               : 07.00.00.01
    FCODE Version                  : N/A
    Board Name                     : SAS9201-8i
    Board Assembly                 : H3-25268-00D
    Board Tracer Number            : SP14346838

    Finished Processing Commands Successfully.
    Exiting SAS2Flash.

It’s actually in IT mode, albeit an old version. LSI 9300-xx Firmware Update | TrueNAS Community

If I tell you HostPath Validation, does it ring any bells?

1 Like

Wow, version 10 firmware on a SAS2 HBA? That’s insanely old, definitely update that ASAP.

That’s actually the wrong one, that’s for SAS3 Gen 1 HBAs using the SAS3008 and related ICs. OP needs to get version P20.00.07.00 from Broadcom.

1 Like

I swear I did read SAS3008.

Thanks for your answers. The problem with the old firmware was only regarding SATA disks, and I’m running SAS. But maybe it would be a good idea to upgrade the firmware anyway. Need to look into how to do it safely.

Regarding ‘HostPath Validation,’ it doesn’t ring any bells, but from what I can tell, it is regarding combining SMB shares and host storage. I only use my TrueNas for SMB sharing, so I don’t think that it is a problem for me.

I have now forced the torrents to resume and then immediately done a forced recheck. Now everyone (except two of them) is working, and no new ones are popping up as missing files. My guess is that there has been some sort of corruption due to forcefully shutting down TrueNas or with Qbittorrent because it lost its storage. Can’t work out why some files wouldn’t work at all or why the web GUI didn’t work…

Oh my god… I can´t belive I´m this stupid. The SAME DAY I starded having big problems, is the same day I had previusly setup backup of my proxmox machines to the same SMB share :face_with_peeking_eye:
I my world that wouldn´t be a problem. But I should have thought about it when the problems started.
Everything stopped working tonight again, and started to think about HostPatch Validation. My gues now is that this might be the cause. Inactivated backup, rebooted Proxmox and will see if I can get everything working again.

Great tip Davvo!

1 Like

Can confirmed that ater I stopped doing snapshots from Proxmox to the same SMMB share everything now works perfect.

2 Likes