Okay I'm at a loss and need to ask for help

TrueNAS General
,
1

I have tried everything I can think of and I’m at a loss here and figure now’s the time to ask for help.

truenas CE/core running on an hp dl 380 g6, 192gb ram, I’ve tried multiple different IT mode HBa’s with and without sas expander, multiple different jbod chassis, and even tried putting 2 test drives directly inside the hp server with the front drive panel directly behind an IT mode HBA so I know for a fact there’s no raid pass-through shenanigans..

nothing I do gets the system to behave any differently than the problem I’m trying to figure out and fix.

everything, everything except full disk wiping, throws IO errors error5, drives are confirmed as 512/4k formatting which all of them are the same and my reading says that should be fine… 4 different enclosures tested if you include the internal drive backplane, 3 different HBA’s, 6 different sas cables, 1 expander/no expander, everything throws IO errors error 5, quick wiping, pool creation regardless of vdev type (even trying a single disk stripe)…..so my testing seems to indicate its nothing as the issue source yet somehow still something.

its not the HBA, its not the cables, its not the disk shelves, and though it seems like its the drives, how on earth can 50 drives do exactly the same thing while also working in other systems?

how on earth do I proceed from here to find and fix the problem because the drives work in other systems/projects but I wish to dedicate them to use in truenas

I’m fluent in testing other things, but far from fluent in command-line troubleshooting especially when it comes to drives, please advise and all help is massively appreciated at this point because I’m stumped.

2

Did you check in the HP variant of the IPMI if you get CPU or RAM errors ?

If errors persist, run a memtest and CPU stresstest. At this point it could be anything, Even a defective mobo.

3

never in a million years would I have had that thought, no I have not, I will check that now, thank you so much. that never would have occurred to me to be a consideration.

4

no errors to speak of what so ever.

5

If you had errors and ran both TrueNAS Core and TrueNAS Scale, that points to hardware issues. Two different OS versions, FreeBSD and Linux. Have you tried moving the HBA slot? Do you have a bad slot? What is ‘error 5’?

Full hardware listing of your system. We would look at what specific HBA you are running and make you check the firmware, etc. Ensure you are not overheating the HBAs and have excellent cooling air flow over the heat sinks.

Check your email or DMs for info on taking the tutorial by TrueNAS-Bot. After that, you should have more rights to post images or links.

We can only go off the detailed info you post.

1 Like
6

I thought I’d included all the needed info in my original post, sorry about that.

system: Hp DL 380 G6 - propriertary motherboard to the system

ram: 12x 16gb ddr3 ecc 1067

cpus: dual intel xeon x5660’s

networking: addon dual port intel 10g sfp+ card x520-da2 with intel optics

drive speeds while full-disk wiping with zeros was 60MBps per disk when wiping 2

error5 was the same thing as the IO errors, it just said “Input/output error 5” but i’ll boot it back up and get the exact error it spat out at me once I finish typing this out with the extra info from the “Joes Rules to Asking for Help” post.

the drives I’m trying to test with, I’m sure most of them are the same but I’m not going to assume here and I’ll add the confirmation with my following reply as well, are all seagate 2tb sas drives, I’ll get the exact model number and add that in my next reply too along with the exact error message.

I’ll also grab the exact model numbers for the hba’s used. so let me send this and then confirm the other bits of info in another reply.

7

You need to test systematically. Start with a small set of drives, a single hba that we can check the model and info using sas2flash or sas3flash. This verifies the firmware version and if it reports ‘IT mode’. Some models are not fit for use with TrueNAS / ZFS. OS version can help with what commands are available. We assume a current, supported TrueNAS Scale unless told otherwise.

Due to the age of your system, it may be wise to boot a live Linux version and run CPU stress tests and, at least 5 passes, of Memtest86 on the RAM. Do you have one of the latest BIOs versions installed on the Motherboard?

8

These are not memory errors.

Most people don’t know how true memory issues show up, so the regurgitating crowd has to utter “bad memory!”.

I have yet to see bad memory create REPEATABLE errors/issues.

Bad memory either crashes the system, produces completely random errors or simply won’t allow it to boot.

9

With all due respect, go away. This is all you do on these forums and it’s childish, shortsighted, and annoying.

Seems that examples of different users with systems that boot up just fine, yet keep hitting I/O errors on reads and writes and keep giving false positives on ZFS checksum errors, is a “repeatable” issue.

Give it a rest already. It takes very little time to run memtests and rule out something. It’s not a big deal. I don’t know why you keep making it into such a hysterical problem that you need to always jump in and lecture other users.

1 Like
10

Can you please copy/paste the error message exactly? Use the </> button in the forum to format it properly.

Also of use would be to see the full sudo smartctl -a /dev/XXX of one of the drives you are having trouble with, obviously replacing X with the corresponding drive letter. Again, use the </> button to aid legibility.

11

working on it, running an exhaustive list of added information in one massive combined aggregate post, pardon the time its taking, but I’m trying to ensure this next reply has anything extra useful I’ve not included thus far due to my quick skimming of information, I had thought I was here prior but clearly I’m thinking of the pfsense forum and so I’m learning fast my assumptions on thinking I knew what and how to add info, were grossly wrong. so…massive exhaustive extra info post incoming, still gathering the info for it though.

12

Exactly, how about proper info, steps, exact errors.

13

I tried to post the replies, but for whatever reason, the sites reply is saying I’m trying to post a link? what? would trying to use the < > options make it trip thinking those are links?

14

ah….found it, nevermind, one of the commands output the smartmontools website.

HBA’s used: LSI 9211-8i flashed to IT modelsi 9200-8e flashed to IT mode

HP sas expander 3g (I cant find a good actual part number to name this thing with but its a 3g sas expander card)I have set the hp server to the highest fan speed bios allows, I think it was called improved cooling mode.

drive model numbers, and confirmed they’re all the same drive model, seagate st2000nx0353

exact error message [EFAULT] [Errno 5] Input/output error

as for the latest reply of testing, I’ve pulled everything except the boot-mirror and 2 test drives out so the only things in the system are 2 drives to test for pool creation with, and a power-backed radian-200 for use as a ZIL which I’ve not even tried mucking with due to the other issues.

I’ll make a bootable USB to run memtest with and report back though I feel like that might be better to run overnight since I’m sure someone’s gonna ask me to test other stuff before that if I am assuming correctly at least. what would you suggest to run for the CPU stress test since I’m woefully out of date on what to use for that, last I tried to was around 2008…so I figure better to ask.

system’s the latest download for truenas CE from the site as of a week ago, though that’s not a version. version is : Current version: 25.10.3 from the system update page

motherboard bios is the latest release before HP EOL’d the things

o.0 sas2flash and sas3flash both return “no lsi card found”

<truenas% sas2flashLSI Corporation SAS2 Flash UtilityVersion 20.00.00.00 (2014.09.18)Copyright (c) 2008-2014 LSI Corporation. All rights reserved

No LSI SAS adapters found! Limited Command Set Available!
Finished Processing Commands Successfully.
Exiting SAS2Flash.

truenas% sas3flashAvago Technologies SAS3 Flash UtilityVersion 16.00.00.00 (2017.05.02)Copyright 2008-2017 Avago Technologies. All rights reserved.

No Avago SAS adapters found! Limited Command Set Available!
Finished Processing Commands Successfully.
Exiting SAS3Flash.

truenas%>

running smartctl on the two drives returns the following info: <truenas_admin@truenas[~]$ sudo smartctl -x /dev/sdd[sudo] password for truenas_admin:smartctl 7.4 2023-08-01 r5530 [x86_64-linux-6.12.33-production+truenas] (local build)Copyright (C) 2002-23, Bruce Allen, Christian Franke, websitename

=== START OF INFORMATION SECTION ===Vendor: SEAGATEProduct: ST2000NX0353Revision: EF02Compliance: SPC-4User Capacity: 2,000,398,934,016 bytes [2.00 TB]Logical block size: 512 bytesPhysical block size: 4096 bytesLU is fully provisionedRotation Rate: 7200 rpmForm Factor: 2.5 inchesLogical Unit id: 0x5000c500998afe2fSerial number: S4611BDX0000E7200SL0Device type: diskTransport protocol: SAS (SPL-4)Local Time is: Sat May 2 09:40:01 2026 PDTSMART support is: Available - device has SMART capability.SMART support is: EnabledTemperature Warning: EnabledRead Cache is: EnabledWriteback Cache is: Disabled

=== START OF READ SMART DATA SECTION ===SMART Health Status: OK

Grown defects during certificationTotal blocks reassigned during formatTotal new blocks reassignedPower on minutes since formatCurrent Drive Temperature: 30 CDrive Trip Temperature: 60 C

Manufactured in week 51 of year 2016Specified cycle count over device lifetime: 10000Accumulated start-stop cycles: 25Specified load-unload count over device lifetime: 300000Accumulated load-unload cycles: 8536Elements in grown defect list: 0

Vendor (Seagate Cache) informationBlocks sent to initiator = 22440Blocks received from initiator = 128000Blocks read from cache and sent to initiator = 4248Number of read and write commands whose size <= segment size = 29Number of read and write commands whose size > segment size = 0

Vendor (Seagate/Hitachi) factory informationnumber of hours powered up = 50665.92number of minutes until next internal SMART test = 49

Error counter log:Errors Corrected by Total Correction Gigabytes TotalECC rereads/ errors algorithm processed uncorrectedfast | delayed rewrites corrected invocations [10^9 bytes] errorsread: 8323 0 0 8323 0 0.011 0write: 0 0 0 0 0 0.069 0

Non-medium error count: 0

[GLTSD (Global Logging Target Save Disable) set. Enable Save with ‘-S on’]No Self-tests have been logged

Background scan results logStatus: scan is activeAccumulated power on time, hours:minutes 50665:55 [3039955 minutes]Number of background scans performed: 352, scan progress: 4.02%Number of background medium scans performed: 352Device does not support General statistics and performance logging

Protocol Specific port log page for SAS SSPrelative target port id = 1generation code = 0number of phys = 1phy identifier = 0attached device type: SAS or SATA deviceattached reason: power onreason: loss of dword synchronizationnegotiated logical link rate: phy enabled; 6 Gbpsattached initiator port: ssp=1 stp=1 smp=1attached target port: ssp=0 stp=0 smp=0SAS address = 0x5000c500998afe2dattached SAS address = 0x5b82a720d2151c00attached phy identifier = 0Invalid DWORD count = 6375Running disparity error count = 6375Loss of DWORD synchronization count = 4Phy reset problem count = 0relative target port id = 2generation code = 0number of phys = 1phy identifier = 1attached device type: no device attachedattached reason: unknownreason: unknownnegotiated logical link rate: phy enabled; unknownattached initiator port: ssp=0 stp=0 smp=0attached target port: ssp=0 stp=0 smp=0SAS address = 0x5000c500998afe2eattached SAS address = 0x0attached phy identifier = 0Invalid DWORD count = 0Running disparity error count = 0Loss of DWORD synchronization count = 0Phy reset problem count = 0

truenas_admin@truenas[~]$ sudo smartctl -x /dev/sdcsmartctl 7.4 2023-08-01 r5530 [x86_64-linux-6.12.33-production+truenas] (local build)Copyright (C) 2002-23, Bruce Allen, Christian Franke, websitename

=== START OF INFORMATION SECTION ===Vendor: SEAGATEProduct: ST2000NX0353Revision: EF02Compliance: SPC-4User Capacity: 2,000,398,934,016 bytes [2.00 TB]Logical block size: 512 bytesPhysical block size: 4096 bytesLU is fully provisionedRotation Rate: 7200 rpmForm Factor: 2.5 inchesLogical Unit id: 0x5000c500991289cbSerial number: S460YDQY0000E715HS3DDevice type: diskTransport protocol: SAS (SPL-4)Local Time is: Sat May 2 09:40:06 2026 PDTSMART support is: Available - device has SMART capability.SMART support is: EnabledTemperature Warning: EnabledRead Cache is: EnabledWriteback Cache is: Disabled

=== START OF READ SMART DATA SECTION ===SMART Health Status: OK

Grown defects during certificationTotal blocks reassigned during formatTotal new blocks reassignedPower on minutes since formatCurrent Drive Temperature: 29 CDrive Trip Temperature: 60 C

Manufactured in week 51 of year 2016Specified cycle count over device lifetime: 10000Accumulated start-stop cycles: 29Specified load-unload count over device lifetime: 300000Accumulated load-unload cycles: 7957Elements in grown defect list: 0

Vendor (Seagate Cache) informationBlocks sent to initiator = 23880Blocks received from initiator = 128000Blocks read from cache and sent to initiator = 4008Number of read and write commands whose size <= segment size = 32Number of read and write commands whose size > segment size = 0

Vendor (Seagate/Hitachi) factory informationnumber of hours powered up = 50633.90number of minutes until next internal SMART test = 49

Error counter log:Errors Corrected by Total Correction Gigabytes TotalECC rereads/ errors algorithm processed uncorrectedfast | delayed rewrites corrected invocations [10^9 bytes] errorsread: 6333 0 0 6333 0 0.012 0write: 0 0 0 0 0 0.070 0

Non-medium error count: 0

[GLTSD (Global Logging Target Save Disable) set. Enable Save with ‘-S on’]No Self-tests have been logged

Background scan results logStatus: scan is activeAccumulated power on time, hours:minutes 50633:54 [3038034 minutes]Number of background scans performed: 352, scan progress: 4.09%Number of background medium scans performed: 352Device does not support General statistics and performance logging

Protocol Specific port log page for SAS SSPrelative target port id = 1generation code = 4number of phys = 1phy identifier = 0attached device type: SAS or SATA deviceattached reason: unknownreason: unknownnegotiated logical link rate: phy enabled; 6 Gbpsattached initiator port: ssp=1 stp=1 smp=1attached target port: ssp=0 stp=0 smp=0SAS address = 0x5000c500991289c9attached SAS address = 0x5b82a720d2151c01attached phy identifier = 1Invalid DWORD count = 8948Running disparity error count = 8948Loss of DWORD synchronization count = 7Phy reset problem count = 0relative target port id = 2generation code = 4number of phys = 1phy identifier = 1attached device type: no device attachedattached reason: unknownreason: unknownnegotiated logical link rate: phy enabled; unknownattached initiator port: ssp=0 stp=0 smp=0attached target port: ssp=0 stp=0 smp=0SAS address = 0x5000c500991289caattached SAS address = 0x0attached phy identifier = 0Invalid DWORD count = 0Running disparity error count = 0Loss of DWORD synchronization count = 0Phy reset problem count = 0

truenas_admin@truenas[~]$>

<truenas_admin@truenas[~]$ lsblk -bolsblk: option requires an argument – ‘o’Try ‘lsblk --help’ for more information.truenas_admin@truenas[~]$ lsblk -bo NAME,LABEL,MAJ:MIN,TRAN,ROTA,ZONED,VENDOR,MODEL,SERIAL,PARTUUID,START,SIZE,PARTTYPENAMENAME LABEL MAJ:MIN TRAN ROTA ZONED VENDOR MODEL SERIAL PARTUUID START SIZE PARTTYPENAMEsda 8:0 sas 1 none HP LOGICAL VOLUME 50123456789ABCDE 99994728960├─sda1 8:1 1 none 7d11056a-44ec-493a-bbcb-05815753abef 4096 1048576 BIOS boot├─sda2 EFI 8:2 1 none 09ae5559-b9cb-419c-bba9-c92b96e781ba 6144 536870912 EFI System└─sda3 boot-pool 8:3 1 none 02550dd3-290a-4cc0-8f30-9d9ad8f219c6 1054720 99454695424 Solaris /usr & Apple ZFSsdb 8:16 sas 1 none HP LOGICAL VOLUME 50123456789ABCDE 99994728960├─sdb1 8:17 1 none 02ccd752-ee0c-4932-b19c-5186a4b1d89b 4096 1048576 BIOS boot├─sdb2 EFI 8:18 1 none a3beef86-564d-42f4-bec7-6cbbe44ede0f 6144 536870912 EFI System└─sdb3 boot-pool 8:19 1 none a505e856-4956-41ad-bc06-9508f5f405fb 1054720 99454695424 Solaris /usr & Apple ZFSsdc 8:32 sas 1 none SEAGATE ST2000NX0353 S460YDQY0000E715HS3D 2000398934016sdd 8:48 sas 1 none SEAGATE ST2000NX0353 S4611BDX0000E7200SL0 2000398934016sr0 11:0 ata 0 none TEAC TEAC DV-W28S-RS TEAC_DV-W28S-RS 1073741312nvme0n1 259:0 nvme 0 none RMS-200 0007127 8581545984nvme1n1 259:1 nvme 0 none RMS-200 0067361 8581545984truenas_admin@truenas[~]$>

<truenas_admin@truenas[~]$ lspci00:00.0 Host bridge: Intel Corporation 5520 I/O Hub to ESI Port (rev 13)00:01.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 1 (rev 13)00:02.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 2 (rev 13)00:03.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 3 (rev 13)00:04.0 PCI bridge: Intel Corporation 5520/X58 I/O Hub PCI Express Root Port 4 (rev 13)00:05.0 PCI bridge: Intel Corporation 5520/X58 I/O Hub PCI Express Root Port 5 (rev 13)00:06.0 PCI bridge: Intel Corporation 5520/X58 I/O Hub PCI Express Root Port 6 (rev 13)00:07.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 7 (rev 13)00:08.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 8 (rev 13)00:09.0 PCI bridge: Intel Corporation 7500/5520/5500/X58 I/O Hub PCI Express Root Port 9 (rev 13)00:0a.0 PCI bridge: Intel Corporation 7500/5520/5500/X58 I/O Hub PCI Express Root Port 10 (rev 13)00:0d.0 Host bridge: Intel Corporation Device 343a (rev 13)00:0d.1 Host bridge: Intel Corporation Device 343b (rev 13)00:0d.2 Host bridge: Intel Corporation Device 343c (rev 13)00:0d.3 Host bridge: Intel Corporation Device 343d (rev 13)00:0d.4 Host bridge: Intel Corporation 7500/5520/5500/X58 Physical Layer Port 0 (rev 13)00:0d.5 Host bridge: Intel Corporation 7500/5520/5500 Physical Layer Port 1 (rev 13)00:0d.6 Host bridge: Intel Corporation Device 341a (rev 13)00:0e.0 Host bridge: Intel Corporation Device 341c (rev 13)00:0e.1 Host bridge: Intel Corporation Device 341d (rev 13)00:0e.2 Host bridge: Intel Corporation Device 341e (rev 13)00:0e.3 Host bridge: Intel Corporation Device 341f (rev 13)00:0e.4 Host bridge: Intel Corporation Device 3439 (rev 13)00:14.0 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub System Management Registers (rev 13)00:14.1 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub GPIO and Scratch Pad Registers (rev 13)00:14.2 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub Control Status and RAS Registers (rev 13)00:1c.0 PCI bridge: Intel Corporation 82801JI (ICH10 Family) PCI Express Root Port 100:1c.2 PCI bridge: Intel Corporation 82801JI (ICH10 Family) PCI Express Root Port 300:1d.0 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #100:1d.1 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #200:1d.2 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #300:1d.3 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #600:1d.7 USB controller: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #100:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 90)00:1f.0 ISA bridge: Intel Corporation 82801JIB (ICH10) LPC Interface Controller00:1f.2 IDE interface: Intel Corporation 82801JI (ICH10 Family) 4 port SATA IDE Controller #101:03.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] ES1000 (rev 02)01:04.0 System peripheral: Compaq Computer Corporation Integrated Lights Out Controller (rev 03)01:04.2 System peripheral: Compaq Computer Corporation Integrated Lights Out Processor (rev 03)01:04.4 USB controller: Hewlett-Packard Company Integrated Lights-Out Standard Virtual USB Controller01:04.6 IPMI Interface: Hewlett-Packard Company Integrated Lights-Out Standard KCS Interface04:00.0 RAID bus controller: Hewlett-Packard Company Smart Array G6 controllers (rev 01)0d:00.0 Non-Volatile memory controller: Radian Memory Systems Inc. RMS-200 (rev 05)10:00.0 Non-Volatile memory controller: Radian Memory Systems Inc. RMS-200 (rev 05)14:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)14:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)17:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03)3e:00.0 Host bridge: Intel Corporation Xeon 5600 Series QuickPath Architecture Generic Non-core Registers (rev 02)3e:00.1 Host bridge: Intel Corporation Xeon 5600 Series QuickPath Architecture System Address Decoder (rev 02)3e:02.0 Host bridge: Intel Corporation Xeon 5600 Series QPI Link 0 (rev 02)3e:02.1 Host bridge: Intel Corporation Xeon 5600 Series QPI Physical 0 (rev 02)3e:02.2 Host bridge: Intel Corporation Xeon 5600 Series Mirror Port Link 0 (rev 02)3e:02.3 Host bridge: Intel Corporation Xeon 5600 Series Mirror Port Link 1 (rev 02)3e:02.4 Host bridge: Intel Corporation Xeon 5600 Series QPI Link 1 (rev 02)3e:02.5 Host bridge: Intel Corporation Xeon 5600 Series QPI Physical 1 (rev 02)3e:03.0 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Registers (rev 02)3e:03.1 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Target Address Decoder (rev 02)3e:03.2 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller RAS Registers (rev 02)3e:03.4 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Test Registers (rev 02)3e:04.0 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 0 Control (rev 02)3e:04.1 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 0 Address (rev 02)3e:04.2 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 0 Rank (rev 02)3e:04.3 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 0 Thermal Control (rev 02)3e:05.0 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 1 Control (rev 02)3e:05.1 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 1 Address (rev 02)3e:05.2 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 1 Rank (rev 02)3e:05.3 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 1 Thermal Control (rev 02)3e:06.0 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 2 Control (rev 02)3e:06.1 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 2 Address (rev 02)3e:06.2 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 2 Rank (rev 02)3e:06.3 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 2 Thermal Control (rev 02)3f:00.0 Host bridge: Intel Corporation Xeon 5600 Series QuickPath Architecture Generic Non-core Registers (rev 02)3f:00.1 Host bridge: Intel Corporation Xeon 5600 Series QuickPath Architecture System Address Decoder (rev 02)3f:02.0 Host bridge: Intel Corporation Xeon 5600 Series QPI Link 0 (rev 02)3f:02.1 Host bridge: Intel Corporation Xeon 5600 Series QPI Physical 0 (rev 02)3f:02.2 Host bridge: Intel Corporation Xeon 5600 Series Mirror Port Link 0 (rev 02)3f:02.3 Host bridge: Intel Corporation Xeon 5600 Series Mirror Port Link 1 (rev 02)3f:02.4 Host bridge: Intel Corporation Xeon 5600 Series QPI Link 1 (rev 02)3f:02.5 Host bridge: Intel Corporation Xeon 5600 Series QPI Physical 1 (rev 02)3f:03.0 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Registers (rev 02)3f:03.1 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Target Address Decoder (rev 02)3f:03.2 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller RAS Registers (rev 02)3f:03.4 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Test Registers (rev 02)3f:04.0 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 0 Control (rev 02)3f:04.1 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 0 Address (rev 02)3f:04.2 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 0 Rank (rev 02)3f:04.3 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 0 Thermal Control (rev 02)3f:05.0 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 1 Control (rev 02)3f:05.1 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 1 Address (rev 02)3f:05.2 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 1 Rank (rev 02)3f:05.3 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 1 Thermal Control (rev 02)3f:06.0 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 2 Control (rev 02)3f:06.1 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 2 Address (rev 02)3f:06.2 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 2 Rank (rev 02)3f:06.3 Host bridge: Intel>

<truenas_admin@truenas[~]$ sudo sas2flash -listLSI Corporation SAS2 Flash UtilityVersion 20.00.00.00 (2014.09.18)Copyright (c) 2008-2014 LSI Corporation. All rights reserved

Adapter Selected is a LSI SAS: SAS2008(B2)   

Controller Number              : 0
Controller                     : SAS2008(B2)   
PCI Address                    : 00:17:00:00
SAS Address                    : 5b82a72-0-d215-1c00
NVDATA Version (Default)       : 07.00.00.19
NVDATA Version (Persistent)    : 07.00.00.19
Firmware Product ID            : 0x2713 (IR)
Firmware Version               : 07.15.08.00
NVDATA Vendor                  : Dell
NVDATA Product ID              : 6GbpsSAS
BIOS Version                   : 07.11.10.00
UEFI BSD Version               : 07.02.04.00
FCODE Version                  : N/A
Board Name                     : 6Gbps SAS HBA
Board Assembly                 : N/A
Board Tracer Number            : N/A

Finished Processing Commands Successfully.
Exiting SAS2Flash.

truenas_admin@truenas[~]$>

<truenas_admin@truenas[~]$ sudo sas3flash -listAvago Technologies SAS3 Flash UtilityVersion 16.00.00.00 (2017.05.02)Copyright 2008-2017 Avago Technologies. All rights reserved.

No Avago SAS adapters found! Limited Command Set Available!
ERROR: Command Not allowed without an adapter!
ERROR: Couldn't Create Command -list
Exiting Program.

truenas_admin@truenas[~]$>

<truenas_admin@truenas[~]$ sudo storcli show allCLI Version = 007.2807.0000.0000 Dec 22, 2023Operating system = Linux 6.12.33-production+truenasStatus Code = 0Status = SuccessDescription = None

Number of Controllers = 0Host Name = truenasOperating System = Linux 6.12.33-production+truenasStoreLib IT Version = 07.2900.0200.0100

truenas_admin@truenas[~]$>

<truenas_admin@truenas[~]$ for disk in /dev/sd*; do; sudo zdb -l $disk; donefailed to unpack label 0failed to unpack label 1

LABEL 2 (Bad label cksum)

version: 5000
name: ‘boot-pool’
state: 0
txg: 14148
pool_guid: 6800215034065985409
errata: 0
compatibility: ‘grub2’
hostname: ‘(none)’
top_guid: 693339685664969014
guid: 6908350626200935775
vdev_children: 1
vdev_tree:
type: ‘mirror’
id: 0
guid: 693339685664969014
metaslab_array: 256
metaslab_shift: 29
ashift: 12
asize: 99449831424
is_log: 0
create_txg: 4
children[0]:
type: ‘disk’
id: 0
guid: 6908350626200935775
path: ‘/dev/sda3’
whole_disk: 0
create_txg: 4
children[1]:
type: ‘disk’
id: 1
guid: 3266362942703890957
path: ‘/dev/sdb3’
whole_disk: 0
create_txg: 4
features_for_read:
com.delphix:hole_birth
com.delphix:embedded_data
labels = 2 3

failed to unpack label 0

failed to unpack label 1failed to unpack label 2failed to unpack label 3failed to unpack label 0failed to unpack label 1failed to unpack label 2failed to unpack label 3

LABEL 0

version: 5000
name: ‘boot-pool’
state: 0
txg: 14148
pool_guid: 6800215034065985409
errata: 0
compatibility: ‘grub2’
hostname: ‘(none)’
top_guid: 693339685664969014
guid: 6908350626200935775
vdev_children: 1
vdev_tree:
type: ‘mirror’
id: 0
guid: 693339685664969014
metaslab_array: 256
metaslab_shift: 29
ashift: 12
asize: 99449831424
is_log: 0
create_txg: 4
children[0]:
type: ‘disk’
id: 0
guid: 6908350626200935775
path: ‘/dev/sda3’
whole_disk: 0
create_txg: 4
children[1]:
type: ‘disk’
id: 1
guid: 3266362942703890957
path: ‘/dev/sdb3’
whole_disk: 0
create_txg: 4
features_for_read:
com.delphix:hole_birth
com.delphix:embedded_data
labels = 0 1 2 3

failed to unpack label 0

failed to unpack label 1

LABEL 2 (Bad label cksum)

version: 5000
name: ‘boot-pool’
state: 0
txg: 14148
pool_guid: 6800215034065985409
errata: 0
compatibility: ‘grub2’
hostname: ‘(none)’
top_guid: 693339685664969014
guid: 3266362942703890957
vdev_children: 1
vdev_tree:
type: ‘mirror’
id: 0
guid: 693339685664969014
metaslab_array: 256
metaslab_shift: 29
ashift: 12
asize: 99449831424
is_log: 0
create_txg: 4
children[0]:
type: ‘disk’
id: 0
guid: 6908350626200935775
path: ‘/dev/sda3’
whole_disk: 0
create_txg: 4
children[1]:
type: ‘disk’
id: 1
guid: 3266362942703890957
path: ‘/dev/sdb3’
whole_disk: 0
create_txg: 4
features_for_read:
com.delphix:hole_birth
com.delphix:embedded_data
labels = 2 3

failed to unpack label 0

failed to unpack label 1failed to unpack label 2failed to unpack label 3failed to unpack label 0failed to unpack label 1failed to unpack label 2failed to unpack label 3

LABEL 0

version: 5000
name: ‘boot-pool’
state: 0
txg: 14148
pool_guid: 6800215034065985409
errata: 0
compatibility: ‘grub2’
hostname: ‘(none)’
top_guid: 693339685664969014
guid: 3266362942703890957
vdev_children: 1
vdev_tree:
type: ‘mirror’
id: 0
guid: 693339685664969014
metaslab_array: 256
metaslab_shift: 29
ashift: 12
asize: 99449831424
is_log: 0
create_txg: 4
children[0]:
type: ‘disk’
id: 0
guid: 6908350626200935775
path: ‘/dev/sda3’
whole_disk: 0
create_txg: 4
children[1]:
type: ‘disk’
id: 1
guid: 3266362942703890957
path: ‘/dev/sdb3’
whole_disk: 0
create_txg: 4
features_for_read:
com.delphix:hole_birth
com.delphix:embedded_data
labels = 0 1 2 3

failed to unpack label 0failed to unpack label 1failed to unpack label 2failed to unpack label 3failed to unpack label 0failed to unpack label 1failed to unpack label 2failed to unpack label 3truenas_admin@truenas[~]$>

<truenas_admin@truenas[~]$ for disk in /dev/sd?; do; sudo hdparm -W $disk; done

/dev/sda:write-caching = not supported

/dev/sdb:write-caching = not supported

/dev/sdc:write-caching = not supported

/dev/sdd:write-caching = not supportedtruenas_admin@truenas[~]$>

<truenas_admin@truenas[~]$ for disk in /dev/sd?; do; sudo smartctl -x $disk; donesmartctl 7.4 2023-08-01 r5530 [x86_64-linux-6.12.33-production+truenas] (local build)Copyright (C) 2002-23, Bruce Allen, Christian Franke, websitename

/dev/sda: requires option ‘-d cciss,N’Please specify device type with the -d option.

Use smartctl -h to get a usage summary

smartctl 7.4 2023-08-01 r5530 [x86_64-linux-6.12.33-production+truenas] (local build)Copyright (C) 2002-23, Bruce Allen, Christian Franke, websitename

/dev/sdb: requires option ‘-d cciss,N’Please specify device type with the -d option.

Use smartctl -h to get a usage summary

smartctl 7.4 2023-08-01 r5530 [x86_64-linux-6.12.33-production+truenas] (local build)Copyright (C) 2002-23, Bruce Allen, Christian Franke, websitename

=== START OF INFORMATION SECTION ===Vendor: SEAGATEProduct: ST2000NX0353Revision: EF02Compliance: SPC-4User Capacity: 2,000,398,934,016 bytes [2.00 TB]Logical block size: 512 bytesPhysical block size: 4096 bytesLU is fully provisionedRotation Rate: 7200 rpmForm Factor: 2.5 inchesLogical Unit id: 0x5000c500991289cbSerial number: S460YDQY0000E715HS3DDevice type: diskTransport protocol: SAS (SPL-4)Local Time is: Sat May 2 09:47:24 2026 PDTSMART support is: Available - device has SMART capability.SMART support is: EnabledTemperature Warning: EnabledRead Cache is: EnabledWriteback Cache is: Disabled

=== START OF READ SMART DATA SECTION ===SMART Health Status: OK

Grown defects during certificationTotal blocks reassigned during formatTotal new blocks reassignedPower on minutes since formatCurrent Drive Temperature: 30 CDrive Trip Temperature: 60 C

Manufactured in week 51 of year 2016Specified cycle count over device lifetime: 10000Accumulated start-stop cycles: 29Specified load-unload count over device lifetime: 300000Accumulated load-unload cycles: 7957Elements in grown defect list: 0

Vendor (Seagate Cache) informationBlocks sent to initiator = 36024Blocks received from initiator = 128000Blocks read from cache and sent to initiator = 4008Number of read and write commands whose size <= segment size = 35Number of read and write commands whose size > segment size = 0

Vendor (Seagate/Hitachi) factory informationnumber of hours powered up = 50634.02number of minutes until next internal SMART test = 42

Error counter log:Errors Corrected by Total Correction Gigabytes TotalECC rereads/ errors algorithm processed uncorrectedfast | delayed rewrites corrected invocations [10^9 bytes] errorsread: 9477 0 0 9477 0 0.018 0write: 0 0 0 0 0 0.070 0

Non-medium error count: 0

[GLTSD (Global Logging Target Save Disable) set. Enable Save with ‘-S on’]No Self-tests have been logged

Background scan results logStatus: scan is activeAccumulated power on time, hours:minutes 50634:01 [3038041 minutes]Number of background scans performed: 352, scan progress: 6.98%Number of background medium scans performed: 352Device does not support General statistics and performance logging

Protocol Specific port log page for SAS SSPrelative target port id = 1generation code = 4number of phys = 1phy identifier = 0attached device type: SAS or SATA deviceattached reason: unknownreason: unknownnegotiated logical link rate: phy enabled; 6 Gbpsattached initiator port: ssp=1 stp=1 smp=1attached target port: ssp=0 stp=0 smp=0SAS address = 0x5000c500991289c9attached SAS address = 0x5b82a720d2151c01attached phy identifier = 1Invalid DWORD count = 8948Running disparity error count = 8948Loss of DWORD synchronization count = 7Phy reset problem count = 0relative target port id = 2generation code = 4number of phys = 1phy identifier = 1attached device type: no device attachedattached reason: unknownreason: unknownnegotiated logical link rate: phy enabled; unknownattached initiator port: ssp=0 stp=0 smp=0attached target port: ssp=0 stp=0 smp=0SAS address = 0x5000c500991289caattached SAS address = 0x0attached phy identifier = 0Invalid DWORD count = 0Running disparity error count = 0Loss of DWORD synchronization count = 0Phy reset problem count = 0

smartctl 7.4 2023-08-01 r5530 [x86_64-linux-6.12.33-production+truenas] (local build)Copyright (C) 2002-23, Bruce Allen, Christian Franke, websitename

=== START OF INFORMATION SECTION ===Vendor: SEAGATEProduct: ST2000NX0353Revision: EF02Compliance: SPC-4User Capacity: 2,000,398,934,016 bytes [2.00 TB]Logical block size: 512 bytesPhysical block size: 4096 bytesLU is fully provisionedRotation Rate: 7200 rpmForm Factor: 2.5 inchesLogical Unit id: 0x5000c500998afe2fSerial number: S4611BDX0000E7200SL0Device type: diskTransport protocol: SAS (SPL-4)Local Time is: Sat May 2 09:47:25 2026 PDTSMART support is: Available - device has SMART capability.SMART support is: EnabledTemperature Warning: EnabledRead Cache is: EnabledWriteback Cache is: Disabled

=== START OF READ SMART DATA SECTION ===SMART Health Status: OK

Grown defects during certificationTotal blocks reassigned during formatTotal new blocks reassignedPower on minutes since formatCurrent Drive Temperature: 30 CDrive Trip Temperature: 60 C

Manufactured in week 51 of year 2016Specified cycle count over device lifetime: 10000Accumulated start-stop cycles: 25Specified load-unload count over device lifetime: 300000Accumulated load-unload cycles: 8536Elements in grown defect list: 0

Vendor (Seagate Cache) informationBlocks sent to initiator = 34920Blocks received from initiator = 128000Blocks read from cache and sent to initiator = 4248Number of read and write commands whose size <= segment size = 32Number of read and write commands whose size > segment size = 0

Vendor (Seagate/Hitachi) factory informationnumber of hours powered up = 50666.03number of minutes until next internal SMART test = 42

Error counter log:Errors Corrected by Total Correction Gigabytes TotalECC rereads/ errors algorithm processed uncorrectedfast | delayed rewrites corrected invocations [10^9 bytes] errorsread: 11215 0 0 11215 0 0.018 0write: 0 0 0 0 0 0.069 0

Non-medium error count: 0

[GLTSD (Global Logging Target Save Disable) set. Enable Save with ‘-S on’]No Self-tests have been logged

Background scan results logStatus: scan is activeAccumulated power on time, hours:minutes 50666:02 [3039962 minutes]Number of background scans performed: 352, scan progress: 6.92%Number of background medium scans performed: 352Device does not support General statistics and performance logging

Protocol Specific port log page for SAS SSPrelative target port id = 1generation code = 0number of phys = 1phy identifier = 0attached device type: SAS or SATA deviceattached reason: power onreason: loss of dword synchronizationnegotiated logical link rate: phy enabled; 6 Gbpsattached initiator port: ssp=1 stp=1 smp=1attached target port: ssp=0 stp=0 smp=0SAS address = 0x5000c500998afe2dattached SAS address = 0x5b82a720d2151c00attached phy identifier = 0Invalid DWORD count = 6375Running disparity error count = 6375Loss of DWORD synchronization count = 4Phy reset problem count = 0relative target port id = 2generation code = 0number of phys = 1phy identifier = 1attached device type: no device attachedattached reason: unknownreason: unknownnegotiated logical link rate: phy enabled; unknownattached initiator port: ssp=0 stp=0 smp=0attached target port: ssp=0 stp=0 smp=0SAS address = 0x5000c500998afe2eattached SAS address = 0x0attached phy identifier = 0Invalid DWORD count = 0Running disparity error count = 0Loss of DWORD synchronization count = 0Phy reset problem count = 0

truenas_admin@truenas[~]$>

<truenas_admin@truenas[~]$ ifconfig -aens5f0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500ether 00:1b:21:4b:50:fc txqueuelen 1000 (Ethernet)RX packets 0 bytes 0 (0.0 B)RX errors 0 dropped 0 overruns 0 frame 0TX packets 0 bytes 0 (0.0 B)TX errors 0 dropped 24 overruns 0 carrier 0 collisions 0

ens5f1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500inet 192.168.10.12 netmask 255.255.255.0 broadcast 192.168.10.255inet6 fe80::21b:21ff:fe4b:50fd prefixlen 64 scopeid 0x20ether 00:1b:21:4b:50:fd txqueuelen 1000 (Ethernet)RX packets 2664 bytes 306506 (299.3 KiB)RX errors 0 dropped 0 overruns 0 frame 0TX packets 1568 bytes 695612 (679.3 KiB)TX errors 0 dropped 1 overruns 0 carrier 0 collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536inet 127.0.0.1 netmask 255.0.0.0inet6 ::1 prefixlen 128 scopeid 0x10loop txqueuelen 1000 (Local Loopback)RX packets 1726 bytes 150799 (147.2 KiB)RX errors 0 dropped 0 overruns 0 frame 0TX packets 1726 bytes 150799 (147.2 KiB)TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0>

okay, I think that covers everything missed, if not, please tell me what I need to add, but I think that covers everything.

15

crap….why did those not get properly formatted using < >…..I’d edit them, but it seems the site still does not allow me to do so, so, apologies for the reply spam, I cant edit my messages yet.

16

:index_pointing_up:

2 Likes
17

high likelihood that one’s the onboard controller the board has built in. the only thing running through that is the boot ssd’s due to not really having any way to get it to play nice with a boot disk, though those are setup in single disk pass-through per ssd.

18

Fair enough.

Do you have another motherboard, even just consumer grade, to try to rule out cpu/ram/mobo ?

19

absolutely, problem is, not many of them short of other servers are practical to get up and running on short notice, so at the risk of a stupid question, would another server work for the task? I can grab one of the other working HBA’s that are IT mode and pop it into another server I have, dl 360 G7 if that would work for isolation testing of the drives? figure better to ask a potentially stupid question than assume and find out your wrong.

20

Post each command results in it’s own Preformatted Text window (</>) or Ctrl+e. I think trying to shove everything in one caused issues. You can switch to a Preview mode in the reply editor by clicking on the far left M / A icon. It may help you to see if the preview looks ‘correct’ to how you intend it. I think you are way behind on the HBA firmware but it also shows a Dell model. It would not be an onboard one as you said it’s an HP computer.

Posting copy since you had so much info

Adapter Selected is a LSI SAS: SAS2008(B2)   

Controller Number              : 0
Controller                     : SAS2008(B2)   
PCI Address                    : 00:17:00:00
SAS Address                    : 5b82a72-0-d215-1c00
NVDATA Version (Default)       : 07.00.00.19
NVDATA Version (Persistent)    : 07.00.00.19
Firmware Product ID            : 0x2713 (IR)
Firmware Version               : 07.15.08.00
NVDATA Vendor                  : Dell
NVDATA Product ID              : 6GbpsSAS
BIOS Version                   : 07.11.10.00
UEFI BSD Version               : 07.02.04.00
FCODE Version                  : N/A
Board Name                     : 6Gbps SAS HBA
Board Assembly                 : N/A
Board Tracer Number            : N/A

Finished Processing Commands Successfully.
Exiting SAS2Flash.

I would not start mixing parts at this point. Keep the hardware the exact same unless we are trying to rule out a single piece of hardware. Cpu and RAM is usually first. Then we would try to rule out a single HBA and the slot being bad, etc.