I have a pretty annoying issue, and I just can’t figure my head around it.
First of all, I’m running Version 25.04.0
Initially, I had a single Disk in a Stripe configuration. Since having a Stripe is pretty dumb (which I’ve unfortunately only recently learned, gladly no data was lost yet), I’ve added another 3 Disks to form a Z1. Then I started to migrate all the Data using Replication. It’s pretty much only Media for Plex and 1-2 VMs which aren’t even running at the moment.
I have also exported the system dataset to a temporary SSD, and I changed the paths in the docker containers to the new pool. My media server runs fine.
Now I wanted to delete that Stripe disk and also add it to the Z1. Instead of deleting it first, though, I thought it would be better to physically remove it first, so I detached it from my raid controller.
Starting the Server without that Disk attached boots TrueNAS, but I cannot access the Web UI or even ping the Server at all. When I reattach the drive, TrueNAS works fine again, just like before. I even made that disk offline using the TrueNAS CLI, and e.g. using SSH there’s no data left on the pool anymore.
But something has to be still on that pool, right? Otherwise, it should work. Any help would be greatly appreciated!
Did you export or move the system dataset to the SSD? Where is the system dataset stored when you view Advanced Setting? I think that is where to look.
I think you will need to post details of what your system was set up as originally, what the pool and vdev structure was and the verbose steps you took adding the 3 disk Raid-Z1 Vdev and pool. You also mention VMs and Apps. Details on that too.
You should also post a detailed hardware listing. My post should have an expandable Details section for a example.
Anyone reading this needs to be able to follow your process so they can comment on anything that may have been wrong or needs to be done. If you need to run CLI commands and post results to explain, please post back using Preformatted Text </> or Ctrl+e on the tool bar. It makes it easier to read. You can also post images.
I will try my best to make this as detailed as possible.
OS Version: 25.04.0
Hardware:
CPU: AMD Ryzen 5 5600G
MB: Gigabyte B550M
RAM: 2 x 32 GB Corsair Vengeance LPX DDR4-3600
GPU: Asus DUAL GeForce GTX 1060 6GB
Drives:
Boot: Samsung 840 Pro 128 GB 2.5"
Data: 4x Seagate Barracuda Compute 8 TB
+2 random NVMe Drives for caching (one is temporarily used for the system storage)
Original Setup:
1 Data Drive in STRIPE. All Data was on that drive (except the boot stuff of course). That includes system data, Apps Data, VMs and my Media.
Migration Steps:
Add 3 new Data Disks and create a Z1 Pool
Create Snapshots of Datasets on Stripe Pool
Replicate all Datasets from Stripe to Z1 Pool
Stop Media Server Containers (they use the Media-Server Dataset)
Change their mounts from Strip to Z1 Pool, then restart the entire Stack
Export System Storage onto one of the NVMe drives
Mark the Stripe Disk as Offline (Pool is in Degraded Status since)
Current Setup:
Stripe Pool:
Degraded
1x 8 TB
Still has all the original Data
Sys Pool:
NVMe
Holds the exported System Storage
The Dataset which is marked as “This dataset is used by the system”
Z1 Pool:
3x 8TB
Replicated Media Server Data
Replicated VMs
Current Problem:
When I physically disconnect the single 8 TB Disk containing the Strip Pool, TrueNAS boots and is available when using a Monitor. It also has internet access (ping 1.1.1.1 works). Trying to access the WebUI or pinging using the IP 192.168.0.60 (which is even written in the Shell and hasn’t changed) doesn’t work, it’s basically inaccessible over the network. Reconnecting the drive and rebooting the server makes it accessible again (thank god).
I don’t think any of my containers use the Stripe Pool anymore, since when I check in SSH the dataset folders do still exist on the pool but they are completely empty (I think that’s because I marked the disk as offline).
One thing that I don’t understand is, where the /mnt/.ix-apps data is stored. I mean on which drive. Is it the NVMe since that contains the sys pool?
EDIT
Okay I actually just found this blog post and I read that I can choose in the Apps Tab in the WebUI on which Pool the .ix-apps is stored. I changed it now from the original Stripe pool to the Z1.
Maybe that alone will be enough. I can only check in the evening though since Im not home right now… If that was the only thing left would be really frustrating At least I found the issue.
These drives are SMR and not CMR. They are not suited for use with ZFS.
SMR vs CMR ServeTheHome
You keep saying ‘exported’ for the system dataset. I expected the system dataset to be ‘moved’ using the GUI. Something like originally on ‘Pool1’, attached NVMe, created ‘sys’ pool on NVMe. Using GUI changed system dataset location from ‘Pool1’ to ‘sys’ pool. This moves the data and changes where TrueNAS looks. I can’t tell if that is how you did it.
Are your pool names, ‘Stripe’, ‘sys’ and ‘z1_pool’, or something like that?
Thanks for posting all that as the details may help other users ask questions or figure out the procedure you took.
I hate to say it, but I would make the switch to CMR drives. You want hard drives from a NAS or Enterprise line. WD Red Pro, WD Gold, Toshiba N300, Segate Ironwolf, Seagate EXOS are all examples.
Since it seems that all you data fit on a 8TB to start with, what was your end goal for your pool storage?
A common setup is Mirror or Raid-Z1 (RaidZ2 recommended over Z1 due to drive size) VDEV and pool for bulk data using hard rives and a Mirror DVEV for apps and VMs using the NVME. You are close to where we don’t recommend using Raid-Z1 because of the 8TB and larger drives. Resilver process is stressful and you can’t have any other drives fail or you loose the pool and all data inside it. Resilvering a Mirror is less stressful since it is mainly just copying data from working drive to new replacement.
I am suggesting Mirror VDEVs because they are flexible. We can start with a single, stripe device and add a second device to create a Mirror VDEV. We can also upgrade pool sizes easier with disk replacements. Raid-Z(1,2,3) requires all disk members to be larger before you can see an increase in pool size.
Could you get by with 8 or 16TB of total pool space for now? I am thinking you would use your current SMR drives in a Mirror configuration.
1st VDEV of Mirror x 2 8TB drives
2nd VDEV of Mirror x 2 8TB drives. Both VDEVs would belong to your main pool
I think you will have less stress in case of a drive failure and, with the mirror VDEV type, you can save up and work on replacing two drives with CMR models in the 1st VDEV.
You could use the NVME to create a second pool and Mirror VDEV for apps and VMs. SSD and NVME pools are better for these but you can have them on HD pools for light usages and performance.
Let us know where you are on the ability to purchase new drives or have to keep current ones. We can then come up with a plan to get your system sorted out for now and in the future.
I can definitely get some other drives; it’s not a big deal. I’ll just sell the others again, the few lost bucks are hopefully worth it
Instead of getting new disks, I read that buying used ones is also fine. So I saw 4x HGST 8TB SAS HDD 12G for sale. I looked online, and they seem to be a good fit for NAS storage.
I’ll just get those and migrate all my data (again, but this time with all of the newly learned knowledge ).
Yea I already have a SATA HBA but I’m probably goona sell it. SAS drives seem to be way cheaper to buy used than SATA drives and I can easily sell my HBA again and just buy a SAS HBA.
EDIT:
I just realized, can I simply just buy other cables? Or does my HBA need to be SAS compatible?
The seller I bought it from flashed it into IT-Mode. Should I put another firmware onto it? It was working fine with my SATA drives and they were not put into a singular drive, I have access to each individual drive connected.
I just checked the controller details and it seems it’s on FW-Version 12.00.02.00 and by the linked docs it’s supposed to be on at least 16.00.10.00 or is that not for my card?
------------------------------------------------------------------------
Controller information
------------------------------------------------------------------------
Controller type : SAS3008
BIOS version : 8.29.02.00
Firmware version : 12.00.02.00
Channel description : 1 Serial Attached SCSI
Initiator ID : 0
Maximum physical devices : 1023
Concurrent commands supported : 11264
Slot : 1
Segment : 0
Bus : 4
Device : 0
Function : 0
RAID Support : No
A bit of paranoia on the flag, mostly. When most people say “RAID controller” they mean something like an Areca, RocketRAID, or Dell PERC card that puts a layer of obfuscation between system and drive, which leads to 1) bad performance and 2) potential data safety issues. Your LSI card is fine here, because it’s running in IT (Initiator/Target) mode - it’s an “HBA” or “Host Bus Adapter” as desired.
SAS3008’s should be on the 16-branch firmware, yes. There’s a 16.00.20.00 I believe that addresses specific drive firmware interactions but your Seagate’s aren’t on that list (although them being SMR is a challenge as outlined by others.)
At least I found the issue.
).