I removed the log vdev from my bool using the GUI, in hopes to use the NVME for L2ARC and use the SSD I was using for the L2ARC for the Log vdev (swap between the two). In hopes to cache metadata files for things like Plex. the removal of the L2ARC quickly removed without issue, but when I selected to remove the Log vdev the removal looked like it started going along well, 20%. However, I waited for a few hours and it had no budged from 20%. I rebooted but its still stuck at 20%. Pretty much nothing is working now, is this recoverable?
Removing a LOG / SLOG vDev when the pool is imported, is generally fast and harmless.
However, based on the first picture you showed, it appears to be a Special vDev which can be slow to remove as all the data has to be moved from the Special vDev device(s) to regular data devices.
If you give us the output of this command, we can confirm what is happening; zpool status
Trying to log back in again to check, but seems hard to even do that, keep timing out, seems to be getting worse by the minute…I’m not sure if its “SLOG” vdev but it definitely said log in it. So likely?
I was also going to suggest you run sudo zpool status -v to see what ZFS says is happening cf. what the TrueNAS UI is interpreting as happening.
Leaving aside the issue you are having removing the SLOG, I have the following more general comments:
Do you really need SLOG anyway? Synchronous writes do 10x-100x as many writes as asynchronous writes, and the performance is noticeably slower, so you only want to do synchronous writes if absolutely necessary and an SLOG is only beneficial if you are doing synchronous writes. In general terms, you only need to do synchronous writes for VM virtual disks/zVolumes/iSCSI & Databases, but if you do need to do synchronous write these are typically high IOPS small random accesses that are better on Mirrors than RAIDZ. You then only need SLOG if you are doing these synchronous writes and only if the SLOG is on significantly faster technology than the data.
Normal sequential access to normal files does NOT usually need to be synchronous writes.
Plex media files are definitely sequential writes of entire files, and do not need synchronous writes. Because Plex is only copying data from the internet, the metadata probably doesn’t need sequential writes either even if internally it is a database of some kind. But Plex metadata definitely benefits from being on SSD.
L2ARC - generally speaking large memory is cheaper and more beneficial than implementing L2ARC, but if you believe that L2ARC is still needed then the general guideline is that you need at least 64GB of memory before you start to see benefits.
A special allocation vDev to hold metadata may help, but you need to know what the benefits and risks are before you implement it, especially for RAIDZ because once added it cannot be removed.
So once you have resolved removing your SLOG, my advice would be to step back and evaluate whether an SLOG or L2ARC is actually beneficial before reimplementing either or both of these.
You should be able to get some stats on cache performance with / without L2ARC from the TrueNAS Reports screens and / or Netdata.
I manually rebooted there system and it came back up luckily. I added a single L2ARC cache vdev to the system (and nothing else) but I don’t know if its a placebo effect but sop far it seems like the library load times are noticeably faster, and seems more of my RAM is free instead of eaten up by ZFS Cache. Currently of my 64GiBs of ECC RAM now: 41GiB ZFS Cache, 11.2GiB, Services 10.3GiB. Think it was about 3~4GiBs were free on average, but then again might drop back down to previous amounts.
