I’ve been running scale for several months now, and have accrued some questions I have been unable to answer.
I have a VM that needs to be duplicated several times. ‘clone’ doesn’t boot. I can’t work out an easy workflow to duplicate the VM and underlying zvol.
I’m getting very slow traversal of the file system (raidz2, 7 HDDs, 20TB used, 5TB free). I’m considering adding a metadata only L2ARC. I’m short on SATA headers so M.2 would be nice. Is optane 118GB 800P a mistake? They’re still a bit expensive on ebay. The system has 32GB RAM, but will be upgrading to 64GB soon, once all VMs are set up there will be about 32GB free/ARC.
Can a Monthly scrub be run on only certain days of the week? Weekly seems very excessive, but if a scrub falls on a weekend there’s likely to be heavy IO when it triggers.
What is the most idiot proof way to set up backups? USB HDD would be ideal.
Can a bridge be easily set up to allow only local SMB access, and not allow traversal of the network/gateway/internet.
20TB used from a total of 25TB is too high - ZFS slows down (on writes) when you exceed 80%. You need to free some disk space reasonably urgently or enlarge your pool.
You need to examine your reports to see what your current (without L2ARC) cache hit rate is. If it is already very high then an L2ARC won’t do much.
Do some research about how much RAM you need before L2ARC becomes worthwhile as I suspect that it is way more than 32GB.
“My understanding is Metadata only L2ARC needs less space and less RAM than a full L2ARC.”
It does indeed - however it also does a lot less than a full L2ARC in that it only caches metadata rather than evicted ARC. In many cases (almost all) a metadata only L2ARC is a good idea (and really can do no harm) where a full L2ARC is a bad idea (and can actively hinder NAS Operations - a bit)
Note that a metadata cache will not populate itself with the entire metadata but will only cache stuff you access, so you may want to force the population by traversing all directories etc before you see much gain
As u/Protopia says your pool is too full for good performance AND the wrong format for VM’s / ZVOL performance - which should be mirrors for IOPS
[Opinion]
The Optane 800p is a PCIe 3*2 unit. So technically its throughput is slow(er) than a 900p. However throughput isn’t the most important thing for an L2ARC, but latency is more important and the 800p should have very low latency.
Regarding point 4.
For backing up zvols, the in built replication function is OK if you want a simple way to just output a backup (which can include snapshots). My linux VMs didn’t boot after though, I had to do the following.
In UEFI Interactive Shell, enter the file system:
fs0:
Following up with creating this file:
edit startup.nsh
Enter this or similar line to it:
\EFI\debian\grubx64.efi
Press CTRL+O to save the file (the turotial I used said CTRL+S, this did not seem to work)
Press ENTER to confirm the file name.
Press CTRL+Q to exit the editor.
Restart the Guest:
reset
Regarding point 5.
Creating a bridge that I don’t connect to the internet facing NIC, with a subnet not accessible to my gateway, seems to have worked. The VM’s NIC is then put on the bridge, and manually configured within the VM (I don’t trust DHCP not to ‘fix’ the isolation). The bridge provides local DNS and SMB from truenas without issue. All good.
For future me and anyone stumbling here from a search engine.
Network > Interfaces > Add.
Type: bridge
Name: br02 (or brNN, where NN are a pair of numbers not assigned to an existing bridge)
[OPTIONAL] Description: Offline bridge
Don’t tick DHCP or IPv6
Bridge members: Don’t select anything
MTU 1500
Aliases: 192.168.55.100 / 24 (Or per your network and preferred subnet)
Then in your VM assign the interface to 192.168.55.101, subnet prefix length 24, and if you have DNS server on your NAS set DNS to 192.168.55.100. Then mount your SMB from 192.168.55.100 or DNS served path.
The scrub issue is fixed with the Threshold Days setting. Set it to 28 or 35 for 4 or 5 weekly respectively, with a weekly schedule. Then it will only run on that day, but not if it has run in the last howevermany days.