I have a Dell r740 and install Truenas Scale it has 256gb of memory and a Xeon Gold 10C40T cpu. I currently have 12 drives with 2 vdevs raidz2 equaling to around 89TB’s of storage.
I have a few options for adding cache for iscsi and nfs shares. I have room for 6 more drives and could put ssd’s for the cache or I could get another boss card and add 2 nvme drives. Or I can also get optane ram and another cpu. I do have another 256gb of ram to put into the server but I prefer not to save on the power bill for adding a cpu.
Now for those that aren’t aware Dell kinda ruined the R740’s due to after a certain idrac version that if you add in a non branded pcie device the server fans will run louder than normal and there is no way to turn down the fans.
Adding RAM before adding L2ARC is always recommended.
I’m also not sure if your workload would even really benefit from an L2ARC to be quite honest. You could go to the trouble and end up seeing a L2ARC hit ratio sub 10% if it’s only your steam library. And Proxmox backups don’t need to be in cache?
Yeah the cache would be only for nfs and more importantly for iscisi. On DSM I had 6 ssd’s for the iscisi and could try and create zfs share with them but if I want to add more ram with the additional spare 256gb in total spare modules. I will have to add another cpu and I really don’t want to do that and 64gb ddr4 modules are still quite expensive.
Fair enough, I’m just saying for your workload I would not expect an L2ARC to be as effective as RAM, and you’d have to add power draw by adding disks anyway. The lower power states on modern CPUs are pretty efficient, so I’m not sure that it’d make much of a difference year-over-year with a mild workload. You would have to measure.
I agree - L2ARC - almost certainly pointless - you have more than enough RAM
SLOG - not required for your use case - just make sure that NFS uses async writes