So as the OP I thought I would update you all. I have received further feedback from the developers on the ticket I raised as follows:
Looking on the debug, I see that 3.2GB of ARC size is used by dnodes. It likely means that your workload includes large number (millions) of accessed files. We’ve recently tested somewhat similar configuration and also noticed some excessive free memory during benchmarks. Memory reported as “services” may be a related OS overhead of inodes/dnodes/etc. In our case we’ve noticed positive effect from this patch now waiting for upstream review: . I expect it to be included into next 24.10 release, but there can be more to this problem, since it is pretty difficult area since the beginning of ZFS, that just complicated by some Linux specifics.
I read that as my potential issue is an actual issue that is probably going to be patched in 24.10. Happy days.