I can confirm that turning off lru_gen solves the excessive-swapping problem on my test system. (My prod system won’t upgrade until later…) That’s with no changes to arc_max or arc_sys_free, or swappiness.
Of course, changing (just) lru_gen also raises the free memory under pressure from 1.2GB to 6.2GB (on a 32G system), severely reducing memory pressure at the cost of ~5GB. I note that setting arc_sys_free to 6GB instead reduces the intensity of persistent swapping but does not eliminate it outright, so it looks like lru_gen itself changes the interaction between the kernel’s memory system and zfs’s arc.
Thanks to everybody who contributed to this thread; it got me a much better starting point on figuring this out.
Cheers
– perry