So:
fuser -vmM /var/log/journal
USER PID ACCESS COMMAND
/var/log/journal: root kernel mount /var/log/journal
root 733 F...m systemd-journal
root 3193 f...m syslog-ng
Shutdown (started after upgrading to fangtooth) or reboot results in failed unmounting of /var/log/journal folder - it forces me to do hard reset, otherwise is stuck forever. I cleaned that folder in single mode, did rotate and vacuum logs and nothing helps. Still after booting dmesg shows:
dmesg | grep journal
[ 15.973228] systemd[1]: Listening on systemd-journald-audit.socket - Journal Audit Socket.
[ 15.973634] systemd[1]: Listening on systemd-journald-dev-log.socket - Journal Socket (/dev/log).
[ 15.974029] systemd[1]: Listening on systemd-journald.socket - Journal Socket.
[ 15.998213] systemd[1]: Starting systemd-journald.service - Journal Service...
[ 16.129922] systemd[1]: Started systemd-journald.service - Journal Service.
[ 16.151453] systemd-journald[727]: Received client request to flush runtime journal.
[ 16.153723] systemd-journald[727]: File /var/log/journal/9a2f5936ccc44b26ab86c36c48b1e82e/system.journal corrupted or uncleanly shut down, renaming and replacing.
[ 77.867097] systemd-journald[727]: Data hash table of /var/log/journal/9a2f5936ccc44b26ab86c36c48b1e82e/system.journal has a fill level at 75.0 (8536 of 11377 items, 6553600 file size, 767 bytes per hash table item), suggesting rotation.
[ 77.867104] systemd-journald[727]: /var/log/journal/9a2f5936ccc44b26ab86c36c48b1e82e/system.journal: Journal header limits reached or header out-of-date, rotating.
[ 78.044461] systemd-journald[727]: Failed to read journal file /var/log/journal/9a2f5936ccc44b26ab86c36c48b1e82e/user-1002.journal for rotation, trying to move it out of the way: Device or resource busy
[ 78.044708] systemd-journald[727]: Failed to read journal file /var/log/journal/9a2f5936ccc44b26ab86c36c48b1e82e/user-1004.journal for rotation, trying to move it out of the way: Device or resource busy
And after a week or so of uptime, webui becomes unresponsive with a lot of sigkills of systemd-journald…
Like I mentioned, after fresh reinstall all is ok, it comes back after restoring of configuration.
See also: https://forums.truenas.com/t/shutdown-or-reboot-takes-30-min/39376/1