Diagnosing slow shutdown?

TrueNAS General
1

I have a SCALE system that often takes 35-40 seconds to shutdown.

In the systemd journal I see nothing unusual in the 5-7 seconds between the issuance of shutdown and when reboot.target is reached.

What’s going on during the next 30 seconds? This is the tail-end of the journal just before lights-out.

image
image1530×276 22.4 KB

I’m wondering if this is related… This happens on every shutdown:

image
image1732×624 112 KB

Also managed to screenshot this just after a hang and before a reboot (only displayed for a fraction of a second):

image
image2391×755 215 KB

One more data point… A user-initiated sync usually runs/exits immediately but sometimes takes 30 seconds to exit. Happens at random. I suspect it’s related to my slow shutdowns if systemd is having to SIGKILL it…

2

The error messages you had, I get those as well and I have seen others who get those. I would not worry about those unless you find out you are having corruption issues. I have not noticed any at all.

As for the slower shutdown, you have not provided any information for someone to really figure it out. No system specs and no mention of how your system is configured, Version of TrueNAS, VM’s, Dockers, etc. Did this ever run faster and if so, under what different conditions?

Cheers

3

Because I have little faith that it’s actionable information for anyone. What I’m asking is how to plug the “logging gap” to determine what’s happening in that final 30 seconds before reboot. This is generic linux troubleshooting that has nothing to do with specs.

I should hope that any VMs or Dockers would be long-since shut down so late in the shutdown process. Regardless, there are neither.

Edit: I don’t recall when this issue began. I plan to back out every customization and tunable and untwist every knob I’ve twisted. Fortunately I have good notes so it won’t take long to undo…

4

Seems to me that a timeout of 30s for the filesystems to sync and unmount timed out.

That seems to be logged.