SOLVED: Apps fail to start after restoring a pool or hardware change

TrueNAS General
, ,
1

If you get a message like this when you unset and set your app pool, this is your lucky day:

image
image658×649 18.7 KB

Symptom: You get “Apps Service Stopped” message and all your apps have disappeared. You try to unset the pool and set the pool for the apps and same problem. If you mouse over the error message, it shows something like: “Application(s) have failed to start: Command ‘(‘mount’, ‘-t’, ‘zfs’, ‘main/ix-applications/k3s/kubelet’, ‘/var/lib/kubelet’)’ returned non-zero exit status”

Cause: is that for some weird reason the mountpoint property for /ix-applications/k3s/kubelet was set to something other than legacy. So a command to mount kubelet on /var/lib/kubelet will fail. Could be you changed your ethernet port for example.

Code fix
The fix in the code is simply to set the mountpoint property to legacy before calling the mount. So it’s a one line code fix to a problem that has been affecting people for at least two years now (for example see this post). I reported this and the workaround fix in Bug NAS-128555

Workaround in meantime
In the meantime, here’s the fast and easy fix for users that survives reboot:

  1. CLI: sudo zfs set mountpoint=legacy main/ix-applications/k3s/kubelet
  2. GUI: Unset pool. Then set it back to your normal pool for apps.
  3. You will see “App service starting” instead of the error.
  4. Then you will wait and see NO apps! But that is a head fake.
  5. Refresh the web page and you will see all your apps are back!

Save your config now that it works

Give this post a “like” for the hours I just saved you :slight_smile:

7 Likes
2

Brilliant post! I accidentally detached the wrong device in my mirror and had to replicate over to a completely new pool (as they are of slightly mismatched sizes and I couldn’t reattach the damn thing…!). Ran into this exact issue and setting a legacy mountpoint sorted things, so thank you very much for the direction! :slight_smile:

I wonder if there’s a particular reason why this needs to be mounted in this way.

This should probably be in resources as it’s super useful!

1 Like
3

I think it’s just old code using deprecated methods only because it was written long ago (or written by someone who didn’t know about the new way to do the mounts for zfs via properties).

4

You are awesome! This fixed my issue. Everyone probably knows this, but you have to update the sudo zfs set mountpoint=legacy main/ix-applications/k3s/kubelet to your own structure. In my case, it was sudo zfs set mountpoint=legacy Main_Tank/ix-applications/k3s/kubelet.

Also, I ran into an issue where I had to actually reboot my system to get it to start toward resolving the issue.

5

thanks. I thought that would save a lot of people a lot of time.

I filled out a bug report a long time ago for them to fix it.

6

Steve, funny thing is I follow you on X for certain information/awareness if you know what I’m saying.

Now you got me creating accounts on truenas forums just to thank you for saving me hours troubleshooting my issue with importing a pool(well replicated pool, but still!)

thanks again, Steve!

1 Like
7

thanks for letting me know. Truenas is so much fun.