Hi All,
After a long history of errors in my server, I’ve encountered one not hardware related. Before this issue I used to have a lot of HDD errors and had to replace them all except one in TrueNAS Cobia, after verifying that the issue is gone by keeping it in Cobia for a month, I decided to upgrade to TrueNAS Scale Dragonfish, the update was cool with new features added but when I went to the Apps tab, I was greeted with a Red Exclamation Sign syaing that Applications are not running and the Blue Wheel of Death (Stuck on “Initializing Apps Service”) as I have encountered the Blue Wheel of Death before, my remedy was to just Unset Pool and Choose Pool again but in this case it didn’t work, I tried deleting the token under k3s but the token didn’t exist, I restarted k3s service via CLI but that didn’t help. I can confirm that none of the apps are working by logging into an IP for a Speedtest app. I am running out of options and I am fed up with this server, the only reason why I am not giving up easily is because I have 3TB worth of DVD ripped TV Shows and Movies and don’t want to lose them. It would be really appreciated if some could help me out. Although this might not be helpful but I still have one HDD that is probably failing because of the checksum errors from that and something related to k3s when I try “zpool status -v” it does say that 4-5 files from the same directory have encountered permanent file damage so that might be something
Thanks!
Ashvath Sridharan
I ran into this recently. My symptoms were a little more like what is shown in this old community post. That is, in my case, I had an unexpected power outage. My Applications, though, were in the same state. I tried the solution that is suggested in that thread and it didn’t work (moving/renaming the $app_pool/ix-applications/k3s/server/token file). But then I found a solution.
When I ran systemctl status k3s I would see:
time="2024-06-01T15:43:45-04:00" level=error msg="failed to ping connection: malformed database schema (sqlite_sequence)"
I found the state.db database in /mnt/data/ix-applications/k3s/server/db/state.db (/mnt/data is my RAID volume).
I did these commands to save a copy of state.db just in case, recover what could be recovered from the database, and then restart the k3s service with the recovered state.db.
systemctl stop k3scd /mnt/data/ix-applications/k3s/server/db/cp state.db state.db.savesqlite3 state.db.save ".dump" > recovered.sqlsqlite3 state.db.recovered < recovered.sqlcp state.db.recovered state.dbsystemctl start k3s
Then check on the service by using systemctl status k3s
Lots of help online will have you combine steps 4 and 5 into a single pipeline. But I did it this way in case some of the SQL ended up needing to be edited by hand. It didn’t. But that was my reasoning.
After i did that, I could see the apps I had launched. I stopped them, updated them, and now I’m in the process of relaunching them. I’m not quite recovered, yet, but I’m a lot closer to being recovered than before I started.
1 Like
Just to close this out, I ended up doing a few more steps, which frankly reset the whole thing.
- I deleted the apps I had configured
- I unset the pool for apps
- I set the pool for apps
- I re-selected and re-created all my container apps
Kinda sucks, but at least I’m back.
1 Like
Hello, is your problem solved? I also encountered the same problem. I replaced the motherboard and CPU for my NAS last night. This morning, all applications could not be started and were displayed as “deploying”. Then I reset the system and deleted the “ix-applications” dataset, but it still did not work.
Hi DarkStark, yes it has been fixed, in fact a few days after I even posted this issue. Thank you to GeneralMarmite for reaching out, you may try his optio, that might actaully work. I say “might” because I haven’t tried out his method, I used a different method. What I did was that usually when you update your NAS, a snapshot will be created for ix-applications, I have no clue how but it just happens. Using the recent snapshot I had, I rebuilt the pool, although there are some disadvantages such as you won’t get the recent changes done to any of you apps nor will you get any of the apps that you would have installed after the recent snapshot but I had no choice as it has already been 2 weeks since I had posted this issue at that time and no one responded. Thinking that I will never get my apps back I made up my mind and just reverted my pool. There are some scary warnings such as “If you go back to a snapshot, there are less chances of your apps to ever work and will lose all the other snapshots” but I was ready to lose them all and take the risk as at that time, it was my only option. And what do you know, my apps came back to normal! Except Tautulli which still doesn’t work but I am trying to mount the pool so that I can retrieve the backup file and get my history.
DISCLAIMER: I do recommend this method to you all but it’s not guaranteed to get your apps back, in fact it may even render you NAS dead but if you have no option and are ready to take the risk. go for it. And also, like I said your NAS can be different to mine and that your system may not be configured to take snapshots automatically, but in my case it did but the backup job wasn’t listed anywhere.
I had this issue after a power outage. There’s a thread on the old forum with similar reports. For me I think my CMOS battery was dead and the time and date got clobbered after the reboot. I was able to fix the issue by rebooting again. I think this solved it because the system got the right time after the first boot. I’ve since replaced the CR2032 battery on my motherboard as well.
There were other solutions on the old forum, something about deleting a token file, and one just directing people to set the time and date.
