Help recovering application data after OS upgrade

Apps and Virtualization
1

I have been using TruNAS scale for a few years now and I have had a great experience with it until today. I had installed an application called tandoor. It is a django application that manages recipies, and we’ve been using it for a few years and love it!

I just upgraded my TruNAS from 22->23->24… and this application is no longer there, and actually most of my applications are not working. I don’t really care about any of them other than tandoor, as they are easy for me to just spin up elsewhere. I see in the forums after hitting my issue that helm based applications are no longer supported. As just a “normal user” this was not made clear in the upgrade process. I did get a warning about S3 no longer being available and I didn’t care about that and proceeded with the upgrades.

OK, complaining over. I’d like to get back my data. I did take a backup of my main pool’s data directory on an external USB drive before I started, including ix-xxxx dirs. I believe this was in a postgres container within the tandoor helm chart but I don’t see how to get this mounted somewhere I can recover from. I feel like I’m pretty competent at docker and helm/k8s but I’m not seeing where the data volume is / was in order to get to it.

Does anyone have ideas on how to recover this? I have several years of recipes on this system and really don’t want to lose them.

I am of the mind I’ll just use this system as a NAS and not bother with running apps on it any more, but I really want to get this data out!

2

When you say 22-23-24 did you go through 23.10, 24.04 and then 24.10?
if you did you can roll back from 24.10 to 24.04 and the apps should have been preserved. Were the apps operational before the step from 24.04 → 24.10?

I suggest reading the apps migration information provided in the release notes:

1 Like
3

Thank you! I did do them in order, and I re-activated my 24.04 boot environment and rebooted but I still see TrueNAS-SCALE-24.10.2 on the main dashboard. Other than system settings > boot ( then activate 24.04), then reboot, what do I need to do to roll back?

4

That should be it, when you check the boot environments after reboot which shows activated?

5

It looks like it is activated. Nothing was running in applications, so I activated a 23.10.2 and things showed back up at least, but won’t start. I tried logs in the ui but could not see why. so I logged in and did k3s kubectl -n ix-recipes:

root@truenas[~]# k3s kubectl get pods -n ix-recipes        
NAME                       READY   STATUS             RESTARTS   AGE
recipes-7879466986-8pvqk   0/2     Init:0/1           0          93m
recipes-postgresql-0       0/1     ImagePullBackOff   0          93m

Getting the details, it looks like the postgres pod is failing to start with:

        message: Back-off pulling image "tccr.io/truecharts/postgresql:14.7.0@sha256:f71c97329aa3bdcd7d9fac73596b74348bee81499a6e132e00045b3cf358cc06"
        reason: ImagePullBackOff
6

could it be this?

(tried posting image, it won’t let me so here is the text of it…)

Failed to sync TRUECHARTS catalog: [EFAULT] Failed to clone 'https://github.com/truecharts/catalog' repository at '/mnt/main/ix-applications/catalogs/github_com_truecharts_catalog_main' destination: Cmd('git') failed due to: exit code(128) cmdline: git clone -v https://github.com/truecharts/catalog /mnt/main/ix-applications/catalogs/github_com_truecharts_catalog_main stderr: 'Cloning into '/mnt/main/ix-applications/catalogs/github_com_truecharts_catalog_main'... fatal: could not read Username for 'https://github.com': No such device or address '

Following the message, I tried cloning that url on my dev machine and get the same issue, github is wanting a username to clone it…

7

The truecharts catalog was eliminated quite a long time ago. They stopped supporting Truenas. when Truenas switched to docker. Likely you can find something on Truenas website on how to migrate to another setup.

I believe someone also has a historical clone of the truecharts catalog that might work to get you back running, however, you are going to have to migrate your apps off truecharts and to either jailmaker, IX apps, custom apps, or the solution from Truecharts whatever that is. Not a Truecharts user just providing an initial historical context.

Likely you went along with their suggestion of not using host mounts, so, you’ll have to extract all the data from pvc storage. It should be in their doc on how to do this.

8

relative to my thinking a long time is not what you think it is :slight_smile:

Point taken, [begin whine] but now I seemingly have to write my own chart to mount my pvcs and copy all the data off to put elsewhere, then hopefully be able to recreate this thing that was working last week just fine until I decided I really should update the os as it was 2 years old. Bad decision on my part. [end whine].

I’ll probably just nuke this OS, keep my files on the ZFS pool TruNAS created for me and go with something more in my control like a openmediavault or even just debian, and run my own rancher RKE on the box for apps/charts. I know rancher won’t decide to pull support for Kubernetes as that is its primary function.

I truly get where the IX folks are coming from with the decision to not support this any more, but I do wish it were different. I do software development for a living and you have to chose your features carefully, and sometimes you do have to abandon some users that are not paying the bills (me) in favor of those that are.

It was a great product for us “home labbers” while it lasted, though. I do thank you all for your work and the charts.

“So long, and thanks for all the fish”

1 Like
9

FYI, here is what I did to get the data out… the tandoor app mounted a postgres pvc and a config pvc

  1. create a chart to run busybox and mount the pvc:
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: busybox-deployment
  namespace: ix-recipes
  labels:
    app: busybox
spec:
  replicas: 1
  selector:
    matchLabels:
      app: busybox
  template:
    metadata:
      labels:
        app: busybox
    spec:
      containers:
        - name: busybox
          image: "busybox:latest"
          command: ["sleep", "360000"]
          volumeMounts:
            - name: db-recipes-postgresql
              mountPath: /data
      volumes:
        - name: db-recipes-postgresql
          persistentVolumeClaim:
            claimName: db-recipes-postgresql-0

(run the above with k3s kubectl apply -f whatever.yaml)
2. copy the files out k3s kubectl exec -n ix-recipes busybox-deployment-dfbccb9cf-kxqqv -- tar cf - /data | tar xf - -C /mnt/backups/recipe

  1. repeat for the config dir.