App migration failure during SCALE 24.04.02 -> 24.10.02 upgrade - mount path already in use

Apps and Virtualization
,
1

This morning I went ahead and performed the SCALE upgrade from 24.04.02 to 24.10.02. All of the apps I use are from the official TrueNAS catalog. All of my apps migrated successfully, except for my Plex app/container. It looks like it is due to one of the mount paths are already in use for another volume mount:

[
    {
        "name": "plex",
        "error": "Failed to create app: [EFAULT] Failed to render compose templates: Traceback (most recent call last):
  File \"/usr/bin/apps_render_app\", line 33, in <module>
    sys.exit(load_entry_point('apps-validation==0.1', 'console_scripts', 'apps_render_app')())
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File \"/usr/lib/python3/dist-packages/catalog_templating/scripts/render_compose.py\", line 47, in main
    render_templates_from_path(args.path, args.values)
  File \"/usr/lib/python3/dist-packages/catalog_templating/scripts/render_compose.py\", line 19, in render_templates_from_path
    rendered_data = render_templates(
                    ^^^^^^^^^^^^^^^^^
  File \"/usr/lib/python3/dist-packages/catalog_templating/render.py\", line 36, in render_templates
    ).render({'ix_lib': template_libs, 'values': test_values})
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File \"/usr/lib/python3/dist-packages/jinja2/environment.py\", line 1301, in render
    self.environment.handle_exception()
  File \"/usr/lib/python3/dist-packages/jinja2/environment.py\", line 936, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File \"/mnt/.ix-apps/app_configs/plex/versions/1.1.6/templates/docker-compose.yaml\", line 32, in top-level template code
    {% do c1.add_storage(store.mount_path, store) %}
  ^^^^^^^^^^^^^^^^^^^^^^^^^
  File \"/mnt/.ix-apps/app_configs/plex/versions/1.1.6/templates/library/base_v2_1_0/container.py\", line 208, in add_storage
    self._storage.add(mount_path, config)
  File \"/mnt/.ix-apps/app_configs/plex/versions/1.1.6/templates/library/base_v2_1_0/storage.py\", line 89, in add
    raise RenderError(f\"Mount path [{mount_path}] already used for another volume mount\")
base_v2_1_0.error.RenderError: Mount path [/transcode] already used for another volume mount
",
        "successfully_migrated": false
    }
]

I tried running midclt call -job k8s_to_docker.migrate <poolname> as specified in the docs, but the migration still failed.

Is there a .yaml file or some sort of config somewhere I can edit to remove the mount point for /transcode, or any other suggestions to get the migration of this app/container to succeed?

Edit: Also, I don’t have any encrypted volumes/pools.

Thank you for any help!

2

Hey there, did you ever resolve this?

I’ve run into the same issue, except with Deluge not Plex.

I also know what the problem is because I’ve run into it before when I was migrating my apps to a different pool, but I don’t know how to solve it.

Basically, when I first installed Deluge it had an option to choose the Downloads directory. I didn’t change it at the time and it defaulted to an ixVolume, which was fine because I mounted a separate media directory as well which I was using to download directly into my media library.

Somewhere down the line I wanted to use the default /downloads directory for some things but didn’t want the downloaded data somewhere hidden in the ix-applications dataset. So I went to change the option, but it turns out that setting is immutable once set:

Excerpt from Deluge chart because I can't post links 
https://github.com/truenas/charts/commit/d77d46640da15f0583b95953539c806861957b11#diff-2a138a84a2f40068bc3102b56a5c94abf430e2ce1bb500545a600280ba0963f5R167-R204
        - variable: downloads
          label: Deluge Downloads Storage
          description: The path to store Deluge Downloads.
          schema:
            type: dict
            attrs:
              - variable: type
                label: Type
                description: |
                  ixVolume: Is dataset created automatically by the system.</br>
                  Host Path: Is a path that already exists on the system.
                schema:
                  type: string
                  required: true
                  default: "ixVolume"
                  enum:
                    - value: "hostPath"
                      description: Host Path (Path that already exists on the system)
                    - value: "ixVolume"
                      description: ixVolume (Dataset created automatically by the system)
              - variable: datasetName
                label: Dataset Name
                schema:
                  type: string
                  show_if: [["type", "=", "ixVolume"]]
                  required: true
                  hidden: true
                  immutable: true
                  default: "downloads"
                  $ref:
                    - "normalize/ixVolume"
              - variable: hostPath
                label: Host Path
                schema:
                  type: hostpath
                  show_if: [["type", "=", "hostPath"]]
                  immutable: true
                  required: true

I’m not sure why that was done, but either way I was able to just add a Host Path mount that mounted to /downloads and that worked for over a year.

Fast forward to me migrating my apps pool when I ran into a similar issue already, it wouldn’t start the container because of the duplicate mount path.

Solution:

  1. Edit application to remove the Host Path mount
  2. Start Deluge once, then stop it again
  3. Edit application to add back the Host Path mount
  4. :tada:

It seems that on first creating the container it checks for the duplicate paths, but on further starts it allows an override? Maybe that was a bug that worked in my favor? IDK how or why that worked, but it did.

Either way, I suspect the solution is the exact same here, but I don’t know how to go about it.

Can I just boot back into Dragonfish, edit Deluge to remove the Host Path, boot back into Electric Eel and to the migration again? Then change or add the Host Path back?

3

OK well, nevermind, doesn’t work.
I rolled the dice on it and just tried it, and no.

It seems on the Dragonfish to Eletric Eel upgrade it creates a snapshot of the Apps pool (which makes sense of course), and when re-attempting the migration with idclt call -job k8s_to_docker.migrate poolname it restores that snapshot and attempts the migration from there. So no matter what you change in the old install, it won’t carry over to the migration.

Makes me wonder of what the point of the note in the Release Notes was:

You can re-initiate a failed migration of any previously-installed Kubernetes apps to Docker at any time after upgrading to Electric Eel. From a shell session enter midclt call -job k8s_to_docker.migrate poolname , where poolname is the name of the applications pool, for example tank .

If it failed the first time, why would it succeed the second try when you literally cannot fix any of the apps?

It seems the only fix is booting back into DF, removing the duplicate path (which I already did), and then re-doing the entire upgrade to EE.

Note on this: Because the midclt call restores the snapshot, you’ll have to do the path removal after booting into DF and then just do the update right away. If you attempt to do the migration in EE again, it’ll restore the deleted path.