Datasets listing and VMs in errors after configuration restore --> use the Password Secret Seed!

My boot-pool disk just crashed and after reinstall and upload of the latest backup I have I realize it’s not up-to-date.

I made somee recent changes I haven’t backup, including some new datasets I’ve created during “pool consolidation” (moving all small pools to a bigger one before migration to a new server).

When I go to menu Datasets, I get a " Failed to load datasets ‘path’" error in the GUI.

Most of the missing datasets are zvol, as far as I can remember.
They are on disk, but the config is missing.

I have snaphots for all of them I can see on disks with zfs list -t snapshot

What’s the best way to recover them?

Shall I restore a snapshots to restore the configuration and overwrite the existing or is there a way to manually restore the configuration ?(provided I can remember all the needed infos?)

Do you mean the config file?


What do you mean here?


Recover what? Are you saying that the datasets and zvols exist, but they do not properly display in the GUI?


It’s not clear what the problem is.

Can you provide more information? Screenshots?

Maybe the output of this:

zfs list -t filesystem

What exactly do you mean by “configuration”?

Users?
Shares? SMB?

These are not part of snapshots. If you roll back datasets to a snapshot you will lose all data written to that dataset after the particular snapshot was taken. So probably better don’t.

The configuration (if it is some or all of the above entities you are seeking) would have been part of a current saved configuration file. It’s nowhere in the datasets.

So you need to recreate whatever is missing manually.

To start best give a more detailled description, possibly a single example of what exactly does not work the way it should.

Output of e.g. zfs list will probably also help.

Sorry for not being clear enough.

By backup, I meant configuration.

My last Configuration backup dates back from Feb7th.
In between, unfortunately, I’ve done numerous changes.
Anything configuration, user changes, shares config, etc. of course, I’ll have to redo them by hand. It’s a personal system, nothing too dramatic.

But I also eliminated some smaller pools and moved datasets to my main pool. These are more important and fortunately I can see the datasets created after Feb with zfs list on my main pool.

But trying to display Datasets ends up in an error in the GUI. (I was mistaken by this into thinking my new zvols did not exist in the pool, they do).

Trying to edit the Devices of my main (and only important) VM fails. (The vm exists, the devices exists but upon edit no information shows up)

Here’s the corresponding error log in middlewared.log:

[2025/03/06 08:44:49] (WARNING) application.call_method():247 - Exception while calling pool.dataset.details(*[])
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 211, in call_method
    result = await self.middleware.call_with_audit(message['method'], serviceobj, methodobj, params, self)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1529, in call_with_audit
    result = await self._call(method, serviceobj, methodobj, params, app=app,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1471, in _call
    return await self.run_in_executor(prepared_call.executor, methodobj, *prepared_call.args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1364, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 183, in nf
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 55, in nf
    res = f(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/pool_/dataset_details.py", line 207, in details
    info = self.build_details(mnt_info)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/pool_/dataset_details.py", line 321, in build_details
    vm['zvol'] = zvol_path_to_name(vm['attributes']['path'])
                                   ~~~~~~~~~~~~~~~~^^^^^^^^
KeyError: 'path'

and here’s the stacktrace corresponding to editing a disk device on a vm:

[2025/03/06 08:46:16] (WARNING) application.call_method():247 - Exception while calling vm.device.disk_choices(*[])
concurrent.futures.process._RemoteTraceback:
"""
Traceback (most recent call last):
  File "/usr/lib/python3.11/concurrent/futures/process.py", line 261, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/worker.py", line 112, in main_worker
    res = MIDDLEWARE._run(*call_args)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/worker.py", line 46, in _run
    return self._call(name, serviceobj, methodobj, args, job=job)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/worker.py", line 34, in _call
    with Client(f'ws+unix://{MIDDLEWARE_RUN_DIR}/middlewared-internal.sock', py_exceptions=True) as c:
  File "/usr/lib/python3/dist-packages/middlewared/worker.py", line 40, in _call
    return methodobj(*params)
           ^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 183, in nf
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/zfs_/dataset_encryption.py", line 52, in unlocked_zvols_fast
    data['attachments'] = get_attachments()
                          ^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/zfs_/dataset_encryption.py", line 42, in get_attachments
    vm_zvols = {
               ^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/zfs_/dataset_encryption.py", line 43, in <dictcomp>
    zvol_path_to_name(i['attributes']['path']): i for i in vm_devices
                      ~~~~~~~~~~~~~~~^^^^^^^^
KeyError: 'path'
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 211, in call_method
    result = await self.middleware.call_with_audit(message['method'], serviceobj, methodobj, params, self)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1529, in call_with_audit
    result = await self._call(method, serviceobj, methodobj, params, app=app,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1460, in _call
    return await methodobj(*prepared_call.args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 179, in nf
    return await func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 49, in nf
    res = await f(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/vm/vm_devices.py", line 53, in disk_choices
    zvols = await self.middleware.call(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1629, in call
    return await self._call(
           ^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1468, in _call
    return await self._call_worker(name, *prepared_call.args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1474, in _call_worker
    return await self.run_in_proc(main_worker, name, args, job)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1380, in run_in_proc
    return await self.run_in_executor(self.__procpool, method, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1364, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
KeyError: 'path'

I think this is happening because some pools that were Disconnected and disks on which they were setup were removed.
But the restored configuration is still expecting them, both for VM disks and for Datasets listing AFAICS.

I’m trying to identify what could be changed in the configuration file db that I used for the restore.
It’d be tedious to remove all reference to removed pools but this could work I guess.

But I’d rather do that on the system itself. I’m not sure how to identify where is the “live” configuration.

I stumble upon /var/db/system/configs-2fa742ae30da433ca735af52bcb65edb which holds 2.6Gb of configs, and contains all my system configs back from FreeNAS-9.3-STABLE-201604041648 !! Funny, not even 10 years of FN/TN yet!

I can see a .db file for each day in /var/db/system/configs-2fa742ae30da433ca735af52bcb65edb/TrueNAS-SCALE-24.10.2/

Would I be able to use on of these to restore my system to the closest config before my boot-pool disk crash ?

I was able to restore to March 02 config using /var/db/system/configs-2fa742ae30da433ca735af52bcb65edb/TrueNAS-SCALE-24.10.2/20250302.db

And it works, I get the latest config for users (I can login with truenas_admin that I recently added), shares are up to date, but the 2 mentionned above issues persist.

I still can’t display Datasets or edit a VM and even less start one.

I’m out of clue now, help very welcome.

Might you kindly?

Even if a command or output will not solve something, it can lead us to a better understanding where to look next.

Allright, sorry I didn’t do it earlier:

root@freenas[~]# zfs list
NAME                                                          USED  AVAIL  REFER  MOUNTPOINT
boot-pool                                                    2.77G   226G    96K  none
boot-pool/.system                                            2.32M   226G   104K  legacy
boot-pool/.system/configs-ae32c386e13840b2bf9c0083275e7941     96K   226G    96K  legacy
boot-pool/.system/cores                                        96K  1024M    96K  legacy
boot-pool/.system/netdata-ae32c386e13840b2bf9c0083275e7941   1.64M   226G  1.64M  legacy
boot-pool/.system/nfs                                         104K   226G   104K  legacy
boot-pool/.system/samba4                                      292K   226G   192K  legacy
boot-pool/ROOT                                               2.76G   226G    96K  none
boot-pool/ROOT/24.10.2                                       2.76G   226G   165M  legacy
boot-pool/ROOT/24.10.2/audit                                  412K   226G   412K  /audit
boot-pool/ROOT/24.10.2/conf                                  6.82M   226G  6.82M  /conf
boot-pool/ROOT/24.10.2/data                                   320K   226G   320K  /data
boot-pool/ROOT/24.10.2/etc                                   7.71M   226G  6.55M  /etc
boot-pool/ROOT/24.10.2/home                                   104K   226G   104K  /home
boot-pool/ROOT/24.10.2/mnt                                     96K   226G    96K  /mnt
boot-pool/ROOT/24.10.2/opt                                     96K   226G    96K  /opt
boot-pool/ROOT/24.10.2/root                                   152K   226G   152K  /root
boot-pool/ROOT/24.10.2/usr                                   2.40G   226G  2.40G  /usr
boot-pool/ROOT/24.10.2/var                                   37.2M   226G  31.7M  /var
boot-pool/ROOT/24.10.2/var/ca-certificates                     96K   226G    96K  /var/local/ca-certificates
boot-pool/ROOT/24.10.2/var/log                               4.67M   226G  1.82M  /var/log
boot-pool/ROOT/24.10.2/var/log/journal                       2.85M   226G  2.85M  /var/log/journal
boot-pool/grub                                               8.40M   226G  8.40M  legacy
subramanya                                                   13.9T  29.5T   224K  /mnt/subramanya
subramanya/.bhyve_containers                                 70.5M  29.5T  70.5M  /mnt/subramanya/.bhyve_containers
subramanya/.system                                           2.60G  29.5T  1.54G  legacy
subramanya/.system/configs-2fa742ae30da433ca735af52bcb65edb   237M  29.5T   237M  legacy
subramanya/.system/cores                                      240K  1024M   240K  legacy
subramanya/.system/netdata-2fa742ae30da433ca735af52bcb65edb   683M  29.5T   683M  legacy
subramanya/.system/nfs                                        264K  29.5T   264K  legacy
subramanya/.system/rrd-2fa742ae30da433ca735af52bcb65edb       131M  29.5T   131M  legacy
subramanya/.system/samba4                                    22.0M  29.5T  2.39M  legacy
subramanya/.system/services                                   192K  29.5T   192K  legacy
subramanya/.system/syslog-2fa742ae30da433ca735af52bcb65edb   15.9M  29.5T  15.9M  legacy
subramanya/.system/webui                                      176K  29.5T   176K  legacy
subramanya/Loki                                               192K  29.5T   192K  /mnt/subramanya/Loki
subramanya/SYNCTHING_TRASHCAN                                39.0G  29.5T  39.0G  /mnt/subramanya/SYNCTHING_TRASHCAN
subramanya/Storage                                           2.51T  29.5T  2.02T  /mnt/subramanya/Storage
subramanya/TimeMachine                                       1.43T  29.5T   719K  /mnt/subramanya/TimeMachine
subramanya/TimeMachine/Asura                                  367G  32.6G   367G  /mnt/subramanya/TimeMachine/Asura
subramanya/TimeMachine/asuritimemachine                       440G   273G   427G  /mnt/subramanya/TimeMachine/asuritimemachine
subramanya/TimeMachine/jean                                   416K  29.5T   264K  /mnt/subramanya/TimeMachine/jean
subramanya/TimeMachine/parvatitm                              655G  80.0G   620G  /mnt/subramanya/TimeMachine/parvatitm
subramanya/calibre-books                                     17.4G  29.5T  17.4G  /mnt/subramanya/calibre-books
subramanya/cinema                                            8.61T  29.5T  8.03T  /mnt/subramanya/cinema
subramanya/container-volumes                                  205G  29.5T   185G  /mnt/subramanya/container-volumes
subramanya/containers-content                                30.2G  29.5T  19.4G  -
subramanya/home                                              18.3M  29.5T  18.3M  /mnt/subramanya/home
subramanya/iSCSI-dockerhost                                  29.8G  29.5T  28.5G  -
subramanya/iocage                                            67.4G  29.5T  16.1M  /mnt/subramanya/iocage
subramanya/iocage/download                                   1.22G  29.5T   176K  /mnt/subramanya/iocage/download
subramanya/iocage/download/12.1-RELEASE                       371M  29.5T   371M  /mnt/subramanya/iocage/download/12.1-RELEASE
subramanya/iocage/download/13.1-RELEASE                       434M  29.5T   434M  /mnt/subramanya/iocage/download/13.1-RELEASE
subramanya/iocage/download/13.2-RELEASE                       444M  29.5T   444M  /mnt/subramanya/iocage/download/13.2-RELEASE
subramanya/iocage/images                                      176K  29.5T   176K  /mnt/subramanya/iocage/images
subramanya/iocage/jails                                      58.5G  29.5T   208K  /mnt/subramanya/iocage/jails
subramanya/iocage/jails/WebServer                            91.2M  29.5T   208K  /mnt/subramanya/iocage/jails/WebServer
subramanya/iocage/jails/WebServer/root                       91.0M  29.5T  2.00G  /mnt/subramanya/iocage/jails/WebServer/root
subramanya/iocage/jails/minio                                1.53G  29.5T   599K  /mnt/subramanya/iocage/jails/minio
subramanya/iocage/jails/minio/root                           1.53G  29.5T  1.53G  /mnt/subramanya/iocage/jails/minio/root
subramanya/iocage/jails/netdata                              2.02G  29.5T   631K  /mnt/subramanya/iocage/jails/netdata
subramanya/iocage/jails/netdata/root                         2.01G  29.5T  2.01G  /mnt/subramanya/iocage/jails/netdata/root
subramanya/iocage/jails/plex                                 17.1G  29.5T   647K  /mnt/subramanya/iocage/jails/plex
subramanya/iocage/jails/plex/root                            17.1G  29.5T  16.0G  /mnt/subramanya/iocage/jails/plex/root
subramanya/iocage/jails/syncthing                            33.1G  29.5T   623K  /mnt/subramanya/iocage/jails/syncthing
subramanya/iocage/jails/syncthing/root                       33.1G  29.5T  27.1G  /mnt/subramanya/iocage/jails/syncthing/root
subramanya/iocage/jails/transmission                         4.66G  29.5T   607K  /mnt/subramanya/iocage/jails/transmission
subramanya/iocage/jails/transmission/root                    4.65G  29.5T  3.52G  /mnt/subramanya/iocage/jails/transmission/root
subramanya/iocage/log                                         496K  29.5T   496K  /mnt/subramanya/iocage/log
subramanya/iocage/releases                                   7.61G  29.5T   176K  /mnt/subramanya/iocage/releases
subramanya/iocage/releases/11.2-RELEASE                      1.40G  29.5T   176K  /mnt/subramanya/iocage/releases/11.2-RELEASE
subramanya/iocage/releases/11.2-RELEASE/root                 1.40G  29.5T   176K  /mnt/subramanya/iocage/releases/11.2-RELEASE/root
subramanya/iocage/releases/12.1-RELEASE                      1.95G  29.5T   192K  /mnt/subramanya/iocage/releases/12.1-RELEASE
subramanya/iocage/releases/12.1-RELEASE/root                 1.95G  29.5T  1.95G  /mnt/subramanya/iocage/releases/12.1-RELEASE/root
subramanya/iocage/releases/13.1-RELEASE                      2.18G  29.5T   192K  /mnt/subramanya/iocage/releases/13.1-RELEASE
subramanya/iocage/releases/13.1-RELEASE/root                 2.18G  29.5T  2.06G  /mnt/subramanya/iocage/releases/13.1-RELEASE/root
subramanya/iocage/releases/13.2-RELEASE                      2.08G  29.5T   192K  /mnt/subramanya/iocage/releases/13.2-RELEASE
subramanya/iocage/releases/13.2-RELEASE/root                 2.08G  29.5T  2.08G  /mnt/subramanya/iocage/releases/13.2-RELEASE/root
subramanya/iocage/templates                                   176K  29.5T   176K  /mnt/subramanya/iocage/templates
subramanya/ix-applications                                   1.25G  29.5T   527K  /mnt/subramanya/ix-applications
subramanya/ix-applications/catalogs                          89.7M  29.5T  89.7M  /mnt/subramanya/ix-applications/catalogs
subramanya/ix-applications/default_volumes                    192K  29.5T   192K  /mnt/subramanya/ix-applications/default_volumes
subramanya/ix-applications/k3s                               1.16G  29.5T  1.15G  /mnt/subramanya/ix-applications/k3s
subramanya/ix-applications/k3s/kubelet                        527K  29.5T   527K  legacy
subramanya/ix-applications/releases                           336K  29.5T   192K  /mnt/subramanya/ix-applications/releases
subramanya/ix-apps                                            332M  29.5T   208K  /mnt/.ix-apps
subramanya/ix-apps/app_configs                                192K  29.5T   192K  /mnt/.ix-apps/app_configs
subramanya/ix-apps/app_mounts                                 192K  29.5T   192K  /mnt/.ix-apps/app_mounts
subramanya/ix-apps/docker                                     328K  29.5T   328K  /mnt/.ix-apps/docker
subramanya/ix-apps/truenas_catalog                            331M  29.5T   331M  /mnt/.ix-apps/truenas_catalog
subramanya/jellyfin-data                                      249G  29.6T  26.0G  -
subramanya/k8s                                                975K  29.5T   208K  /mnt/subramanya/k8s
subramanya/k8s/iscsi                                          384K  29.5T   192K  /mnt/subramanya/k8s/iscsi
subramanya/k8s/iscsi/vols                                     192K  29.5T   192K  /mnt/subramanya/k8s/iscsi/vols
subramanya/k8s/nfs                                            384K  29.5T   192K  /mnt/subramanya/k8s/nfs
subramanya/k8s/nfs/vols                                       192K  29.5T   192K  /mnt/subramanya/k8s/nfs/vols
subramanya/minio                                             1.03M  29.5T  1.03M  /mnt/subramanya/minio
subramanya/music                                              447G  29.5T   442G  /mnt/subramanya/music
subramanya/photovideo                                         165G  29.5T   154G  /mnt/subramanya/photovideo
subramanya/proxmox                                           21.7G  29.5T  1.53G  -
subramanya/proxmox-backup                                     146M  29.5T   145M  -
subramanya/proxmox-dockerhost                                30.5G  29.5T  30.5G  -
subramanya/tftproot                                           984M  29.5T   984M  /mnt/subramanya/tftproot
subramanya/ubuntudockerhost-0vhbns                           52.0G  29.5T  34.4G  -
subramanya/xcp-storage                                       15.6G  29.5T  15.6G  -

Some more commands to hopefully narrow this down:

zfs list -r -t filesystem,volume -o name,type,encryption,encryptionroot,keystatus,mountpoint subramanya

Are you using ZFS encryption?

I’m not using any encryption

root@freenas[~]# zfs list -r -t filesystem,volume -o name,type,encryption,encryptionroot,keystatus,mountpoint subramanya
NAME                                                         TYPE        ENCRYPTION   ENCROOT  KEYSTATUS    MOUNTPOINT
subramanya                                                   filesystem  off          -        -            /mnt/subramanya
subramanya/.bhyve_containers                                 filesystem  off          -        -            /mnt/subramanya/.bhyve_containers
subramanya/.system                                           filesystem  off          -        -            legacy
subramanya/.system/configs-2fa742ae30da433ca735af52bcb65edb  filesystem  off          -        -            legacy
subramanya/.system/cores                                     filesystem  off          -        -            legacy
subramanya/.system/netdata-2fa742ae30da433ca735af52bcb65edb  filesystem  off          -        -            legacy
subramanya/.system/nfs                                       filesystem  off          -        -            legacy
subramanya/.system/rrd-2fa742ae30da433ca735af52bcb65edb      filesystem  off          -        -            legacy
subramanya/.system/samba4                                    filesystem  off          -        -            legacy
subramanya/.system/services                                  filesystem  off          -        -            legacy
subramanya/.system/syslog-2fa742ae30da433ca735af52bcb65edb   filesystem  off          -        -            legacy
subramanya/.system/webui                                     filesystem  off          -        -            legacy
subramanya/Loki                                              filesystem  off          -        -            /mnt/subramanya/Loki
subramanya/SYNCTHING_TRASHCAN                                filesystem  off          -        -            /mnt/subramanya/SYNCTHING_TRASHCAN
subramanya/Storage                                           filesystem  off          -        -            /mnt/subramanya/Storage
subramanya/TimeMachine                                       filesystem  off          -        -            /mnt/subramanya/TimeMachine
subramanya/TimeMachine/Asura                                 filesystem  off          -        -            /mnt/subramanya/TimeMachine/Asura
subramanya/TimeMachine/asuritimemachine                      filesystem  off          -        -            /mnt/subramanya/TimeMachine/asuritimemachine
subramanya/TimeMachine/jean                                  filesystem  off          -        -            /mnt/subramanya/TimeMachine/jean
subramanya/TimeMachine/parvatitm                             filesystem  off          -        -            /mnt/subramanya/TimeMachine/parvatitm
subramanya/calibre-books                                     filesystem  off          -        -            /mnt/subramanya/calibre-books
subramanya/cinema                                            filesystem  off          -        -            /mnt/subramanya/cinema
subramanya/container-volumes                                 filesystem  off          -        -            /mnt/subramanya/container-volumes
subramanya/containers-content                                volume      off          -        -            -
subramanya/home                                              filesystem  off          -        -            /mnt/subramanya/home
subramanya/iSCSI-dockerhost                                  volume      off          -        -            -
subramanya/iocage                                            filesystem  off          -        -            /mnt/subramanya/iocage
subramanya/iocage/download                                   filesystem  off          -        -            /mnt/subramanya/iocage/download
subramanya/iocage/download/12.1-RELEASE                      filesystem  off          -        -            /mnt/subramanya/iocage/download/12.1-RELEASE
subramanya/iocage/download/13.1-RELEASE                      filesystem  off          -        -            /mnt/subramanya/iocage/download/13.1-RELEASE
subramanya/iocage/download/13.2-RELEASE                      filesystem  off          -        -            /mnt/subramanya/iocage/download/13.2-RELEASE
subramanya/iocage/images                                     filesystem  off          -        -            /mnt/subramanya/iocage/images
subramanya/iocage/jails                                      filesystem  off          -        -            /mnt/subramanya/iocage/jails
subramanya/iocage/jails/WebServer                            filesystem  off          -        -            /mnt/subramanya/iocage/jails/WebServer
subramanya/iocage/jails/WebServer/root                       filesystem  off          -        -            /mnt/subramanya/iocage/jails/WebServer/root
subramanya/iocage/jails/minio                                filesystem  off          -        -            /mnt/subramanya/iocage/jails/minio
subramanya/iocage/jails/minio/root                           filesystem  off          -        -            /mnt/subramanya/iocage/jails/minio/root
subramanya/iocage/jails/netdata                              filesystem  off          -        -            /mnt/subramanya/iocage/jails/netdata
subramanya/iocage/jails/netdata/root                         filesystem  off          -        -            /mnt/subramanya/iocage/jails/netdata/root
subramanya/iocage/jails/plex                                 filesystem  off          -        -            /mnt/subramanya/iocage/jails/plex
subramanya/iocage/jails/plex/root                            filesystem  off          -        -            /mnt/subramanya/iocage/jails/plex/root
subramanya/iocage/jails/syncthing                            filesystem  off          -        -            /mnt/subramanya/iocage/jails/syncthing
subramanya/iocage/jails/syncthing/root                       filesystem  off          -        -            /mnt/subramanya/iocage/jails/syncthing/root
subramanya/iocage/jails/transmission                         filesystem  off          -        -            /mnt/subramanya/iocage/jails/transmission
subramanya/iocage/jails/transmission/root                    filesystem  off          -        -            /mnt/subramanya/iocage/jails/transmission/root
subramanya/iocage/log                                        filesystem  off          -        -            /mnt/subramanya/iocage/log
subramanya/iocage/releases                                   filesystem  off          -        -            /mnt/subramanya/iocage/releases
subramanya/iocage/releases/11.2-RELEASE                      filesystem  off          -        -            /mnt/subramanya/iocage/releases/11.2-RELEASE
subramanya/iocage/releases/11.2-RELEASE/root                 filesystem  off          -        -            /mnt/subramanya/iocage/releases/11.2-RELEASE/root
subramanya/iocage/releases/12.1-RELEASE                      filesystem  off          -        -            /mnt/subramanya/iocage/releases/12.1-RELEASE
subramanya/iocage/releases/12.1-RELEASE/root                 filesystem  off          -        -            /mnt/subramanya/iocage/releases/12.1-RELEASE/root
subramanya/iocage/releases/13.1-RELEASE                      filesystem  off          -        -            /mnt/subramanya/iocage/releases/13.1-RELEASE
subramanya/iocage/releases/13.1-RELEASE/root                 filesystem  off          -        -            /mnt/subramanya/iocage/releases/13.1-RELEASE/root
subramanya/iocage/releases/13.2-RELEASE                      filesystem  off          -        -            /mnt/subramanya/iocage/releases/13.2-RELEASE
subramanya/iocage/releases/13.2-RELEASE/root                 filesystem  off          -        -            /mnt/subramanya/iocage/releases/13.2-RELEASE/root
subramanya/iocage/templates                                  filesystem  off          -        -            /mnt/subramanya/iocage/templates
subramanya/ix-applications                                   filesystem  off          -        -            /mnt/subramanya/ix-applications
subramanya/ix-applications/catalogs                          filesystem  off          -        -            /mnt/subramanya/ix-applications/catalogs
subramanya/ix-applications/default_volumes                   filesystem  off          -        -            /mnt/subramanya/ix-applications/default_volumes
subramanya/ix-applications/k3s                               filesystem  off          -        -            /mnt/subramanya/ix-applications/k3s
subramanya/ix-applications/k3s/kubelet                       filesystem  off          -        -            legacy
subramanya/ix-applications/releases                          filesystem  off          -        -            /mnt/subramanya/ix-applications/releases
subramanya/ix-apps                                           filesystem  off          -        -            /mnt/.ix-apps
subramanya/ix-apps/app_configs                               filesystem  off          -        -            /mnt/.ix-apps/app_configs
subramanya/ix-apps/app_mounts                                filesystem  off          -        -            /mnt/.ix-apps/app_mounts
subramanya/ix-apps/docker                                    filesystem  off          -        -            /mnt/.ix-apps/docker
subramanya/ix-apps/truenas_catalog                           filesystem  off          -        -            /mnt/.ix-apps/truenas_catalog
subramanya/jellyfin-data                                     volume      off          -        -            -
subramanya/k8s                                               filesystem  off          -        -            /mnt/subramanya/k8s
subramanya/k8s/iscsi                                         filesystem  off          -        -            /mnt/subramanya/k8s/iscsi
subramanya/k8s/iscsi/vols                                    filesystem  off          -        -            /mnt/subramanya/k8s/iscsi/vols
subramanya/k8s/nfs                                           filesystem  off          -        -            /mnt/subramanya/k8s/nfs
subramanya/k8s/nfs/vols                                      filesystem  off          -        -            /mnt/subramanya/k8s/nfs/vols
subramanya/minio                                             filesystem  off          -        -            /mnt/subramanya/minio
subramanya/music                                             filesystem  off          -        -            /mnt/subramanya/music
subramanya/photovideo                                        filesystem  off          -        -            /mnt/subramanya/photovideo
subramanya/proxmox                                           volume      off          -        -            -
subramanya/proxmox-backup                                    volume      off          -        -            -
subramanya/proxmox-dockerhost                                volume      off          -        -            -
subramanya/tftproot                                          filesystem  off          -        -            /mnt/subramanya/tftproot
subramanya/ubuntudockerhost-0vhbns                           volume      off          -        -            -
subramanya/xcp-storage                                       volume      off          -        -            -

I keep looking up what could cause the issue.

Looking at the VM issue, when creating a new VM I can see its conf file in /etc/libvirt/qemu. But there’s no traces of the 2 VMs I have in this directory.

Searching into the 20250302.db conf file I found in /var/db/system/configs-2fa742ae30da433ca735af52bcb65edb, I can find the VMs’ definition in vm_vm table.
There must be something that prevented the VM to be restore/reconfigured correctly.

The python stack traces all fail on trying to read some VMs attributes. It must be linked.

I can’t find a specific log that would be linked to a configuration upload/restoration. Does this exists?
If not where else could I track what happened during config restoration?

When trying to create a VM, “Select Existing Zvol” doesn’t work.

Same type of error:

[2025/03/07 22:07:52] (WARNING) application.call_method():247 - Exception while calling vm.device.disk_choices(*[])
concurrent.futures.process._RemoteTraceback:
"""
Traceback (most recent call last):
  File "/usr/lib/python3.11/concurrent/futures/process.py", line 261, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/worker.py", line 112, in main_worker
    res = MIDDLEWARE._run(*call_args)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/worker.py", line 46, in _run
    return self._call(name, serviceobj, methodobj, args, job=job)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/worker.py", line 34, in _call
    with Client(f'ws+unix://{MIDDLEWARE_RUN_DIR}/middlewared-internal.sock', py_exceptions=True) as c:
  File "/usr/lib/python3/dist-packages/middlewared/worker.py", line 40, in _call
    return methodobj(*params)
           ^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 183, in nf
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/zfs_/dataset_encryption.py", line 52, in unlocked_zvols_fast
    data['attachments'] = get_attachments()
                          ^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/zfs_/dataset_encryption.py", line 42, in get_attachments
    vm_zvols = {
               ^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/zfs_/dataset_encryption.py", line 43, in <dictcomp>
    zvol_path_to_name(i['attributes']['path']): i for i in vm_devices
                      ~~~~~~~~~~~~~~~^^^^^^^^
KeyError: 'path'
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 211, in call_method
    result = await self.middleware.call_with_audit(message['method'], serviceobj, methodobj, params, self)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1529, in call_with_audit
    result = await self._call(method, serviceobj, methodobj, params, app=app,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1460, in _call
    return await methodobj(*prepared_call.args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 179, in nf
    return await func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 49, in nf
    res = await f(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/vm/vm_devices.py", line 53, in disk_choices
    zvols = await self.middleware.call(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1629, in call
    return await self._call(
           ^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1468, in _call
    return await self._call_worker(name, *prepared_call.args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1474, in _call_worker
    return await self.run_in_proc(main_worker, name, args, job)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1380, in run_in_proc
    return await self.run_in_executor(self.__procpool, method, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1364, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
KeyError: 'path'

Not familiar with SCALE.

Is it correct that the dataset is <poolname>/ix-apps, but the mountpoint uses /mnt/.ix-apps without the pool name in the path?

I agree it sounds odd, but ls /mnt/.ix-apps returns content.
This is system, I don’t think anything could get wrong here.

I’m looking in the middlewared project to understand what’s the missing path info.
Looking for plugins/zfs_/dataset_encryption.py in https://github.com/truenas/middleware.git, it seems that master has diverged a great deal from what is found on 24.10.2 already.

Branch release/24.10.2 has matching code.

vm_devices = self.middleware.call_sync('vm.device.query', [['dtype', '=', 'DISK']])
vm_zvols = {
    zvol_path_to_name(i['attributes']['path']): i for i in vm_devices
}

I could be wrong but it seems that self.middleware.call_sync('vm.device.query', [['dtype', '=', 'DISK']]) is querying db table vm_device, filtering on field dtype.

Looking into 20250302.db that I used to restore conf, I can see an attributes field. But the content looks like this:
gS5B9pn0w7sD1J9uZjk3hwtSrgdilFcAiezeZApe2RhEendCFRsbJQZUD+7EbfRy6nd4qNNr1RzUmDy/Hc02KBrDI9/rJpqwhHtAJ98UQpyXHjsriaG0Oq7NCLo1ufYdy+v/5w2D5n/Ro2PiQ+1QnOcoiNo2zxE/59O0cygW24KkZUES9x4L1HBTqUIOysPvt5H3vMHIScF+PG2qnIaLk6wEIIkNVxMq

I don’t see how zvol_path_to_name(i['attributes']['path']): i for i in vm_devices could retrieve a path value from it.

I’m probably wrong but it sounds like there’s a mismatch between the middlewared version and the database one.

Where’s the “live” database on Truenas system?

Should be under /data/, but I wouldn’t go messing around with system files manually.

Maybe this deserves a bug report? You might be right that the old pools, which no longer exist, are still being referenced by the middleware.

I just wondered if I could check what was actually read when I click on Datasets in the GUI.
But I think I’m on the wrong track anyway, I stumble upon the sqlachemy model of vm_device:


class VMDeviceModel(sa.Model):
    __tablename__ = 'vm_device'

    id = sa.Column(sa.Integer(), primary_key=True)
    dtype = sa.Column(sa.String(50))
    attributes = sa.Column(sa.JSON(encrypted=True))
    vm_id = sa.Column(sa.ForeignKey('vm_vm.id'), index=True)
    order = sa.Column(sa.Integer(), nullable=True)

attributes is encrypted, hence the odd value.
It could contain anything, it’s hard to understand how to decrypt and see what’s really in there.

Where would I fill in a bug?

Thinking of it, maybe I haven’t restored the proper secret so that middlewared can decrypt the content of the db.

I checked /data/freenas-v1.db and it’s exactly the content of 20250302.db I restored, or at least it seems to be.
It looks clean, having exactly the amount of DISK device that are attached to my 2 VMs.

In 'Manage configuration ’ > ‘Upload’, I see

All passwords are reset when the uploaded configuration database file was saved without the Password Secret Seed.

Could it be that i was missing the pwenc_secret Secret Seed?
I would have expected this to serve other purposes than encrypting VMs’ DISK attributes but if it’s what is missing, I could package a tar file with latest 20250302.db and an older pwenc_secret that I have downloaded.

So far looking at a couple of conf backup including the Password Secret Seed, the content is the same.

What would actually be expected to be missing when you restore a conf WITHOUT the Password Secret Seed?
AFAICS everything looks fine without it, except maybe the VMs DISK attributes…

I believe it also obfuscates other sensitive information, such as dataset encryption keys. The keystrings are not saved as plaintext in the database.

I always check “Export Secret Seed” whenever I save my config file, so I wouldn’t know the implications of trying to restore a TrueNAS system without one.