Lost data(?) on 25.10.1

WootWoot · July 25, 2026, 2:16am

Please help!

$ sudo zfs list -r pool-data-03 -o name,used | grep -i "media"pool-data-03/dataset-media-001 395K

TrueNASVersion: 25.10.1 - Goldeye

My pool-data-03/dataset-media-001 now says it’s empty, up until before some recent troubleshooting on a different issue, described below, I had well over 1000 movies, a number of episodes, etc organized into folders. I have a backup from a March 25 (months ago), when I migrated to a bare-metal TrueNAS from TruNAS VM - and that data is still in tact via the VM, but I’m hoping I don’t have to re-migrate everything and re-acquire 3 months of content (I actually have a text list of these files.) and re-organize it.

I was using Google Gemini and working on building/fixing an ollama container running on top of my TrueNAS… AI had me do something it called a lazy unmount that may have cause this? among other possibilities that I thought I was mostly understanding at the time? Further, I’m not even certain the AI related commands caused this and could have been cause by something else?

There are several datasets, only this one was affected. In spite of the fact that the containers are on a different dataset… err.. at least NOT in that dataset (dataset-media-001).

Interestingly, all my other datasets are fine… just this one is a problem and it happens to have been the largest. It inherits encryption from it’s parent dataset, and the rest of the datasets are fine…

Regrettably I have no snapshots, I’m pretty sure, unless that’s automatic.

pool-data-03
Pool Status: ONLINE
Used Space:14.53%
Disks with Errors: 0 of 4
Last Scrub:
Free Space: 31.52 TiB
Total Disks: 4
Data: 1 vdev
Caches: 0
Spares: 0

The data missing is from a child dataset of the above.
pool-data-03/dataset-homes-01 32T 512K 32T 1% /mnt/pool-data-03/dataset-homes-01
pool-data-03/dataset-software-01 33T 1.1T 32T 4% /mnt/pool-data-03/dataset-software-01
pool-data-03/apps-chat-01 150G 384K 150G 1% /mnt/pool-data-03/apps-chat-01
pool-data-03/dataset-nextcloud-01 33T 532G 32T 2% /mnt/pool-data-03/dataset-nextcloud-01
pool-data-03/dataset-media-001 32T 512K 32T 1% /mnt/pool-data-03/dataset-media-001

What other data do you need from me to assist?

Here are my goals:

Restore the data if possible (?) if it’s just some kind of encryption/bad mount/something? Perhaps a list of commands/checks I can do?
Find out what cause this/how this happened…
If step 1 is not an option, remigrate and re-download. (this I can do on my own, easily, though time consuming)

winnielinnie · July 25, 2026, 2:46am

That’s a convoluted way to do this:

zfs list -o space pool-data-03/dataset-media-001

Do you still have those commands? Did you save your chat history? It might be helpful to troubleshoot with.

These can help. Please use preformatted text when you paste the output. Put them in their own boxes to make it easier for everyone to read.

zpool status -v pool-data-03

zpool list pool-data-03

zfs list -r -t fs,vol -o space,mountpoint pool-data-03

zfs mount

zpool history pool-data-03 | grep 'rename\|destroy' | grep 'dataset-media-001'

Use sudo if needed.

WootWoot · July 25, 2026, 4:37am

username@server:~$ zfs list -o space pool-data-03/dataset-media-001

NAME                            AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD
pool-data-03/dataset-media-001  31.5T   395K        0B    395K             0B         0B

username@server:~$ zpool status -v pool-data-03

  pool: pool-data-03
 state: ONLINE
  scan: scrub repaired 0B in 02:31:28 with 0 errors on Sun Jul 19 02:31:30 2026
config:

        NAME                                      STATE     READ WRITE CKSUM
        pool-data-03                              ONLINE       0     0     0
          raidz1-0                                ONLINE       0     0     0
            13a0551c-7684-4efd-9830-c91f846b30e7  ONLINE       0     0     0
            beaf27a1-fd08-4493-b021-320df04a349c  ONLINE       0     0     0
            56485580-c6e2-492d-9536-f33b7a171a2b  ONLINE       0     0     0
            f91b33b1-6461-4cbc-b3e9-f9cb00041385  ONLINE       0     0     0
errors: No known data errors

username@server:~$ zpool list pool-data-03

NAME           SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
pool-data-03  50.9T  7.37T  43.5T        -         -     0%    14%  1.00x    ONLINE  /mnt

username@server:~$ zfs list -r -t fs,vol -o space,mountpoint pool-data-03

NAME                                                           AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD  MOUNTPOINT
pool-data-03                                                   31.5T  5.36T        0B   3.61T             0B      1.75T  /mnt/pool-data-03
pool-data-03/.system                                           31.5T  2.29G        0B   1.82G             0B       481M  legacy
pool-data-03/.system/configs-[ID-000000000000000]  	       31.5T  16.7M        0B   16.7M             0B         0B  legacy
pool-data-03/.system/cores                                      921M   103M        0B    103M             0B         0B  legacy
pool-data-03/.system/netdata-[ID-000000000000000]	       31.5T   359M        0B    359M             0B         0B  legacy
pool-data-03/.system/nfs                                       31.5T   360K        0B    360K             0B         0B  legacy
pool-data-03/.system/samba4                                    31.5T   889K        0B    889K             0B         0B  legacy
pool-data-03/.system/vm                                        31.5T   279K        0B    279K             0B         0B  legacy
pool-data-03/apps-chat-01                                       150G   790K        0B    279K             0B       512K  /mnt/pool-data-03/apps-chat-01
pool-data-03/apps-chat-01/ollama-models                        31.5T   256K        0B    256K             0B         0B  /mnt/pool-data-03/apps-chat-01/ollama-models
pool-data-03/apps-chat-01/openclaw01                           31.5T   256K        0B    256K             0B         0B  /mnt/pool-data-03/apps-chat-01/openclaw01
pool-data-03/dataset-homes-01                                  31.5T   477K        0B    477K             0B         0B  /mnt/pool-data-03/dataset-homes-01
pool-data-03/dataset-media-001                                 31.5T   395K        0B    395K             0B         0B  /mnt/pool-data-03/dataset-media-001
pool-data-03/dataset-nextcloud-01                              31.5T   532G        0B    532G             0B         0B  /mnt/pool-data-03/dataset-nextcloud-01
pool-data-03/dataset-software-01                               31.5T  1.21T      204G   1.01T             0B         0B  /mnt/pool-data-03/dataset-software-01
pool-data-03/ix-apps                                           31.5T  16.9G        0B    186K             0B      16.9G  /mnt/.ix-apps
pool-data-03/ix-apps/app_configs                               31.5T   192K        0B    192K             0B         0B  /mnt/.ix-apps/app_configs
pool-data-03/ix-apps/app_mounts                                31.5T   140K        0B    140K             0B         0B  /mnt/.ix-apps/app_mounts
pool-data-03/ix-apps/docker                                    31.5T  16.4G        0B   16.4G             0B         0B  /mnt/.ix-apps/docker
pool-data-03/ix-apps/truenas_catalog                           31.5T   520M        0B    520M             0B         0B  /mnt/.ix-apps/truenas_catalog

username@server:~$ zfs mount

boot-pool/ROOT/25.10.1          /
boot-pool/ROOT/25.10.1/audit    /audit
boot-pool/ROOT/25.10.1/conf     /conf
boot-pool/ROOT/25.10.1/data     /data
boot-pool/ROOT/25.10.1/etc      /etc
boot-pool/ROOT/25.10.1/home     /home
boot-pool/ROOT/25.10.1/mnt      /mnt
boot-pool/ROOT/25.10.1/opt      /opt
boot-pool/ROOT/25.10.1/root     /root
boot-pool/ROOT/25.10.1/usr      /usr
boot-pool/ROOT/25.10.1/var      /var
boot-pool/ROOT/25.10.1/var/ca-certificates  /var/local/ca-certificates
boot-pool/ROOT/25.10.1/var/lib  /var/lib
boot-pool/ROOT/25.10.1/var/lib/incus  /var/lib/incus
boot-pool/ROOT/25.10.1/var/log  /var/log
boot-pool/ROOT/25.10.1/var/log/journal  /var/log/journal
boot-pool/grub                  /boot/grub
pool-data-03                    /mnt/pool-data-03
pool-data-03/dataset-homes-01   /mnt/pool-data-03/dataset-homes-01
pool-data-03/dataset-software-01  /mnt/pool-data-03/dataset-software-01
pool-data-03/apps-chat-01       /mnt/pool-data-03/apps-chat-01
pool-data-03/apps-chat-01/openclaw01  /mnt/pool-data-03/apps-chat-01/openclaw01
pool-data-03/apps-chat-01/ollama-models  /mnt/pool-data-03/apps-chat-01/ollama-models
pool-data-03/dataset-nextcloud-01  /mnt/pool-data-03/dataset-nextcloud-01
pool-data-03/.system            /var/db/system
pool-data-03/.system/cores      /var/db/system/cores
pool-data-03/.system/nfs        /var/db/system/nfs
pool-data-03/.system/samba4     /var/db/system/samba4
pool-data-03/.system/vm         /var/db/system/vm
pool-data-03/.system/configs-[ID]  /var/db/system/configs-[ID]
pool-data-03/.system/netdata-[ID]  /var/db/system/netdata
pool-data-03/.system/cores      /var/lib/systemd/coredump
pool-data-03/ix-apps            /mnt/.ix-apps
pool-data-03/ix-apps/app_configs  /mnt/.ix-apps/app_configs
pool-data-03/ix-apps/truenas_catalog  /mnt/.ix-apps/truenas_catalog
pool-data-03/ix-apps/docker     /mnt/.ix-apps/docker
pool-data-03/ix-apps/app_mounts  /mnt/.ix-apps/app_mounts
pool-data-03/dataset-media-001  /mnt/pool-data-03/dataset-media-001

username@server:~$ sudo zpool history pool-data-03 | grep ‘rename’ | grep ‘dataset-media-001’

[no output]

username@server:~$ sudo zpool history pool-data-03 | grep ‘destroy’ | grep ‘dataset-media-001’

[no ouput]

WootWoot · July 25, 2026, 4:39am

I asked in the same AI chat (that started as troubleshooting ollama, and then trying to recover the lost data/dataset):
Can you output a history of all the commands you had me execute on this server? output but in the order it was executed?


# 1. Initial attempt to fix CDI driver paths for the custom App
sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml

# 2. Checking if the tool successfully populated the GPU device array
sudo nvidia-ctk cdi list

# 3. Restarting the system Docker service to read the new path
sudo systemctl restart docker

# 4. Attempting alternative TrueNAS/Podman directory mappings when list returned 0
sudo mkdir -p /var/run/cdi
sudo nvidia-ctk cdi generate --output=/var/run/cdi/nvidia.yaml
sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.json

# 5. Overriding the naming strategy to force standard indexing labels
sudo nvidia-ctk cdi generate --device-name-strategy=type-index --output=/etc/cdi/nvidia.yaml
sudo nvidia-ctk cdi generate --device-name-strategy=type-index --output=/var/run/cdi/nvidia.yaml

# 6. Re-checking the populated devices
sudo nvidia-ctk cdi list

# 7. Attempting to reload the host container manager daemon
sudo systemctl restart docker

# 8. Forcing the legacy TrueNAS app registry to re-evaluate system hardware topology
midclt call app.kubernetes.update_gpu_information

# 9. Sending a HUP signal to dockerd to reload configurations without dropping container links
sudo kill -s SIGHUP $(pidof dockerd)

# 10. Querying the physical hardware configuration to find the active video card modules
lspci -nnk | grep -A 3 -i vga

# 11. Trying to remove the competing open-source nouveau module from memory
sudo modprobe -r nouveau 2>/dev/null
sudo echo "blacklist nouveau" | sudo tee /etc/modprobe.d/blacklist-nouveau.conf
sudo rmmod -f nouveau

# 12. Forcing the official nvidia driver module to manually claim the card
sudo modprobe nvidia

# 13. Retrying the legacy TrueNAS middleware system sync
midclt call app.kubernetes.update_gpu_information

# 14. Generating the CDI definitions again now that the driver was bound
sudo nvidia-ctk cdi generate --device-name-strategy=type-index --output=/var/run/cdi/nvidia.yaml

# 15. Requesting the explicit hardware registration keys from the TrueNAS API
midclt call app.gpu_choices | jq .

# 16. Attempting a middleware payload injection to assign the GPU via long-form flag
midclt call -job app.update ollama-openclaw-01 '{"values": {"resources": {"gpus": {"use_all_gpus": false, "nvidia_gpu_selection": {"YOUR_PCI_SLOT_HERE": {"use_gpu": true, "uuid": "YOUR_GPU_UUID_HERE"}}}}}}}'

# 17. Attempting the payload parameter using the shortened job token syntax
midclt call -j app.update 'ollama-openclaw-01' '{"values": {"resources": {"gpus": {"use_all_gpus": false, "nvidia_gpu_selection": {"0000:01:00.0": {"use_gpu": true, "uuid": "YOUR_GPU_UUID_HERE"}}}}}}}'

# 18. Tracking background update jobs in the middleware queue
midclt call core.get_jobs '[["id", "=", 42]]' | jq ..state

# 19. Initializing the application bundle from the shell
midclt call app.start ollama-openclaw-01

# 20. Resending the payload parameters using strict dictionary separation strings
midclt call -j app.update "ollama-openclaw-01" '{"values": {"resources": {"gpus": {"use_all_gpus": false, "nvidia_gpu_selection": {"0000:01:00.0": {"use_gpu": true, "uuid": "YOUR_GPU_UUID_HERE"}}}}}}}'

# 21. Attempting another shell-level deployment execution
midclt call app.start "ollama-openclaw-01"

# 22. Checking paths to find hidden system configuration templates
cd /mnt/ix-applications/releases/ollama-openclaw-01/charts/
sudo nano values.yaml

# 23. Attempting to force-start the container infrastructure stack
midclt call app.restart ollama-openclaw-01

# 24. Reading directory variables from the base mount layer
ls -al

# 25. Reviewing the hidden configuration tree mappings
cd /mnt/.ix-apps/app_configs/ollama-openclaw-01/
sudo nano metadata.yaml

# 26. Attempting to modify hidden template definitions inside the version folders
sudo nano /mnt/.ix-apps/app_configs/ollama-openclaw-01/compose.yaml
cd /mnt/.ix-apps/app_configs/ollama-openclaw-01/versions/*/templates/
sudo nano docker-compose.yaml

# 27. Retrying the app deployment cycle
midclt call app.restart "ollama-openclaw-01"

# 28. Attempting to explicitly command separate app terminations and executions
midclt call app.stop "ollama-openclaw-01"
midclt call app.start "ollama-openclaw-01"

# 29. Force-killing active container hooks to unlock stuck middleware actions
sudo docker rm -f ollama-brain open-webui openclaw-agent 2>/dev/null
sudo docker network rm ix-ollama-openclaw-01_ai-internal 2>/dev/null

# 30. Attempting to clear the cached state engine data by restarting app structures
sudo systemctl restart truenas-apps

# 31. Running a terminal command to clean stuck threads
sudo ps -EF | grep -i "ollama" | awk '{print $2}' | xargs sudo kill -9 2>/dev/null

# 32. Detaching mount locks via lazy unmounting
sudo umount -l /mnt/.ix-apps/app_configs 2>/dev/null

# 33. Cleaning ghost loops via native process tools
sudo pkill -9 -f "ollama"
sudo pkill -9 -f "open-webui"

# 34. Jumping straight to the template paths and launching with raw Docker Compose
cd /mnt/.ix-apps/app_configs/ollama-openclaw-01/versions/*/templates/
sudo docker compose up -d
sudo docker ps

# 35. Checking why paths appeared unmounted
ls -al /mnt/tank/apps/

# 36. Attempting to restore unlinked ZFS file system mount points
sudo mount -a
sudo zfs mount -a

# 37. Creating an isolated, independent configuration folder on the persistent pool
sudo mkdir -p /mnt/tank/apps/ollama-openclaw/
sudo nano /mnt/tank/apps/ollama-openclaw/docker-compose.yaml
sudo docker compose up -d

# 38. Re-checking the persistent base paths
ls -al

# 39. Creating the final isolated tracking space on the primary pool
sudo mkdir -p /mnt/tank/apps/ollama-openclaw-compose/
sudo nano /mnt/tank/apps/ollama-openclaw-compose/docker-compose.yaml

# 40. Forcing the detached deployment through specific runtime engine sockets
cd /mnt/tank/apps/ollama-openclaw-compose/
sudo DOCKER_HOST=unix:///var/run/docker.sock docker compose up -d
sudo DOCKER_HOST=unix:///var/run/truenas-apps/docker.sock docker compose up -d
sudo DOCKER_HOST=unix:///var/run/truenas-apps/docker.sock docker ps

# 41. Verifying if the master docker engine socket was alive
sudo docker info
sudo find /var/run/ -name "*docker*.sock"

# 42. Force-restarting the host engine and wiping the network state lock files
sudo systemctl restart local-fs.target
sudo rm -f /mnt/.ix-apps/docker/network/files/local-kv.db
sudo systemctl restart containerd
sudo systemctl restart middlewared

# 43. Triggering a full system reboot to clear deadlocks
sudo reboot

# 44. Post-reboot container stack deployment execution
sudo docker ps
cd /mnt/tank/apps/ollama-openclaw-compose/
sudo docker compose up -d
sudo docker ps

# 45. Verifying hardware utilization
nvidia-smi

# 46. Re-editing the composition rules to handle device passthroughs
sudo nano /mnt/tank/apps/ollama-openclaw-compose/docker-compose.yaml
cd /mnt/tank/apps/ollama-openclaw-compose/
sudo docker compose up -d

# 47. Diagnostic logs analysis
sudo docker logs ollama-brain --tail 20

# 48. Editing composition variables to toggle privileged features
sudo nano /mnt/tank/apps/ollama-openclaw-compose/docker-compose.yaml
cd /mnt/tank/apps/ollama-openclaw-compose/
sudo docker compose up -d

# 49. Reviewing model acceleration steps
nvidia-smi

# 50. Re-editing configuration volume arrays to pass library paths
sudo nano /mnt/tank/apps/ollama-openclaw-compose/docker-compose.yaml
cd /mnt/tank/apps/ollama-openclaw-compose/
sudo docker compose up -d

# 51. Clearing container mount configurations
sudo docker rm -f ollama-brain 2>/dev/null
cd /mnt/tank/apps/ollama-openclaw-compose/
sudo docker compose up -d

# 52. Checking hardware processing parameters
nvidia-smi

# 53. Removing environmental port variables and adding local URL binders
sudo nano /mnt/tank/apps/ollama-openclaw-compose/docker-compose.yaml
cd /mnt/tank/apps/ollama-openclaw-compose/
sudo docker compose down
sudo docker compose up -d

# 54. Verifying token utilization logs
nvidia-smi

# 55. Adjusting image tags to use structural environment settings
sudo nano /mnt/tank/apps/ollama-openclaw-compose/docker-compose.yaml
cd /mnt/tank/apps/ollama-openclaw-compose/
sudo docker compose down
sudo docker compose up -d
nvidia-smi

# 56. Appending library path environments to resolve internal compilation loops
sudo nano /mnt/tank/apps/ollama-openclaw-compose/docker-compose.yaml
cd /mnt/tank/apps/ollama-openclaw-compose/
sudo docker compose down
sudo docker compose up -d
nvidia-smi

# 57. Pulling initialization trace frames from the Ollama debug engine
sudo docker logs ollama-brain 2>&1 |加载-i -E "gpu|cuda|nvidia|detect" | tail -n 25

# 58. Adding watchdog timeout overrides to the environment configuration
sudo nano /mnt/tank/apps/ollama-openclaw-compose/docker-compose.yaml
cd /mnt/tank/apps/ollama-openclaw-compose/
sudo docker compose down
sudo docker compose up -d
nvidia-smi

# 59. Scanning active runtime trace logs
sudo docker logs ollama-brain 2>&1 | tail -n 30

# 60. Tracking missing storage pools and scanning dataset structures
ls -al /mnt/tank/apps/
sudo zfs list -r tank
sudo find /mnt/ -maxdepth 4 -type d -name "ollama"
sudo zfs list -o name,mounted,mountpoint

# 61. Attempting explicit unlinks and mounts on the media dataset folder
sudo zfs mount POOL_NAME/dataset-media-001
sudo zfs list pool-data-03/dataset-media-001
sudo zfs list -r pool-data-03 -o name,used,mounted,mountpoint
sudo zfs list -r pool-data-03/dataset-media-001

# 62. Terminating active containers to check for overlapping namespace masks
sudo docker stop $(sudo docker ps -q) 2>/dev/null
sudo umount -f /mnt/pool-data-03/dataset-media-001
sudo zfs mount pool-data-03/dataset-media-001

# 63. Querying system-wide snapshot allocations
sudo zfs list -t snapshot -r pool-data-03/dataset-media-001

# 64. Forcing alternative overlay configurations
sudo zfs mount -O pool-data-03/dataset-media-001

# 65. Stripping zombie placeholder layers using lazy unmount flags on the media directory
sudo umount -f -l /mnt/pool-data-03/dataset-media-001

sudo rm -rf /mnt/pool-data-03/dataset-media-001  #note that dataset-media-001 was empty already at this time.
sudo mkdir -p /mnt/pool-data-03/dataset-media-001
sudo zfs mount pool-data-03/dataset-media-001

# 66. Reading filesystem capacities
df -h /mnt/pool-data-03/dataset-media-001
df -h | grep media

# 67. Wiping empty targets and forcing recursive mounts
sudo zfs unmount -f pool-data-03/dataset-media-001
sudo rm -rf /mnt/pool-data-03/dataset-media-001 #note that dataset-media-001 was empty already at this time.
sudo mkdir -p /mnt/pool-data-03/dataset-media-001
sudo zfs mount pool-data-03/dataset-media-001
df -h /mnt/pool-data-03/dataset-media-001
ls -la /mnt/pool-data-03/dataset-media-001

# 68. Clearing the cryptographic layer memory cache keys
sudo zfs unmount -f pool-data-03/dataset-media-001
sudo zfs unload-key pool-data-03/dataset-media-001
sudo rm -rf /mnt/pool-data-03/dataset-media-001	 #note that dataset-media-001 was empty already at this time.
sudo mkdir -p /mnt/pool-data-03/dataset-media-001
sudo zfs load-key pool-data-03/dataset-media-001
sudo zfs mount pool-data-03/dataset-media-001
df -h /mnt/pool-data-03/dataset-media-001

69. Attempting to unlock the master root storage pool key configuration
sudo zfs load-key pool-data-03sudo zfs mount pool-data-03/dataset-media-001
df -h /mnt/pool-data-03/dataset-media-001
ls -al /mnt/pool-data-03/dataset-media-001

70. Removing the blocker directory entirely and triggering automated pool mountings
sudo zfs unmount -f pool-data-03/dataset-media-001 2>/dev/null
sudo rm -rf /mnt/pool-data-03/dataset-media-001	 #note that dataset-media-001 was empty already at this time.
sudo zfs mount -asudo zfs mount pool-data-03/dataset-media-001
df -h /mnt/pool-data-03/dataset-media-001

71. Attempting another shadow unmount iteration
sudo umount -f -l /mnt/pool-data-03/dataset-media-001
sudo rm -rf /mnt/pool-data-03dataset-media-001 #note that dataset-media-001 was empty already at this time.
sudo mkdir -p /mnt/pool-data-03/dataset-media-001
sudo zfs mount pool-data-03/dataset-media-001
df -h | grep media

72. Querying master dataset snapshots and performing casing-agnostic searches
sudo zfs list -t snapshot -r pool-data-03
sudo zfs list -r pool-data-03 -o name,used | grep -i "media"

WootWoot · July 25, 2026, 4:58am

A few irrelevant lines were removed, I’m certain nothing that will effect the analysis.

$ history 200

  742  sudo tail -n 50 /var/log/app_lifecycle.log
  743  sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml
  744  sudo nvidia-ctk cdi list
  746  sleep 120 && sleep 20
  747  lspci -nnk | grep -A 3 -i vga
  748  sudo modprobe -r nouveau 2>/dev/null
  749  sudo echo "blacklist nouveau" | sudo tee /etc/modprobe.d/blacklist-nouveau.conf
  750  sudo rmmod -f nouveau
  751  midclt call app.kubernetes.update_gpu_information
  752  midclt call app.gpu_choices | jq .
  753  midclt call -job app.update ollama-openclaw-01 '{"values": {"resources": {"gpus": {"use_all_gpus": false, "nvidia_gpu_selection": {"0000:01:00.0": {"use_gpu": true, "uuid": "[GPU-ID-000000000000000000000000]"}}}}}}}'
  754  midclt call -j app.update 'ollama-openclaw-01' '{"values": {"resources": {"gpus": {"use_all_gpus": false, "nvidia_gpu_selection": {"0000:01:00.0": {"use_gpu": true, "uuid": "[GPU-ID-000000000000000000000000]"}}}}}}}'
  755  midclt call -j app.update "ollama-openclaw-01" '{"values": {"resources": {"gpus": {"use_all_gpus": false, "nvidia_gpu_selection": {"0000:01:00.0": {"use_gpu": true, "uuid": "[GPU-ID-000000000000000000000000]"}}}}}}}'
  757  cd /mnt/ix-applications/releases/ollama-openclaw-01/charts/
  760  sudo find /mnt/.ix-apps/ -name charts
  763  cd /mnt/tank/apps/
  764  sudo find . -name charts
  765  cd /mnt
  767  ls -al
  768  cd /mnt/.ix-apps/app_configs/ollama-openclaw-01/
  769  sudo nano metadata.yaml
  770  sudo nano /mnt/.ix-apps/app_configs/ollama-openclaw-01/compose.yaml
  771  cd /mnt/.ix-apps/app_configs/ollama-openclaw-01/versions/
  774  cd 1.0.0/
  777  cd templates/
  779  cd rendered/
  781  sudo nano docker-compose.yaml
  783  midclt call app.stop "ollama-openclaw-01"
  784  midclt call app.start "ollama-openclaw-01"
  785  sudo docker rm -f ollama-brain open-webui openclaw-agent 2>/dev/null
  786  sudo docker network rm ix-ollama-openclaw-01_ai-internal 2>/dev/null
  787  sudo systemctl restart truenas-apps
  788  sudo systemctl restart middlewared
  789  sudo ps -EF | grep -i "ollama" | awk '{print $2}' | xargs sudo kill -9 2>/dev/null
  790  sudo ps -EF | grep -i "open-webui" | awk '{print $2}' | xargs sudo kill -9 2>/dev/null
  791  sudo ps -EF | grep -i "ollama" | awk '{print $2}' | xargs sudo kill -9 2>/dev/null
  792  sudo pkill -9 -f "ollama"
  793  sudo pkill -9 -f "open-webui"
  794  sudo umount -l /mnt/.ix-apps/app_configs 2>/dev/null
  795  cd /mnt/.ix-apps/app_configs/ollama-openclaw-01/versio
  796  cd /mnt/.ix-apps/app_configs/
  797  ll
  798  ls -al
  799  cd ..
  800  sudo mount -a
  801  cd /mnt/.ix-apps/app_configs/
  802  ls -al
  803  sudo zfs mount -a
  804  ls -al
  805  cd /mnt/tank/apps/
  806  ls -al
  807  sudo mkdir -p /mnt/tank/apps/ai-apps-jsd/
  808  touch ai-apps-jsd/2026-07-23_manually-created.txt
  809  sudo touch ai-apps-jsd/2026-07-23_manually-created.txt
  810  cd ai-apps-jsd/
  811  sudo nano docker-compose.yaml
  812  ll
  813  ls -al
  814  sudo docker compose up -d
  815  sudo DOCKER_HOST=unix:///var/run/docker.sock docker compose up -d
  816  sudo systemctl restart middlewared
  817  sleep 20
  818  sudo docker info
  819  sudo reboot
  820  cd /mnt/pool-data-03/dataset-media-001/media/
  821  ll
  822  ls -al
  823  cd ..
  824  sudo zfs mount -a
  825  cd media/
  826  ll
  827  ls -al
  828  cd ..
  829  sudo zfs list dataset-media-001
  830  sudo zfs list pool-data-03/dataset-media-001
  831  sudo zfs list -r pool-data-03/dataset-media-001
  832  cd ..
  833  ll
  834  ls -al
  835  sudo zfs list -t snapshot -r pool-data-03/dataset-media-001
  836  sudo zfs mount pool-data-03/dataset-media-001
  837  sudo umount -l /mnt/pool-data-03/dataset-media-001
  838  sudo zfs mount pool-data-03/dataset-media-001
  839  df -h /mnt/pool-data-03/dataset-media-001
  840  sudo zfs unmount -f pool-data-03/dataset-media-001
  841  sudo mv /mnt/pool-data-03/dataset-media-001/ /mnt/pool-data-03/dataset-media-001_BAD
  842  sudo mkdir -p /mnt/pool-data-03/dataset-media-001
  843  sudo zfs mount pool-data-03/dataset-media-001
  844  df -h /mnt/pool-data-03/dataset-media-001
  845  zfs list
  846  sudo reboot
  847  sudo docker ps
  848  cd /mnt/tank/apps/
  849  ll
  850  ls -al
  851  cd ai-apps-jsd/
  852  ll
  853  ls -al
  854  sudo docker compose up -d
  855  docker ps
  856  sudo docker ps
  857  cat docker-compose.yaml
  858  sudo docker ps
  859  sleep 30 && sudo docker ps
  860  nvidia-smi
  861  sudo nano /mnt/tank/apps/ollama-openclaw-compose/docker-compose.yaml
  862  :q
  863  #sudo nano docker-compose.yaml
  864  sudo nano docker-compose.yaml
  865  sudo docker compose down
  866  sudo docker compose up -d
  867  sudo docker compose down
  868  sudo nano docker-compose.yaml
  869  sudo docker compose up -d
  870  sudo docker compose down
  871  sudo nano docker-compose.yaml
  872  sudo docker compose up -d
  873  nvidia-smi
  874  sudo nano docker-compose.yaml
  875  sudo ls -al /usr/lib/x86_64-linux-gnu/libnvidia-ml.so.570.172.08
  876  sudo docker compose down
  877  sudo docker compose up -d
  878  sudo nano docker-compose.yaml
  879  sudo docker rm -f ollama-brain 2>/dev/null
  880  sudo docker compose up -d
  881  sudo docker compose down
  882  sudo nano docker-compose.yaml
  883  sudo docker compose up -d
  884  sudo nano docker-compose.yaml
  885  sudo docker compose down
  886  sudo docker compose up -d
  887  sudo docker compose down
  888  sudo nano docker-compose.yaml
  889  sudo docker compose up -d
  890  nvidia-smi
  891  sudo nano docker-compose.yaml
  892  sudo docker compose down
  893  sudo docker compose up -d
  894  nvidia-smi
  895  sudo docker logs ollama-brain 2>&1 | grep -i -E "gpu|cuda|nvidia|detect"
  896  sudo docker compose down
  897  sudo docker compose up -d
  898  sudo docker compose down
  899  sudo nano docker-compose.yaml
  900  sudo docker compose up -d
  901  sleep 10
  902  nvidia-smi
  903  sudo docker logs ollama-brain 2>&1 | tail -n 30
  904  watch -n 0.1 nvidia-smi
  905  sudo zfs load-key pool-data-03/dataset-media-001
  906  sudo zfs load-key pool-data-03
  907  sudo zfs unmount -f pool-data-03/dataset-media-001 2>/dev/null
  908  df -h
  909  df -h | grep media
  910  sudo rm -rf /mnt/pool-data-03/dataset-media-001
  911  sudo zfs mount pool-data-03/dataset-media-001
  912  df -h | grep media
  913  sudo umount -f -l /mnt/pool-data-03/dataset-media-001
  914  ls -al /mnt/pool-data-03/dataset-media-001
  915  sudo rm -rf /mnt/pool-data-03/dataset-media-001
  916  sudo mkdir -p /mnt/pool-data-03/dataset-media-001
  917  sudo zfs mount pool-data-03/dataset-media-001
  918  df -h | grep media
  919  sudo zfs list -t snapshot -r pool-data-03
  920  sudo zfs list -r pool-data-03 -o name,used | grep -i "media"
  921  storage dataset query
  922  midclt call pool.dataset.query
  923  df -h
  924  df -h | grep pool-data-03
  925  zfs list -o space pool-data-03/dataset-media-001
  926  zpool status -v pool-data-03
  927  zpool list pool-data-03
  928  zfs list -r -t fs,vol -o space,mountpoint pool-data-03
  929  zfs mount
  930  zpool history pool-data-03 | grep 'rename\|destroy' | grep 'dataset-media-001'
  931  sudo zpool history pool-data-03 | grep 'rename\|destroy' | grep 'dataset-media-001'
  932  sudo zpool history pool-data-03 | grep 'rename' | grep 'dataset-media-001'
  933  sudo zpool history pool-data-03 | grep 'destroy' | grep 'dataset-media-001'
  935  history 200

WootWoot · July 25, 2026, 4:59am

Note:

The data directory was renamed to dataset-media-001_BAD before rm -rf was used. Note it was already ‘empty’ at this point, but AI wanted me to rm -rf it, so I moved it instead.

As a side note, the AI thinks:

You are currently struggling with a ZFS cryptographic layer issue (load-key). I dont think this is true, as I’m pretty sure this dataset inherits encryption from it’s parent, and the other child datasets are working properly.

winnielinnie · July 25, 2026, 12:39pm

The dataset pool-data-03/dataset-media-001 is currently empty.

The pool is consuming 7.4 TB of space. The root dataset of the pool has 3.6 TB directly saved to it.

Do you still have this directory and if so, how large is it? Something needs to account for the 3.6 TB on the root dataset.

du -hs /mnt/pool-data-03/dataset-media-001_BAD

If not, what is the output for this?

ls -1 /mnt/pool-data-03

We might have to unmount the empty dataset in order to reveal the missing files.

Looking at your commands, the AI seriously put your data at risk with rm and umount commands. I don’t think we’re at a point where AI can safely advise someone on dealing with esoteric goals with ZFS.

WootWoot · July 25, 2026, 1:39pm

Upon opening the dir, I can see all the data now, and it has a filesize! Strange it was not there before. It’s all back in the “_BAD” folder. In this case, BAD is GOOD!

$ du -hs /mnt/pool-data-03/dataset-media-001_BAD

3.7T    /mnt/pool-data-03/dataset-media-001_BAD

$ ls -al /mnt/pool-data-03/

total 140
...
drwxrwx--- 5 myuser myuser  6 May 10 23:29 dataset-media-001
drwxr-x--- 3 myuser myuser  3 May 20 23:06 dataset-media-001_BAD
...

Is the fix at this point as simple as (?):

mv dataset-media-001 dataset-media-001_empty_deletelater
mv dataset-media-001_BAD dataset-media-001

winnielinnie · July 25, 2026, 1:52pm

No.

Somehow the files currently live on the root dataset.

You need to move or copy them to the mountpoint path of the pool-data-03/dataset-media-001 dataset.

If the output you pasted earlier is still true, then you can move, copy, or rsync everything from the “BAD” folder to the live file-system of the mounted dataset.

Make a snapshot of the root dataset.

zfs snapshot pool-data-03@discard-me-later

Create a pool checkpoint.

zpool checkpoint pool-data-03

Copy, move, or rsync everything over. Rsync will let you confirm the operation with a dry run to make sure nothing was missed. Do not remove the trailing slashes in the command.

rsync -n -avhHxx --progress /mnt/pool-data-03/dataset-media-001_BAD/ /mnt/pool-data-03/dataset-media-001/

This was a dry run to crawl everything. Remove the -n flag and run the command again to actually transfer the files to the proper location.

After it succeeds, do another dry run (-n). Does it still show files that remain to be transferred? If not, you should be good.

Confirm your files exist in the new location. Confirm that they live on the dataset.

zfs list -t fs -o space pool-data-03/dataset-media-001

Now you can remove the old directory and everything inside.

<the rm command> -r /mnt/pool-data-03/dataset-media-001_BAD

I left out the actual command, -f, and sudo. You need to do these steps slowly and understand what is going to happen.

Do not remove the snapshot or checkpoint until you confirm everything is ok.

If everything is ok, you can remove the snapshot and checkpoint.

zfs destroy pool-data-03@discard-me-later
zpool checkpoint -d pool-data-03

EDIT: My steps seem redundant and inefficient, but it’s for a reason. It’s better for this to take longer, but with minimum risk of losing everything. That’s why I favor rsync over mv which results with two copies of everything. You then delete the old copy later. This gives you a chance to keep your data if something goes wrong.

etorix · July 25, 2026, 1:55pm

No because you’re mixing dataset and directory.
dataset-media-001 is a genuine dataset (think: partition); dataset-media-001_BAD is a directory in the root dataset. If you run these commands you will have a directory with the same name as a dataset: One of these will take precedence and prevent you from accessing the other.

You really should have 0 B saved to the root dataset and all data in child datasets under the root datset. So you should move the underlying data.

WootWoot · July 25, 2026, 1:59pm

THANK YOU FOR YOUR HELP!

rsync is migrating the data now, it’s gonna take a bit, but I’ll update when it’s finished.

winnielinnie · July 25, 2026, 2:00pm

I forgot to confirm. I know yesterday you pasted output that shows your dataset is currently mounted, but it’s not too late to double check. You can do this in another window:

zfs mount | grep dataset-media-001

This should show usage is increasing by now:

zfs list -t fs -o space pool-data-03/dataset-media-001

WootWoot · July 25, 2026, 2:04pm

$ zfs mount | grep dataset-media-001 # Yes, it would appear to be mounted

pool-data-03/dataset-media-001  /mnt/pool-data-03/dataset-media-001

$ zfs list -t fs -o space pool-data-03/dataset-media-001

NAME                            AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD
pool-data-03/dataset-media-001  31.5T  27.9G        0B   27.9G             0B         0B

winnielinnie · July 25, 2026, 2:06pm

Cancel everything. I just asked Google Gemini and it said the best solution is to ship me your NAS server and HDDs. I’ll DM you my shipping address.

I don’t question AI. I’m sorry. We don’t have choice.

WootWoot · July 25, 2026, 2:24pm

The NAS is actually just an old PC. You probably wouldn’t want it.

The drives are pretty swanky… and new as of this year.
4x WD Red Pro 14tb
…and one of those drives I got freeish via WD screwing up and saying sorry. I actually tested a series of 6 drives in total and TWO of the first set (of 3) came back with SMART errors (the drive not working kind) on the first run (NEW DRIVES)… after a series of months, talks with support, and trouble, I ended up with 4 drives at the price of 3, that said, it took ~5 months to get it all worked out.

It’s an old desktop with:
Intel Core i7-4770K (4th-generation “Haswell” )
32GB DDR3 RAM

Dropped in recently for network upgrades and adding the ai container:
NIC for higher end networking
GeForce RTX 3060 12GB GDDR6

LarsR · July 25, 2026, 2:26pm

Still feel I should run behind you holding up a big fat sarcams-sign…

etorix · July 25, 2026, 2:33pm

Given the current price of hard drives, it was worth the trouble.
Thanks for contributing your experience in support of some form of burn-in tests!

NickF1227 · July 25, 2026, 3:45pm

I think as a general lesson learned in this thread

Don’t listen to an AI that tells you to do an rm -rf
Don’t mount and unmount datasets on TrueNAS
Don’t give your AI root access to your TrueNAS
Don’t use bare ZFS commands, use the API Contents — TrueNAS API v25.10.4 documentation
For AI assisted workloads, use the TrueNAS MCP server so you can stay within the Middleware playground and reduce your likelihood of doing horribly wrong things TrueNAS MCP Server | TrueNAS Research Labs

etorix · July 25, 2026, 4:40pm

Is it just me, or do all the examples listed on the MCP pages relate to information you’d just see by logging into TrueNAS?

“What version of TrueNAS am I running?” sure requires the most advanced AI to find out…

NickF1227 · July 25, 2026, 5:14pm

Well, both the MCP server and the WebUI are bound by the same API. So from that perspective, their capabilities would inherently be similar. But the “interface” is geared towards an Agent, and so the usecase is entirely different.

My personal usecase for the use of AI in my homelab trascends just the TrueNAS MCP server, that’s just a piece of the puzzle. For all intents and purposes, I treat my hermes agent as if it were a JR sysadmin and my homelab is my production environment. It has access into my TrueNASs, my entire home network stack, several VMs and applications and it’s own dedicated Mac. It helps me automate processes, it helps me investigate issues, I can dictate actions to it and it can perform them. It’s like having a buddy helping me out.

For me personally, this is helpful. I’m still a relatively new dad, and the amount of time I have available to me for my computer hobby is alot smaller now. It’s been convenient to train my agent to do stuff for me so that I don’t have to do it.

There’s alot of danger in this workflow. Trusting your agent to do things that you yourself don’t fully understand can result in disaster. I think I have enough experience at this point in my career that I can steer things in the right direction, but even I struggle with my AI doing silly things. I personally treat this as a learning experience, both for me and my AI, because I see it more as a protege than as a SME. Sure there are things it knows way more about than I do, but it lacks context and intuition that I hold in my head. It’s a balancing act.

Being able to say, “hey buddy, build me a new virtual machine on XYZ TrueNAS node, install ABC software, and let me know when you are done” while I go make dinner is great. Then I get to spend time working on what I wanted to, rather than speniding time building out a VM and getting it into whatever state I need it in. I’ve done this enough times in my life, I don’t need to practice anymore

But that’s just my musings and opinon

Topic		Replies	Views
Apps service refuses to start no matter what after update TrueNAS General SCALE	42	188	July 6, 2026
All of my data gone TrueNAS General	13	365	November 16, 2024
Docker settings reset to defaults by 24.10.1 update TrueNAS General SCALE , Docker	12	937	December 19, 2024
Datasets listing and VMs in errors after configuration restore --> use the Password Secret Seed! TrueNAS General	20	246	March 7, 2025
How to recover from rm -rf /* ? (lost everything) TrueNAS General SCALE , ZFS	197	2324	June 27, 2025

Lost data(?) on 25.10.1

Related topics