Data missing after ZPool import

Hahah i think i created two of those files, however i think the rest of it worked

root@truenas[~]# 
root@truenas[~]# zpool checkpoint Primus
root@truenas[~]# mv -v /mnt/Primus/winder /mnt/Primus/winder_old
renamed '/mnt/Primus/winder' -> '/mnt/Primus/winder_old'
root@truenas[~]# zfs mount Primus/winder
root@truenas[~]# zpool get bcloneused Primus        
NAME    PROPERTY    VALUE        SOURCE
Primus  bcloneused  0            -
root@truenas[~]# dd if=/dev/urandom bs=1M count=4096 status=progress of=/mnt/Primus/winder_old/b
3980394496 bytes (4.0 GB, 3.7 GiB) copied, 8 s, 497 MB/s
4096+0 records in
4096+0 records out
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 8.92977 s, 481 MB/s
root@truenas[~]# cp -va /mnt/Primus/winder_old/bigfile.dat /mnt/Primus/winder/
cp: cannot stat '/mnt/Primus/winder_old/bigfile.dat': No such file or directory
root@truenas[~]# cp -va /mnt/Primus/winder_old/bigfile.dat /mnt/Primus/winder/
cp: cannot stat '/mnt/Primus/winder_old/bigfile.dat': No such file or directory
root@truenas[~]# dd if=/dev/urandom bs=1M count=4096 status=progress of=/mnt/Primus/winder_old/bigfile.dat
4182769664 bytes (4.2 GB, 3.9 GiB) copied, 9 s, 465 MB/s
4096+0 records in
4096+0 records out
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 9.51738 s, 451 MB/s
root@truenas[~]# cp -va /mnt/Primus/winder_old/bigfile.dat /mnt/Primus/winder/                            
'/mnt/Primus/winder_old/bigfile.dat' -> '/mnt/Primus/winder/bigfile.dat'
root@truenas[~]# zpool get bcloneused Primus
NAME    PROPERTY    VALUE        SOURCE
Primus  bcloneused  0            -
root@truenas[~]# 

Can you check this again? Sometimes it might take a minute to update:

zpool get bcloneused Primus

You are on the latest SCALE, right? It’s what you wrote in your second post.

I just thought that SCALE prevents you from logging in as the user root?

Same results about 10 minutes apart

root@truenas[~]# zpool get bcloneused Primus
NAME    PROPERTY    VALUE        SOURCE
Primus  bcloneused  0            -
root@truenas[~]# zpool get bcloneused Primus
NAME    PROPERTY    VALUE        SOURCE
Primus  bcloneused  0            -
root@truenas[~]# 

I believe im on scale community edition (thats the download link i went to)

Linux truenas 6.6.44-production+truenas #1 SMP PREEMPT_DYNAMIC Mon Dec 16 20:59:32 UTC 2024 x86_64

        TrueNAS (c) 2009-2024, iXsystems, Inc.
        All rights reserved.
        TrueNAS code is released under the LGPLv3 and GPLv3 licenses with some
        source files copyrighted by (c) iXsystems, Inc. All other components
        are released under their own respective licenses.

        For more information, documentation, help or support, go here:
        http://truenas.com

Welcome to TrueNAS
Last login: Sat Jan 11 20:38:46 PST 2025 on pts/10

Warning: the supported mechanisms for making configuration changes
are the TrueNAS WebUI, CLI, and API exclusively. ALL OTHERS ARE
NOT SUPPORTED AND WILL RESULT IN UNDEFINED BEHAVIOR AND MAY
RESULT IN SYSTEM FAILURE.

root@truenas[~]# 
root@truenas[~]# 
root@truenas[~]# lsb_release -a.
lsb_release: invalid option -- '.'
root@truenas[~]# lsb_release -a 
No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 12 (bookworm)
Release:        12
Codename:       bookworm
root@truenas[~]# uname -a
Linux truenas 6.6.44-production+truenas #1 SMP PREEMPT_DYNAMIC Mon Dec 16 20:59:32 UTC 2024 x86_64 GNU/Linux
root@truenas[~]#    

Can you delete the b file (the mistake) and also the “copy” you made in /mnt/Primus/winder/bigfile.dat ?

Then try the copy again, but this time invoke the --spase=never flag.

cp -va --sparse=never /mnt/Primus/winder_old/bigfile.dat /mnt/Primus/winder/

Here you go

root@truenas[/]# dd if=/dev/urandom bs=1M count=4096 status=progress of=/mnt/Primus/winder_old/bigfile.dat
4240441344 bytes (4.2 GB, 3.9 GiB) copied, 8 s, 530 MB/s
4096+0 records in
4096+0 records out
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 8.15883 s, 526 MB/s
root@truenas[/]# cp -va --sparse=never /mnt/Primus/winder_old/bigfile.dat /mnt/Primus/winder/             
'/mnt/Primus/winder_old/bigfile.dat' -> '/mnt/Primus/winder/bigfile.dat'
root@truenas[/]# ls -slapth /mnt/Primus/winder_old 
total 4.0G
4.0G -rwxrwx---    1 root    1000 4.0G Jan 11 20:51  bigfile.dat
 12K drwxrwxrwx   13 donkey1 1000   25 Jan 11 20:51  ./
 512 drwxr-xr-x    6 root    root    6 Jan 11 20:24  ../
698K drwxrwxrwx 2204 donkey1  972 2.2K Dec 16 22:29 'TV New'/
 71K drwxrwxrwx  405 donkey1  972  406 Jul  8  2024 'TV Watched'/
 512 drwxrwxrwx    4 donkey1  972    4 Jul  8  2024  Music/
 512 drwxrwxrwx    4 donkey1 1000    4 Dec 11  2023 'Files to sort'/
 12K drwxrwxrwx    4 donkey1  972    8 Dec  9  2023  Film/
105K drwxrwxrwx    2 donkey1  972    2 Apr 29  2022 'TV Sort'/
 512 -rwxrwx---    1 donkey1 1000    0 Sep 13  2021  .temp_write_2e13f4be
 512 -rwxrwx---    1 donkey1 1000    0 Sep 13  2021  .temp_write_b621286b
 512 -rwxrwx---    1 donkey1 1000    0 Sep 13  2021  .temp_write_a3a6ef4e
 12K drwxrwxrwx   18 donkey1  972   18 Oct 18  2020  localhost/
 512 drwxrwxrwx    3 donkey1  972    3 Jan  2  2019  Downloads/
 512 drwxrwxrwx    5 donkey1  972    5 Nov 24  2018  Game_Movies/
 512 drwxrwxrwx    3 donkey1  972    3 May  5  2018  .plugins/
 12K drwxrwxrwx    7 donkey1  972    7 Dec 21  2015  Movie/
6.5K -rwxrwx---    1 donkey1  972  267 Dec 19  2015  .mailrc
6.5K -rwxrwx---    1 donkey1  972  909 Dec 19  2015  .shrc
 512 -rwxrwx---    1 donkey1  972   91 Dec 19  2015  .login_conf
6.5K -rwxrwx---    1 donkey1  972  898 Dec 19  2015  .cshrc
6.5K -rwxrwx---    1 donkey1  972  680 Dec 19  2015  .profile
6.5K -rwxrwx---    1 donkey1  972  212 Dec 19  2015  .rhosts
6.5K -rwxrwx---    1 donkey1  972  186 Dec 19  2015  .login
6.5K -rwxrwx---    1 donkey1  972  301 Dec 19  2015  .mail_aliases
root@truenas[/]# ls -slapth /mnt/Primus/winder    
total 4.0G
 36K drwxrwx--x 10 filac filac   11 Jan 11 20:52  ./
4.0G -rwxrwx---  1 root   1000 4.0G Jan 11 20:51  bigfile.dat
 512 drwxr-xr-x  6 root  root     6 Jan 11 20:24  ../
 512 drwxr-xr-x  2 root  root     2 Jan 11 01:13 'Windows File Access'/
 512 drwxr-xr-x  2 root  root     2 May 13  2018  .warden-template-standard-11.0-x64-newconfig-clone/
 512 drwxr-xr-x  2 root  root     2 May 13  2018  .warden-template-standard-11.0-x64/
 512 drwxr-xr-x  2 root  root     2 May 13  2018  .warden-template-pluginjail-11.0-x64-20180505233651/
 512 drwxr-xr-x  2 root  root     2 May 13  2018  .warden-template-pluginjail-11.0-x64/
 512 drwxr-xr-x  2 root  root     2 May 13  2018  .warden-template-pluginjail/
 512 drwxrwxrwx  3 filac filac    3 May  5  2018  .warden-files-cache/
 512 drwxr-xr-x  2 root  root     2 May  5  2018  .warden-template-standard/
root@truenas[/]# 

returned something different now

root@truenas[/]# 
root@truenas[/]# 
root@truenas[/]# 
root@truenas[/]# 
root@truenas[/]# zpool get bcloneused Primus
NAME    PROPERTY    VALUE        SOURCE
Primus  bcloneused  4.00G        -
root@truenas[/]# 
root@truenas[/]# 

From waiting longer (from the previous test), or because you invoked the flag --sparse=never this time?

Because of the flag

1 Like

Before we continue:

To confirm something:

zfs mount | grep winder
ls -lh /mnt/Primus/winder_old/bigfile.dat
ls -lh /mnt/Primus/winder/bigfile.dat
zpool get bcloneused Primus
zpool get checkpoint Primus

If it all looks good, you can delete the 4-GiB test files. (Don’t forget the accidental “b” file! :wink:).


Lastly, do you know how to use tmux and its shortcut keys?

Returns

root@truenas[~]# zfs mount | grep winder
Primus/winder                   /mnt/Primus/winder
root@truenas[~]# ls -lh /mnt/Primus/winder_old/bigfile.dat
-rwxrwx--- 1 root 1000 4.0G Jan 11 20:51 /mnt/Primus/winder_old/bigfile.dat
root@truenas[~]# ls -lh /mnt/Primus/winder/bigfile.dat
-rwxrwx--- 1 root 1000 4.0G Jan 11 20:51 /mnt/Primus/winder/bigfile.dat
root@truenas[~]# zpool get bcloneused Primus
NAME    PROPERTY    VALUE        SOURCE
Primus  bcloneused  4.00G        -
root@truenas[~]# zpool get checkpoint Primus
NAME    PROPERTY    VALUE    SOURCE
Primus  checkpoint  3.41M    -
root@truenas[~]# rm /mnt/Primus/winder/bigfile.dat 
root@truenas[~]# rm /mnt/Primus/winder_old/bigfile.dat 
root@truenas[~]# ls -lh /mnt/Primus/winder/bigfile.dat
ls: cannot access '/mnt/Primus/winder/bigfile.dat': No such file or directory
root@truenas[~]# ls -lh /mnt/Primus/winder_old/bigfile.dat
ls: cannot access '/mnt/Primus/winder_old/bigfile.dat': No such file or directory
root@truenas[~]# 

I do not, sorry.

Quick rundown and practice.

Start a new tmux session with:

tmux new -t practice

Upon automatically entering the session, “detach” from it, without killing it:
CTRL + B, release, then press D.

(Make sure you don’t accidentally hit CTRL + D, as that would kill the session.)

Now that you are outside the session, list the current tmux sessions:

tmux ls

To attach back into the “last used” session:

tmux a

To attach to a specific session:

tmux a -t practice

Once you are back in the practice session, run a command that will go on for some time:

sleep 15; echo "It's been 15 seconds!"; sleep 15; echo "Now it's been 30!"; sleep 15; echo "The program is done! You can safely exit this tmux session!";

Quickly, before it finishes, safely detach from the session. (See above.)

Now attach to the session again, and watch the command “complete”. See how it didn’t get killed, even though you safely detached? The process kept running in the background.

Now terminate the session with a simple exit or CTRL + D.

Does it make sense? You’ll be running a giant copy command in a tmux session so that you don’t risk the process being interrupted from a timeout or disconnected SSH connection.

1 Like

That command (sleep et al) doesnt seem to do anything in the tmux session but i can get in and out of the session easy enough. pretty familiar with this in winboxen we just call it something different

If you’re ready.

You’re going to “copy” everything inside winder_old to its new proper location on the dataset winder. Hopefully, it will only use block-cloning, which means it won’t duplicate another 11+ TiB, nor will it “delete” anything, since this is not a traditional “move” command that “deletes the old files”. (This should be safer.)

Start a new tmux session:

tmux new -t migrate

Copy everything over:

cp -va --sparse=never /mnt/Primus/winder_old/. /mnt/Primus/winder/

(Yes, that “dot” is in the command for a reason.)

I included -v so that you can see it’s doing something. It’ll flood your terminal with pages of text, since I assume a lot of files are involved. You do’t need to use -v if you don’t want to see any text output. It’ll just be harder to know where the copy operation is currently at.

You can safely detach from the tmux session (CTRL + B, then D), and reattach whenever you want to check if it’s finished:

tmux a

After it’s done, double-check with this, and you’ll hopefully see around 11 TiB of block-cloning used:

zpool get bcloneused Primus

If you think you’ve migrated everything over successfully, you can delete the old folder that is stuck in the root dataset:

rm -rf /mnt/Primus/winder_old

This should now be close to zero (or actually zero):

zpool get bcloneused Primus

Try to access the media files and play them with Plex. (But Plex and permissions/ownership are a separate issue.)

:warning: Don’t discard the pool checkpoint yet, until you’re 100% sure you won’t need to rewind the pool to before you started this process.

2 Likes

Looks like its working. The automated scrub tasks started inthe middle of it all and set the process back like 4 hours, but otherwise it seems to be working. Ill just need to chown things up a bit once the copy is all done.

1 Like

Can you confirm with the above get bcloneused command?

Even though it’s not done, you want to ideally see that the BCLONEUSED value is increasing… otherwise you might inadvertently be making actual copies of the data blocks, which is not good!

haha didnt think of that, looks like it just finished

root@truenas[~]# zpool get bcloneused Primus
NAME    PROPERTY    VALUE        SOURCE
Primus  bcloneused  11.3T        -
root@truenas[~]#
1 Like

Just to double check, i can just rm -rf the winder_old right ?

root@truenas[/mnt/Primus]# 
root@truenas[/mnt/Primus]# ls -slapth                             
total 73K
36K drwxrwxrwx 12 filac   filac 12 Jan 12 14:05 winder/
12K drwxrwxrwx 13 donkey1  1000 24 Jan 11 21:08 winder_old/
512 drwxr-xr-x  6 root    root   6 Jan 11 20:24 ./
512 drwxr-xr-x  4 root    root   4 Jan 11 01:18 ../
12K drwxr-xr-x  9 root    root  12 Jan 11 01:13 iocage/
12K drwxr-xr-x  9 root    root   9 Nov 16  2018 jails/

If everything seems correct, you can.

The Plex, permissions, and ownership thing will be a separate issue.

Your data, however, is safely stored in the dataset Primus/winder, assuming that the migration succeeded without any errors.

Do a sanity check that your checkpoint still exists:

zpool status Primus | grep checkpoint

Then delete the old folder.


EDIT: It’s possible to do a rudimentary check that no files were skipped.

rsync -nvahHxx --ignore-existing /mnt/Primus/winder_old /mnt/Primus/winder/

The rsync “dry-run” should eventually complete with an empty list of files that need to be sent over.

The presence of the trailing slashes (/) in the command is important.

That all looks good, I just need to remember how to install plex now

Thanks again for all your help. You’ve been an absolute Legend

1 Like

Don’t forget to eventually discard the checkpoint.

:warning: Once you discard the checkpoint, there is no longer a chance to rewind back to that pool’s state. (You also don’t want to sit on a checkpoint for too long, either.)

If the rsync dry-run produces an empty list, and after you delete the old folder you can still access your files under /mnt/Primus/winder, then it’s safe to discard the checkpoint.

zpool checkpoint -d Primus
1 Like