Let’s face it: Some NAS systems are under-powered, yet serve a vital “storage-only” role, which does not require much in the way of CPU and RAM.
Let’s also face it: Encrypting and decryping a stream over your own local and secure network is redundant, and needlessly cooks your dual-core Celeron.
Let’s also also face it: OpenSSH has removed the “none” cipher from its default build options.
Let’s also also also also face it: Sometimes doing something in the command-line wins you “nerd points” among your peers.
With a few simple configurations and built-in commands, whether you use Core or SCALE, you can leverage SSH + mbuffer + Netcat to replicate large datasets over your local network, even if one or both systems are “under-powered”. This can be useful for preparing a “dedicated backup server” with the initial (and largest) replication locally, before moving it offsite or placing it into monthly cold storage.
The problem
Pure SSH
flowchart LR
pc1[["Main Server\n(weak)"]]
pc2[["Backup Server\n(weak)"]]
proc1{"encrypt"}
proc2{"decrypt"}
pc1 --> proc1 -->|"over the\nnetwork"| proc2 --> pc2
pc1 -...-|"authentication"| pc2
The solution
SSH authentication + mbuffer and Netcat
flowchart LR
pc1[["Main Server\n(weak)"]]
pc2[["Backup Server\n(weak)"]]
pc1 -->|"over the\nnetwork"| pc2
pc1 -...-|"authentication"| pc2
This concludes the guide! You should be able to extrapolate everything from the above charts.
You can keep reading if you like.
For the following examples, the Main server is 192.168.1.100
and the Backup server is 192.168.1.200
.
The Main server has a pool named “mainpool” and a pseudo-root dataset named “myrootdata”.
The Backup server has a pool named “backpool”.
Preparing the Backup server
Make sure the following prerequisites are met for the Backup server:
- The SSH service is enabled
- You allow “password login” for the
root
user over SSH. (This can be disabled later.)
Preparing the Main server
Make sure the following prerequisites are met for the Main server:
- The SSH service is enabled
- You are able to login via
ssh
with a user of your choice - Your user has an
.ssh
directory with700
permissions.
With your user of choice (preferably root
or an “admin” account, since I will omit sudo
from any and all commands):
- Login to the Main server via SSH
- Generate a keypair
ssh-keygen -f $HOME/.ssh/id_weaksauce
- Send the public key to the Backup server
ssh-copy-id -i $HOME/.ssh/id_weaksauce root@192.168.1.200
- Test your ability for passwordless SSH login
ssh -i $HOME/.ssh/id_weaksauce root@192.168.1.200
The replication
From within an SSH session on your Main server, first start a tmux
session, so that you do not need to keep the terminal window open at all times.
tmux new -t weakness
Now, with the power of passwordless authentication, tell the Backup server to listen on the local network (Netcat) for an incoming connection, using performant values of mbuffer
(which you can adjust based on your own research.)
ssh -n -i $HOME/.ssh/id_weaksauce root@192.168.1.200 "nc -l 2525 | mbuffer -s 1M -m 2G | zfs recv -F -s backpool/myrootdata" &
It’s easy to miss the single trailing &
at the end of the command. This is important. Do not forget it.
The Netcat listener on the Backup server, which you started remotely from the Main server, will automatically terminate upon completion of the replication stream.
The port to listen on, 2525
, sounds cool. “In the year 2525!” There’s no technical reason for this.
Change the values, accordingly. I find -s 1M
and -m 2G
work well for my usage.
A “resume token” will be generated on the Backup server, which allows you to resume an interrupted replication. How to leverage this is beyond the scope of this guide.
Still in your tmux session, create a recursive snapshot to use for this one-time migration.
zfs snap -r mainpool/myrootdata@migrate
Still in your tmux session, send the entire replication stream, with all properties, datasets, volumes, snapshots, and clones. This will replicate everything.
zfs send -R -w mainpool/myrootdata@migrate | mbuffer -s 1M -m 2G | nc -w 30 192.168.1.200 2525
You should soon see a “progress”, as the stream is sent over the network without encryption/decryption, using mbuffer
to (hopefully) maintain a stable performance.
You can gauge the progress based on the “used” size of the entire pseudo-root dataset, and how much has transferred in the progress monitor.
To safely exit the tmux session, issue CTRL + B, then press D.
To list active tmux sessions:
tmux ls
To re-attach to the tmux session:
tmux a -t weakness
The aftermath
Once the replication is complete, the Backup server’s Netcat listener will have automatically terminated itself.
Be sure to check the Backup server’s pool, datasets, and snapshots. Try to access the data.
This guide is 99.9999% likely filled with errors and typos.
Rather than ridicule me and inform me how incompetent I am, I would prefer if you ridicule me and inform me how incompetent I am, and point out the mistakes so that I can correct them.