ZFS on ZFS (Truenas on Proxmox)

TrueNAS General
, ,
1

Hi everyone,
I wanted to know what are the contraindications of virtualizing Truenas on Proxmox WITHOUT doing disk passthrough but creating a simple virtual datastore as for any other virtual machine.
In other words having ZFS filesystem on ZFS filesystem.
I found many posts around the web that were very chaotic and often contradictory.

This makes it easier to replicate Proxmox from node A to node B even on different hardware like in my case.
I’ll try to explain myself better:

Node A: VM turned ON and replicate Truenas SO and DATA on node B.
If there are problems with the VM on node A, I turn it off and simply turn on the one on node B.

While if I had to have the disks in passthrough mode with different disks from node A to node B I would have to keep both VMs powered on on both nodes with different IPs and replicate the data with some service like rsync or similar

I don’t know any other ways, but I’m definitely not an expert :grin:.

2

Proxmox has mounted the ZFS file systems of TrueNAS and users have lost their data.

Passthrough of the entire disk controller and blacklisting from Proxmox is highly suggested.

1 Like
3

Thanks, I’d already read the article.
Even losing data? No, I hadn’t read that online. I know they don’t recommend it, but I wanted to understand the technical reasoning behind it.

So how would you manage the replicate from node A to node B with different disks and therefore different pools in passthrough mode?

4

You replicate the data stored on TrueNAS from ZFS to ZFS in TrueNAS. The pools need not have the same names, topology or anything.

5

Ok but this is a problem if as in my case I use Truenas as an active directory fileserver under Windows domain with many PCs that use shared folders that point to a specific IP and specific paths…

In case I need to use Truenas on node B for emergency reasons, I should change all the paths on the PCs

6

The best recommendation I have is to buy an HA TrueNAS Enterprise from iX, then. Redundant power, redundant control nodes, redundant data paths, seamless failover.

In case you want to “hack” something with Proxmox VM HA I would recommend against that. No idea how to do the storage part in that scenario. The OS boot “disk” is easy, of course.

But do you really think having terabytes of data in a virtual disk and then leaving it to the hypervisor to live-replicate that … can Proxmox do that, even?

If you place the data on some storage system common to both hypervisor hosts, you already have redundant (hopefully) storage, so why TrueNAS?

1 Like
7

You need manual intervention, anyway. The path of an SMB share visible to a client is not tied to the pool or even dataset name. You could name the shares identically.

And switch the IP address to the standby unit when the primary fails, switch off read-only for the datasets, start SMB service, back in business. 15 to 30 minutes downtime - poor man’s HA. We do the same.

1 Like
8

I’m using Proxmox to manage the daily backups, which involve 3TB of company data.
Proxmox also replicates the machine from node A to node B twice a day.
Trunas manages the internal data snapshots.

The system seems to be working well… I just wanted more technical details on the ZFS-on-ZFS “problem.”

Yes, of course, the proxmox base pool is in Raidz2.
Excellent observation, I use Truenas because it’s fast and “easy” to configure for Windows Active Directory.
I tried Turnkey, but it’s a bit difficult for me. Do you have any other suggestions or alternatives?

9

Yes, you’re right, but with “my” system (which is still a poor man’s HA), all you have to do is power up the virtual machine on node two without changing anything. Logically, only half a day’s replicas are lost.

10

But you want to run 2 Proxmox nodes and replicate between them, correct? Are you doing that already? Replicate TBs of virtual disks sounds quite challenging to me.

EDIT: read your second post now, got it.

And if you don’t want to replicate you need a central iSCSI or NFS storage system external to both Proxmox nodes like common in most larger VMware installations. That gives you seamless failover.

To address the initial question: yes there is a high risk of data loss running TrueNAS on virtual disks. War stories on this and the old forum galore.

11

Replications between the two nodes are done as snapshots, so only the different data moves. They take very little time.
Backups, on the other hand, happen at night on rotating disks on an external NAS, and those take a few hours to finish.

This worries me a lot. Especially if the corruption of the entire Truenas datastore happens suddenly and without warning.

12

As soon as proxmox decides to locally mount the zfs disks while they are mounted by the TrueNAS instance they are toast.

EDIT: here is just one example:

3 Likes