SAS Multipathing on Core and Scale using ESX

TrueNAS General
1

I use a SuperMicro 24-disk chassis with a dual SAS-HBA-Controllers + dual-port SAS Expander Setup and used to run it with TrueNAS with Multipathing, which worked fine.

I too was affected by the announcement that Core and Scale no longer support SAS Multipathing and as I need Scale with my OpenShift System, I needed a solution. And I found one.

This is what I did:

  • I installed the free version of VMware ESX on to the RAID-1 boot-drive (two small SSD’s) that used to host TrueNAS natively before.
  • In ESX, I created big fat Virtual machine with a 30GB virtual disk (which is protected by the hardware RAID-1) and installed TrueNAS Scale on it.
  • The massive TrueNAS VM is the only VM on ESX and has been given all the CPU’s except 2 (ESX needs some CPU to) and 90% of RAM (reserved)
  • I passed through all the ZFS Drives into the, now virtualized NAS OS so it controls them directly and natively (reading the temperature for example, works fine)
  • I used ESX PCI-passthrough to give the Network-cards directly to Scale, again for maximum performance and bypass ESX
  • I then re-created my TrueNAD config and imported the ZFS Pool so all my data was available again.

The kicker is, ESX supports dual-controller/dual-backplane/sas-wide-port natively so I let ESX handle that part and Scale is simply presented a single SSD drive, so it does not freak out over seeing “the same disk twice”.

Performance is still kick-ass, still pumping out well over 2 GB/s reads and writes, just as before.

During daily life, working in the TrueNAS WebGUI or CLI, I totally forget ESX is underneath.

So basically i’m cheating a bit, but letting ESX deal with the SAS multipathing effectively solved my problem. I can keep using this machine as I always have and thus protect my investment.

Hope this helps folks in the same position.

2

Why not go for the obvious one? Just remove the double path cabling.

If you’re not passing the controllers, old wisdom says that your setup will work until it doesn’t…

3

Indeed. See also:

4

One does not need to pass the controllers. In fact, you don’t WANT to pass through the controllers.
Only pass through the resulting single SSD (that ESX hides having two paths) as a RDM (Raw Device Mapping).

12 disks are served to the VM as an RDM on a VMware Paravirtual SCSI Controller with the disks being presented to the VM in “physical mode”. Same goes for the other 12 SSD’s.

I passed-through all 24 SSD’s this way, then created two 12-disk (12-wide) RAID-Z2 vDev’s together into a single pool.

Little nit-pick: this is not the same style of “passing through” a raw NVMe Device, as that is doing PCI Passthrough (as I do with the NICs).
One “passes through” an ESX LUN to a VM, just as one does a (Multipath’d) FC LUN or Hardware-iSCSI LUN (also Multipath’d) in ESX to a VM so that the VM accesses it directly.
One needs to make sure that the VMware storage-layer is robust, before attempting to present such Storage Devices to a VM. Local multipathing SAS is very stable in VMware ESX.

.
“old wisdom says that your setup will work until it doesn’t”

My old ZFS / FreeNAS/TrueNAS / VMware wisdom (have been working with ZFS, *NAS and VMware since Napoleon’s days) says it works just fine.
The very fact that this setup has been:

  • running stable for well over a year now, even under heavy sustained load.
  • surviving two power-outages (I need to buy a UPS one day…)
  • came up just fine after those crashes (reacting to the situation the way TrueNAS does).
  • that it survived one SSD Disk failure, which was replaced in a normal fashion.

says… it works perfectly fine and never failed or showed undesirable behaviour.

5

Not “indeed”. That person’s situation is a different scenario. ProxMox does things differently than VMware in this context.

6

Keep telling yourself that.

7

Well, I could be wrong. I’m only human after all.

As you seem to have in-depth knowledge on how both parties implemented this functionality into their hypervisors, please explain to me, and others who are interested, in great detail, how they do not differ.
I am truly curious. Please.

8

Oh man, RDM, that’s an acronym I haven’t heard in a while. That gave me nightmares from the FreeNAS 8.x forums days.

Congratulations, you’re one of the lucky ones with a good anecdotal experience. Many others on these forums who have lost TB’s of their data cannot say the same. Like you, they have also been running “fine”; some up to 2 years until one day, it didn’t; unfortunately, the vast majority of them did not have any backups. If I earn $10 for everytime someone comes on the forums to cry about their data loss due to their non-recommended ways of virtualizing since the FreeNAS days, I’d be possibly up a few hundred dollars.

9

Well I cannot speak for the unfortunate ones. Maybe it’s because I use actual enterprise grade hardware, respect the VMware HCL, making sure controller firmware matches the recommended driver-versions, and having a very long, professional history with VMware (since GSX and ESX v2).

I understand what I’m doing. I’m a storage and virtualization professional, going back to the Solaris days. I used(or “abused” as you would say) NexentaStor this way too and never had issues.

As long as nobody can explain to me, in great technical, gory detail, why I should not do what I’m doing, what the possible failure-modes/scenarios are, I regard statements like yours, “dan” and “etorix”, as “not constructive” and not informative.

The TrueNAS VM, in my build, is served reliable single Disks to build vDEVs and Pools out of. It is not unlike FC or Hardware-iSCSI LUN’s which are “multipathed under the hood” and therefore more tolerant against path-failures. TrueNAS knows nothing of this, it simply sees the Disks and recognises them as actual disks, reading the actual product names, serialnumbers, temperatures etc. etc.
As far as the TrueNAS VM is concerned, all disks are single pathed. I do not pass-through the storage controllers on purpose (unlike solutions like the old Dell PowerStor X, Cisco HyperFlex and SAS-based Nutanix HCI, because those Storage-Controller VM’s handle multiple SAS controllers and multipathing of wide-Port devices natively, they do not need the hypervisor to do that for them).

I’m sorry but I see the proof every day: I have a VMware cluster consuming VMFS6 Datastores from this TrueNAS System (over iSCSI) and well as 2 OpenShift Clusters using the TrueNASCSI and the performance is very good and consistent. It never skips a beat, stable as a rock.
And I cannot think of a reason why that should change. But i’m hoping that you, or somebody else, can explain it to me. Maybe I’m missing something, despite giving it a great deal of thought.

Oh, and I do make/have backups. Every day.

10

The advice is a published article by TrueNAS Enterprise also. We are just going off their advice. Just hit the summary at the end.

11

Thanks SmallBarky. That is a good read.

My build conforms to all recommendations (CPU, RAM, NUMA pinning etc. etc.) except maybe the storage-part. The article discusses the pittfalls using virtualized disks. But I don’t use that.

I don’t do “full PCI passthrough” either, as the summary in the article recommends. What I do is Passthrough, but of highly-available physical storage devices, because that is what hypervisor-controlled multipathing + RDM (Raw Device Mapping) does.

The article correctly speaks of “ZFS expects direct access to your disks in order to work properly, in order to issue direct SCSI or SATA commands and receive an expected and predictable response” and that is what pRDM’s provide.
Remember that VMware pRDM’s are fully compliant with all relevant parts of the SCSI-3 standard and are officially supported as shared disks for, for example, Microsoft Fail-Over clusters, which heavily rely on low-level commands and things like SCSI-3 Persistent Reservations to enable such clusters. After all, ESX transparently passes or emulates these commands at the virtual SCSI (VSCSI) layer and ZFS happily uses it (I never see kernel messages saying that the TrueNAS Scale kernel or storage-subsystem(s) have issues with the ZFS disks).

I also use a separate SCSI controller for the bootdisk (which is a normal VMDK) and two other, Paravirtual SCSI Controllers (comparable to VirtIO Controllers) for 12 disks each (and the Z2 vDEV they form).

I have the impression that “my way of doing things”, has not been tested to the same degree (or at all) and even though the article speaks of “various alternative configurations for RAID controllers (with or without “HBA Mode” or “JBOD-Like” behavior), paravirtualized disks, and local drive mapping”, none of that really matches my approach, depending on how one interprets “paravirtualized disks, and local drive mapping”

For many years, my NexentaStor VM, which worked in the exact same way as with TrueNAS Scale since 14 months or so, has been running like this without any issues. So I literally have many stable happy years of doing it this way, under my belt.

I think I’m in a kind of grey area where objective, measurable reality clearly says my approach works, is robust, is stable.
Just because my specific way of doing things is not something that has been extensively tested by the folks at TrueNAS, does not mean i’m wrong.