Scale - NIC is grabbed by VFIO instead of left for Truenas to use

TrueNAS General
1

Hello,

I’m a linux Noob so seeking some assistance please.

I passed through my GPU to a VM using the Menu for isolation. Something went wrong and now the NIC is also isolated from the host.

The VM GPU passthrough works but the Host cannot use the NIC anymore.

Output from lspci -v
I need the NIC to not be VFIO if that makes sense

//How do I create a code block?//
01:00.0 Ethernet controller: Aquantia Corp. AQC107 NBase-T/IEEE 802.3bz Ethernet Controller [AQtion] (rev 02)
Subsystem: Aquantia Corp. AQC107 NBase-T/IEEE 802.3bz Ethernet Controller [AQtion]
Flags: fast devsel, IRQ 11, IOMMU group 2
Memory at f6c00000 (64-bit, non-prefetchable) [disabled] [size=64K]
Memory at f6c10000 (64-bit, non-prefetchable) [disabled] [size=4K]
Memory at f6800000 (64-bit, non-prefetchable) [disabled] [size=4M]
Capabilities: [40] Express Endpoint, MSI 00
Capabilities: [80] Power Management version 3
Capabilities: [90] MSI-X: Enable- Count=32 Masked-
Capabilities: [a0] MSI: Enable- Count=1/32 Maskable- 64bit+
Capabilities: [c0] Vital Product Data
Capabilities: [100] Advanced Error Reporting
Capabilities: [150] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
Capabilities: [180] Secondary PCI Express
Kernel driver in use: vfio-pci
Kernel modules: atlantic

02:00.0 VGA compatible controller: NVIDIA Corporation GA104 [GeForce RTX 3070] (rev a1) (prog-if 00 [VGA controller])
Subsystem: ASUSTeK Computer Inc. GA104 [GeForce RTX 3070]
Flags: bus master, fast devsel, latency 0, IRQ 59, IOMMU group 2
Memory at f5000000 (32-bit, non-prefetchable) [size=16M]
Memory at e0000000 (64-bit, prefetchable) [size=256M]
Memory at f0000000 (64-bit, prefetchable) [size=32M]
I/O ports at e000 [size=128]
Expansion ROM at f6000000 [disabled] [size=512K]
Capabilities: [60] Power Management version 3
Capabilities: [68] MSI: Enable+ Count=1/1 Maskable- 64bit+
Capabilities: [78] Express Legacy Endpoint, MSI 00
Capabilities: [b4] Vendor Specific Information: Len=14 <?> Capabilities: [100] Virtual Channel Capabilities: [250] Latency Tolerance Reporting Capabilities: [258] L1 PM Substates Capabilities: [128] Power Budgeting <?>
Capabilities: [420] Advanced Error Reporting
Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?> Capabilities: [900] Secondary PCI Express Capabilities: [bb0] Physical Resizable BAR Capabilities: [c1c] Physical Layer 16.0 GT/s <?>
Capabilities: [d00] Lane Margining at the Receiver <?> Capabilities: [e00] Data Link Feature <?>
Kernel driver in use: vfio-pci
Kernel modules: nouveau, nvidia_current_drm, nvidia_current

02:00.1 Audio device: NVIDIA Corporation GA104 High Definition Audio Controller (rev a1)
Subsystem: ASUSTeK Computer Inc. GA104 High Definition Audio Controller
Flags: bus master, fast devsel, latency 0, IRQ 18, IOMMU group 2
Memory at f6080000 (32-bit, non-prefetchable) [size=16K]
Capabilities: [60] Power Management version 3
Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [78] Express Endpoint, MSI 00
Capabilities: [100] Advanced Error Reporting
Capabilities: [160] Data Link Feature <?>
Kernel driver in use: vfio-pci
Kernel modules: snd_hda_intel

Happy to run any commands needed for more info.
The NIC is not showing up under the Isolated device menu, if it did I would have un selected it there.

I am aware this is not the preferred brand NIC.

Cheers for any help,
Mugle

2

It seems both your GPU and your NIC are in IOMMU Group 2.

For it to work they need to be in different ones.
Otherwise they will always be passed through together.

Try using different PCIe slots.

3

Oh yeah you are right. Thanks for that.
I moved the GPU to the 3rd slot but they are still same group.

I think I added ACS overwrite using
midclt call system.advanced.update ‘{“kernel_extra_options”: “pcie_acs_override=downstream”}’
but am not sure if this actually did anything.

Would you happen to know how I can remove the GPU from VFIO/Passthrough? I need the NIC more than the GPU. The gpu is not showing up under isolated devices but still acts like it is passed through.

Cheers

4

Nevermind, I simply removed the GPU from the system and now it’s back to normal. Will need to investigate a newer system with better IOMMU groupings. Thanks a lot for your help!

5

I am not sure if the acs overwrite patch works on Truenas SCALE. VM configuration is sadly kind of limited aswell.