Guide: GPU PCIE passthrough to Plex on VM in CORE

,

I have been a long time lurker online, I thought I would contribute my experience if it helps another person out as I have seen many mixed responses of this either being possible but with no details or cannot be done.

After many hours of searching in forums and threads I have found you can passthrough PCIE GPU for Plex Server with hardware transcoding on TrueNAS Core 13 U6. I have not seen a good guide so this is my sketch of one.

For those who are interested, this is how I did this using the TrueNAS Core, VM Bhyve running Ubuntu Server 20.04. Hardware: Nvidia GeForce GTX 1650, Core i3 8100, Asus B360M-E, 32GB RAM.

  1. Make sure VT-D is enabled in your BIOS first, and make sure your CPU supports it.
    I am running Core i3 8100 with no issues. Older generation CPU may not support it so please check
    Later if you have not done this step in your BIOS, you will encounter an IOMMU error in step 3 (I found this out the hard way).

  2. Add 2 new Tunables with in TrueNAS
    Variable: pptdevs, value: 1/0/0, type: LOADER, Enable
    Variable: vmm_load, value: YES, type: LOADER, Enable
    *Your pptdev variable may be different, please see the links under PCIE Passthrough, checkout this guide: bhyve/pci_passthru - FreeBSD Wiki

  3. Make NFS shares in TrueNAS, for your VM to mount and for Plex to access

  4. Create VM, Under Add Device - Select PCI Passthrough Device, Select device: 1/0/0 or your correct pci config

As a side note, when creating the VM, make CDROM device order to be the highest number, as once you have installed Ubuntu Server, the device order for the NIC does not change when you remove the CDROM device, this is the simplest method to ensure the VM still has internet access.

  1. Install Ubuntu Server
    Making sure the following packages are installed:
    software-properties-common
    ubuntu-drivers-common
    ubuntu-drivers
    nfs-common
    plexmediaserver

  2. Create NFS mount points

  3. Install Plex, Add directory to library of previous NFS mount points.
    GPU will pop up in Hardware Transcoding Device now
    Enjoy!

I have added all the forums/threads that guided me in doing this:

VM Set up

PCIE Passthrough
https://wiki.freebsd.org/bhyve/pci_passthru

Nvidia Drivers

https://ubuntu.com/server/docs/nvidia-drivers-installation

NFS

Plex

I hope this helps someone

1 Like

Thankyou very much for taking the time for this writeup.
I have a Xeon 1275v6 but the Supermicro board I have wont let me access quicksync. I’m going to purchase a 3050 single slot card for the purposes of plex transcodes, and since I’m running core I think this process will work perfectly. I will let the community know once I am done.

My plan differs a little here. I’ll be using Windows Server 2022 in a Bhyve environment, already running Plex quite well.
CPU = Xeon E3 1275v6
Ram = ECC DDR 64GB Crucial
Motherboard = Supermicro X11-SSM-F (the 8 sata port version with IPMI)
HDD Raid = Onboard SATA controllers to run 8x 18tb Ironwolf Pro
USB to SATA = OS Drive = Booted from USB to SATA adaptors, 2x 120gb 2.5" drives in a mirror

PCIE slot 1 = Chelsio 320 2x 10gbe SFP card
PCIE slot 2 = Nvidia RTX 3050 thin
PCIE slot 3+4= 2x Silverstone SST-ECM22 Dual M.2 to PCI-E x4 NVME SSD/ (2x 4tb SSD NVME PCIe 3.0 in a mirror) + (2x 512gb SSD SATA) for VMs - if I can find a way to attach them to a USB port

Suggestions and constructive criticism very welcome. Would love a sanity check before I go further

Unfortunately after upgrading to TN CORE 13.3, i915 GPU passthrough to bhyve stopped working. Can’t find bhyve related changes in TN CORE 13.3. Anyone got a hint?

loader.conf:

vmm_load="YES"
pptdevs="0/2/0"

bhyve command/log:

/usr/sbin/bhyve -c cpus=4,sockets=1,cores=4,threads=1 -m 6144 -S -A -w -H -s 0:0,hostbridge -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd -s 30:0,xhci,tablet -s 31,lpc -s 5:0,virtio-net,tap1,mac=00:a0:98:3f:33:dc -s 3:0,virtio-blk,/dev/zvol/Data/VM/lumiere-r6xob -l com1,/dev/nmdm9A -s 4:0,passthru,0/2/0  9_lumiere
bhyve: gvt_d_setup_gsm: Unable to get graphics stolen memory base: No such file or directory
bhyve: gvt_d_init: Unable to setup Graphics Stolen Memory
Device emulation initialization error: No such file or directory

Probably caused byt this https://lists.freebsd.org/archives/dev-commits-src-main/2023-June/016325.html

There is no hw.intel_graphics_stolen_base or hw.intel_graphics_stolen_sizesysctl on 13.3.

Answering to myself: this patch git: e8988d60d24b - main - pci: expose intel_graphics_stolen as sysctl is missing from stable13 branch FreeBSD.

Did TN-13.3-RELEASE build from core-build with the above patch applied. Works again:

$ sudo dmesg | grep i915
[   32.939100] i915 0000:00:04.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns=io+mem
[   32.950491] [drm] Initialized i915 1.6.0 20200917 for 0000:00:04.0 on minor 0
[   33.578814] i915 0000:00:04.0: [drm] fb1: i915drmfb frame buffer device
[   33.671185] i915 0000:00:04.0: firmware: direct-loading firmware i915/kbl_dmc_ver1_04.bin
[   33.674358] i915 0000:00:04.0: [drm] Finished loading DMC firmware i915/kbl_dmc_ver1_04.bin (v1.4)