Taking in consideration of what you said, I managed to pass the motherboard’s sata controller, I had to figure out the name of the device in the proxmox CLI. Thank you.
2 Likes
Yes, unfortunately, even running iSCSI you are adding a lot of latency by accessing the data over the network. Latency impacts different games and best suited for multi-user or cross-system environments. iSCSI can provide higher performance for small file transfers and is ideal when a single computer requires block-level access, such as for virtualization or databases
Guess, I’m late to the party. Using Proxmox you can pass the ssds/hdds through if you can not pass the disk controllers. This is typical for systems with single or no add-on controllers. Passing through the disk will keep the integrity of ZFS on the physical disk; you can pull the drives and have them recognized by another instance of TrueNAS.
Another good mention is 1Gbps NICs will only max theoretical 125 MB/s (performance of slower single hdd).
OSes will also be a contender as Windows blows for iSCSI connectors. It works but has inconsistent performance throughput compared to LInux. The purpose of iSCSI and NVMe-oF is to minimize the layers between compute and data set, though in Windows 10/11, I’ve found SMB to be more reliable than iSCSI for Steam games. Games require high IOPS and low latency access to storage.
Danger:
2 Likes
I agree it’s a danger & it’s a risk the user will have to accept if they choose to go that route. Some will do it regardless for testing or due to limited resources. Passing through the controller is the best way to do it, but older home based systems will only have one controller that manages the boot disk. It is highly recommended to purchase a separate controller to pass through the controller to manage the disks.
IMHO gaming over iscsi is a meme made up by youtubers.
Almost all cheap SSDs will outperform iSCSI for a fraction of the price, at better performance and less issues.
Depends on what you are looking for. Having your games in a single point over 10GbE or faster works well. I have all my Steam games from Windows 11 installed over a SMB mapped letter drive on one system, and Steam, EA, GOG over NVMe-oF/TCP in Linux on another. A good SATA SSD will hit about 500 MB/s max transfer. 10GbE will hit about 1 GB/s throughput, but it’s more about IOPS since games rely on a lot of small file transactions. Outside of installs, you really don’t need the insane speeds. All my games load over 10GbE very well, and it’s not noticeable. I do recommend using multiple vdevs or at the very least an svdev for the high IOPS. As I stated iSCSI on Windows blows as it can freeze all other processes while the write queues is pending. You’ll get massive spikes and freezes where iSCSI or NVMe-oF in Linux has been a very pleasant experience. See my testing - Gaming using TrueNAS SMB/iSCSI/NVMe-oF
1 Like
Sure, but IMHO there are three extremes when it comes to games.
A: They would even run from an HDD.
B: They need some sequential read performance and a little bit of IOPS, so perfect for a 500MB/s SATA SSD or cheap NVMe SSD.
C: They really need performance. Like Windows DirectStorage performance. Like +5GB/s.
As a result:
C is very hard if not impossible to achieve with iSCSI, while A and B can be achieved way cheaper with local storage.
Your link / thread / build is IMHO the perfect example for that. Despite you having a quiet beefy and relatively expensive setup with 40GBit NICs, you still have some issues and only reach the seq read performance of a very mediocre and cheap NVMe SSD.
Something like a Patriot P410 will be faster.
Something like a Crucial P510 will be way, way faster.
I experimented with this, over multichannel SMB to a RAIDz2 of SATA SSDs on dual 10gbs links, 32GB ARC, everything ran smoothly once permissions were correct (I mostly tested on Proton). Compared with my local btrfs SATA SSD mirror I didn’t notice a difference.
1 Like
You would need to buy 1 SSD for each system. For a NAS you can do a bunch of vdevs with special vdev and make a mass storage for each system with their own space. I can’t think of a game that requires +5GB/s load speeds. None of my games that were 40+ GB in sized or any of the games I owned required extensive load speeds beyond what 10 or 40 Gbe supplied. Racing games loading textures every < 1 second wasn’t noticeable in 4k with all setting turned on.
10 Gbe will download 1 GB every second just about. For a 40 GB game you can download the entire content in 40 seconds. If Blizzard made a distribution for Linux, I could test it out for NVMe-oF, but because of the nature of Proton, it emulates from the local OS drive. That was the only platform I tested that wouldn’t run on the NAS because of the developers specifications (and it was developed for Windows not Linux). Access time for NVMe-oF in Linux was 0.15-0.17 ms. iSCSI in Windows is poorly done, so yeah, I wouldn’t recommend it. SMB out performed it by far.
Benefit of local storage, it works, but you have to buy one for each. They are fixed size, so you’ll have to buy larger than needed, buy additional, or replace with larger when filled.
Benefit of NAS storage, it works for most games from what I can see. It can scale. It can offload your local storage, and can have redundancy fail-over. You can detach/attach storage.
I mean to each his own. I used to have local drives for each system, but now I just connect it to the NAS.
Sure, but you need a beefy TrueNAS with in your case 4x 1TB SSDs plus the networking infrastructure.
Requires? Not many, Ratched & Clank maybe.
Profits from faster loading times on the other hand? Some games.
PS5 requires AFIK 5.5GB/s for the internal SSD.
Sure. Most games run fine from a cheap SATA or even a HDD, because they don’t need much IOPS or bandwidth and have other loading times bottlenecks than storage.
Helldivers is a prime example for that. It used to be 140GB, because they optimized the data for HDD seeks. So data was there multiple times. They then released a none HDD optimized version that does not have the data multiple times. This is only 40GB.
To their surprise, it turns out that even most HDD seem to be able to handle that version just fine. HDD takes 43.6 seconds, while a NVME SSD takes 37.3 seconds to load.
This is IMHO another reason why IMHO iSCSI gaming is a meme.
Maybe that is just me, because I don’t shuffle that many games around all the time.
Never understood the people upgrading their PS5 NVME. If you really switch games that often, why not just slap a 2.5HDD onto the PS5 and move them on and off from there?
If you can download a 40GB game in a few minutes, by downloading it
(from either TrueNAS 125MB/s or 1GB/s, or WAN 125MB/s or 1GB/s, or some dusty old HDD you have laying around with 180MB/s)
to your local NVME SSD, that alone would rule out the hustle for iSCSI IMHO. Let alone the network equipment needed for it.
I have 10GBit/s WAN, so storing games would be a wast of storage for me personally. I even downgraded my gaming PC LAN from 5Gbit/s to 1Gbit/s because the extra 4W was not worth it in my opinion.
If you work with Snapshots and have many PCs that run the same game, I might see a small use case for saving storage. What network hardware do you use?
I use 10GbE mostly, but I do have 10Gb SFP+ fiber in some areas. My NAS has both 10GbE and 40Gb connections. The 40Gb is a Connectx-3 Mellanox card, which you can find cheaper used than 10GbE NICs. Would I recommend this setup for one pc? No, local storage wins. I think this is more for those who already have an existing infrastructure and applying additional uses to what already exist.
The pool used is just a test pool and will get reworked, but 4x sata ssds striped is probably way more than needed for this case.
1 Like