…and PCIe switches are supposed to be essentially transparent to the OS. 
Eureka!!!
The secret to making everything work:
Go into the BIOS->Chipset->North Bridge and change “VT-d” to Disabled, then save BIOS settings and reboot.
With VT-d disabled all of the issues using drives behind the ASM2806 go away - they all just work on a vanilla TrueNAS Scale 24.04 install.
I was able to create a raidz3 with four drives behind the ASM2806 (NVMe_1, NVMe_2, NVMe_3, NVMe_4) and one more directly connected (NVMe_5) and it worked just fine
With a clean install and VT-d disabled, I do still get a pcie bus error severity:corrected message every minute or two - it’s not annoyingly spamming, but might still add back “pcie_aspm=off” to kernel parameters to get rid of that error.
Disabling VT-d should more or less not impact anything unless you are running VMs and trying to do something funky with virtual device passthrough. “Apps” (Kubernetes/Docker) should not be affected by VT-d
Note that I’m about 30min into testing here so who knows what else works or doesn’t work - but it certainly looks a lot better!
8 Likes
This looks very promising!!
I’ll try this sometime next week! If it works for my unit too, I’ll edit the original post with an info block.
1 Like
Marking post #22 as “Solution” will do it.
1 Like
I can confirm that recipe by @vimb allows TrueNas installer proceed and succeed.
It is yet to stand the test of time. But I’m back to the hopes of near-silent home-friendly NAS appliance on a mature software platform w/o hardware lock-in.
2 Likes
Thank you for your post! I hope scale gets more support. I wanted to create a NVME and also didnt want to use TerraMasterOS.
I have confirmed the tweak posted by @vimb. Disabling VT-d in the BIOS allowed me to install TrueNAS SCALE. I was able to easily upgrade my previous CORE installation. I can also confirm the built-in Aquantia ethernet device works fine with Linux drivers. I did not see any PCIe errors, so I did not add the pcie_aspm=off parameter. Thank you for your experimentation and for posting here on the forums!
Now all can use the TerraMaster F8 SSD (Plus) without worry 
, at least as long as they don’t need hardware passthrough for their VMs.
I am unable to edit my original post, however I did mark the reply with the tweak as the “Solution” to this thread as recommended by @etorix.
Here is the new list of required BIOS changes for SCALE:
- Security → Secure Boot → Secure Boot → Disabled
- Required to boot TrueNAS SCALE
- Chipset → North Bridge → VT-d → Disabled
- Required to boot TrueNAS SCALE
- Boot → TOS Boot First → Disabled
- Required to enable editing the boot order
And optional kernel command-line tweaks for SCALE:
pcie_aspm=off- Reduces PCIe errors reported in console.
5 Likes
Thank you very much everyone involved finding the right settings to get TNS running on the F8 It’s back on the menu for me now!
Although the problem itself is solved, I still wonder how VT-d is connected to ZFS, because it only made problems when using ZFS, right?
If you don’t mind, I have a few questions and I would be very happy if one of you finds the time to give me some answers
- When shutting down the device from within TNS, does Wake On Lan work to wake it up again?
- When you disable ASPM, did it make a change in actual power consumption for you?
- As far as I understand VT-d, it is only used to passthrough PCIe devices. I think I have seen options in TNS when creating a VM, that you can passthrough the GPU (experimental I think), but since the GPU is part of the CPU, it does not count as a PCIe device and this would still be possible with VT-d disabled?
@Wolfizen I wonder if the pcie_aspm=off is not necessary because you use other drivers than @vimb does? Would that make sense? Because everything else in the unit has to be the same, right?
Thanks to both of you, I think I will lose some money very soon. The F8, although not cheap, is a compact package that I would not be able to build myself. And this makes it very interesting for me.
@vimb That disabling VT-d helped makes sense to me in the way that it is a chipset feature and used to passthrough PCIe devices to VMs. And the issues you had with one side of the board and more than one SSD connected to it, somewhat feels like there is a connection. Still, it is strange that this has only an effect when using ZFS. I can’t wrap my head around it, but as you have proven and @Wolfizen confirmed, VT-d seems to be the culprit. Now I just wonder why you get the errors with ASPM on and Wolfizen not.
EDIT - I’ve added the Secure Boot setting - thanks for the reminder @bacardi8!
Thanks so much for this very useful and very informative work! My F8 SSD Plus arrived yesterday, I tried TOS but was getting terribly inconsistent performance when transferring data (ISO files, Parallels VM files, video footage, thousands of smaller documents) from my Mac mini. I’ve got 10GbE on all devices involved in data copies, so the impact was terrible and across all my devices.
I had read this article previously and was worried, but then noticed the updates which have worked for me. Even with the standard 16GB RAM, TrueNAS SCALE is performing superbly on the device. CPU and drive temperatures also seem to be lower than when using TOS.
I’ve moved my 4x 4TB Crucial P3 Plus NVMEs to the CPU side of the board; these drives are in a single RAIDZ1 vdev. This gives me just under 11TB usable, which is 2.5x what I had before on a DIY TrueNAS (running HP Z240 SFF with PCIE adapters for NVMEs). On the other side of the board, I have a single Patriot P300 NVME for the TrueNAS OS.
As I write this, I’m transferring the data from a Thunderbolt NVME on the Mac Mini, and I’ve seeing 785MB/sec up to 1.1GB/sec. This is incredible, as with TOS I was seeing long delays (‘Preparing to copy…’) and then speeds around 60MB/sec up to 300MB/sec.
The only BIOS changes I made on the F8 SSD Plus were:
- Disable Secure Boot
- Disable VT-d
- Disable ‘Boot TOS first’
Have not seen any of those console errors related to the ‘pcie_aspm’
SSH’ing into the TrueNAS SCALE install, and running a ‘dd’ test on one of the datasets within the pool, I’m seeing 4.4GB/sec throughput. This was tested with bs=1M count=1000 (1G), 2000 (2G), 4000, (4G) and 8000 (8G). So the NVME devices themselves are performing as expected.
Again, thanks everyone for sharing their successes here! This F8 SSD Plus is amazingly small, light, and quiet. Absolutely love it, and with TrueNAS SCALE working, it’s perfect. Kudos and massive appreciation for your labours!
1 Like
Hi Guys,
First of all, thanks for a great thread.
I did all the same as mentioned above, but I disabled the secure boot and all still went good.
But now after its up running it has made several restarts by itself and I need manually to shut down and start up again. this happens unexpected and I dont know why. Could that have something to do with the disabled secure boot…??
Should I enabled that again…???
BR
@ClanMudhorn3 glad to hear it went well!
Does this mean you were able to leave Secure Boot as “Enabled”? Can you check the current value of this setting? If you are able to boot both the setup media and the installed OS with this setting as Enabled then we can disregard the TrueNAS setup instructions to disable Secure Boot… very interesting.
Also, can you report which file sharing method you are using to achieve the 1.1GB/s throughput? Samba/iSCSI/etc.
Sorry about that omission : I did disable Secure Boot. I’ve edited my post to reflect this. Thanks to @bacardi8 for the reminder.
It’s late, I should read properly - you asked about file sharing method. I’m using SMB for my Mac clients (Mini with 10GbE, MacBook with QNAP 10GbE Thunderbolt adapter when I need to transfer large amounts).
1 Like
My apologies, I did disable Secure Boot too (have edited my post to reflect this). Thank you for nudging me and helping me share more accurate info.
Sorry to hear you’re having restarts: my only advice would be to check logs. Have you added a different RAM module? Intel officially reports the N305 to only support 16GB, and I’ve left the default 16GB module in as TrueNAS seems to be performing perfectly (and I’ve just copied 600GB of data to my NAS without any performance drops).
Hope you can find and solve the cause of your restarts 
1 Like
All good, no problem.
Yes I actually changed the Ram for 32GB already before first start up.
That could be a point, to change back to original if it continues.
Thanks for the info.
1 Like
All OK! I was genuinely curious. I would not have been surprised to see a confirmation that it was not necessary… it could have been a faulty Secure Boot implementation, or maybe iXSystems could have added a signed shim removing the need to disable Secure Boot, etc.
Thanks also . I have been getting only 40MB/s via SMB with 1GbE equipment on the client end. With your info, I can rule out TrueNAS SMB performance as a factor.
1 Like
If it helps, years ago I started some ‘SMB tuning’ on my Macs and did notice improvements (to varying degrees) with things like QNAP NAS, DIY NAS, so perhaps those settings are helping?
Here they are:
-
I enable 9000 byte Jumbo Frames on my Macs
-
Edit or create the /etc/nsmb.conf file and have the following contents:
[default]
signing_required=no -
I run this command from a terminal session; it is used to remove TCP acknowledgements, and I’ve not had any issues with data integrity, streaming, or anything else:
sysctl -w net.inet.tcp.delayed_ack=0 -
And finally, there’s another command which stops macOS from saving those pesky .DS_Store files all over the NAS. Not sure if it has any performance benefits, but I have used this for years without issue:
defaults write com.apple.desktopservices DSDontWriteNetworkStores -bool TRUE
Hope some of this helps!
1 Like
Thanks to everyone here involved getting this up and running, especially Wolfizen and vimb, I pulled the trigger yesterday and I think I got a pretty good deal. In total I got 140€ off the MSRP and maybe this helps others in the Eurozone.
It went like this:
I looked for prices and have seen that on the TerraMaster Homepage, the F8 SSD Plus is 100€ off. If you purchase in the TerraMaster Store at Amazon, it has the normal retail price. When trying to check out, I have seen that only Creditcards where accepted and I wanted to pay via PayPal. Using the chat button on the homepage, the person told me to use the International page for my order, because there is checkout via PayPal available. So I did this and prices were USD. But USD and EUR is 1:1 in their Shop. So when finally checking out via PayPal, they did theyr currency conversion, which made it 700€ total with shipping Maybe this helps others who are on the fence getting this device.
@bacardi8 Would you mind sharing which specific 32 GB module you purchased. My plan is also to max it out with 32 GB like it is advertised at the TerraMaster homepage. Since I also was shopping in the Mini PC are in the last few months, I have seen that the N-platforms seem to be a bit picky about the modules they accept and work reliable. It would be interesting to know which is not working, so I can avoid it.
Thanks again to everyone involved. I can’t wait until my unit arrives
1 Like
But wait with buying that…I have a lot of restarts here…so now I switched back to original 16gb ram…just to see if thats the issue I have.
My issue was the Ram. For me this 32GB Crucial memory did not run well.
The unit kept restarting and freezing, and after I went back to the original 16GB
I don’t see that again. I have copied TB’s of data now without any issues.
Maybe another brand of memory would function, I don’t know, but fact is this one don’t work on this unit.
2 Likes