First TrueNas build. Whats going on? +some little doubts

[karl_azytzeen]

Hi there comunity, i was a long time lurker. i start getting some parts years ago when this was Freenas, but life happens and i had to leave this forgotten in a closet.
This is the hardware i assambled.

[QUOTE]

Chasis: SuperChassis 743TQ-865B (right now im running a Tecens Radix 700W PSU 'cause the server is in my desk and supermicro PSU doesen’t shut off fans, never!!), but it wil eventually work on the PWS-865-PQ (Delta 80+ PSU).
SC743TQ-865B | 4U | Chassis | Products | Super Micro Computer, Inc.

MB: Supermicro X11SSH-CTF
X11SSH-CTF | Motherboards | Products | Supermicro

CPU: Intel Xeon E3 1270 v6

RAM: Timetec 71TT24EU2R8-16G DDR4 UDIMM ECC 2400 CL17

SO disk: Intel DC S3610 400Gb

APP disk: Crucial T500 1Tb

Pool Disks: 5x WDC WD80EZAZ-11TDBA0

SO: TrueNAS Scale

Rigth now i set up the system and is working and i have connectivity. Im running it tright now as test unit and learning the system. I was fighting with Truenas (Sorry to say that but im 100% windows user and every little thing i need to do requewires for hours of searching and leraning, a pretty step curve but will climb it).

This is how the basic network looks

I’ll tell you what i did and a few questions just to be sure im doing any fatal thing or something that will end in a catastrophic failure.

Right now i did a single stripped pool on the nvme SSD (run it standalone in flat language), which is supposed to run that way casue it will only hold apps. Is this wrong, is here important data stored? In case of failures here, it’s just replace and reinstall apps? My board comes with a single NVME port so mirroring this will require an aditional card and i dont have many pcie expansion slots avaible (1 x2 and 1 x4).

The board comes with an LSI SAS3008. I read a hundred times to flash it into IT mode (it can be done with TrueNAS??). Right now im not using it (dissabled in BIOS) cause it gets really hot even with nothing attached to it. Also it is supossed to be MiniSAS HD connector but i saw a bunch of different connectors that are supossed to be that. Can someone point me the real name?

Also related with the MB; Fan control, is driving me nuts!!!. BIOS is utterly simple, 4 profiles, but of rpm go below 600 they start to rampup every 30secs, so it forces me to run them at 100% to not throw it to the window. Can fan be managed on TrueNAS? The system comes with 4 5000 rpm SanACE 80 i wil not be using. Provisonally its working with a 120 for cpu and 140 noctuas for drives.

Plus with the board, Sideband and I2C. Im new to server hardware but hard drive backplane is not showing up state leds, im asumming this works throught this interfaces? (i downt know beacuse when i got the chassis it doesent have any HD attached) (was a display output server in a VideoWall). How i cand do this leds plus state alarms work?

Last thing related to hardware for now. WD80EZAZ are supossed to be HGST Deskstar Helium without TLER. I read varied things. I do need to activate TLER on them manullay (on every boot) or is TrueNAS managing this on its side? If so, how can i do it?

Coming back to TrueNAS. Im getting abysmal network perfomance in the oncetion with windows. I setup a SMB dataset and tested moving files back and forth, even to a ramdisk. No matter how they are not going past 100MBs, averaging 60-70MBs. I replaced all cables from UTP to STP, and fiddled with setting a hundred of times (dissabled overflow control, set jumbo packs and buffers to maximum and many other adjustments). In windows side i got a serrated crescent graph that tops at 1Gb/s.

But if i set an iperf3 server on windows and run test they usally are in the ballpark of 9.6-7Gb/s even i see 10+Gb/s ressults sometimes. How i can get this on windows. One of the reasons to make the server (apart of backups and reliability) is to move away all HD from my PC and only have SSD, and with 10Gb i can do this without loosing so much performance; is not the case.
How can i make this work at its full potential?

Also i want to centralize torrent download in the server. I think a mirrored disk for this is waste. How is the best way to set the disk configuration for this. Im thinking in purposing for this an old WD20EARS or even a 1-2Tb 2,5 drive i had here, Will using this slow disk degrade system performance?

I had many other thigs to ask for, but for now i will end here and continue asking google to not overload you.

PS:Sorry, english is not my native language, if anythig is now correctly explained or understandable, ask for it and i will gladly try to explain it better.

Thanks ppl, have a nice day!

[oxyde]

Welcome!
Just a little tip on this:

@karl_azytzeen

Right now i did a single stripped pool on the nvme SSD (run it standalone in flat language), which is supposed to run that way casue it will only hold apps. Is this wrong, is here important data stored? In case of failures here, it’s just replace and reinstall apps? My board comes with a single NVME port so mirroring this will require an aditional card and i dont have many pcie expansion slots avaible (1 x2 and 1 x4).

Is much safer if you setup a dayli replication task on the storage pool. In case of need (Eg drive fail) It works as a “backup”.
When you install app (or better, for me) have much sense to store some data on the fast pool, so redundancy Is high adviced.
Btw a 4x slot Is enough for a pcie–>NVME adapter

[etorix]

@karl_azytzeen

My board comes with a single NVME port so mirroring this will require an aditional card and i dont have many pcie expansion slots avaible (1 x2 and 1 x4).

A x4 PCIe slot would be perfect.

@karl_azytzeen

The board comes with an LSI SAS3008. I read a hundred times to flash it into IT mode (it can be done with TrueNAS??). Right now im not using it (dissabled in BIOS) cause it gets really hot even with nothing attached to it. Also it is supossed to be MiniSAS HD connector but i saw a bunch of different connectors that are supossed to be that. Can someone point me the real name?

To flash if need be, put sas3flash.efi and the apropriate P16.00.12.00 on a FAT-formatted thumbdrive and boot to UEFI shell.
You connector is MiniSAS-HD or SFF-8643.
And, yes, some airflow pointed at the HBA is a must.

@karl_azytzeen

Also related with the MB; Fan control, is driving me nuts!!!. BIOS is utterly simple,

Fan control is done by IPMI. Search the old forum for fan control scripts for Supermicro boards, these were written for X10SDV and the like but may well work with your X11SSH.

[WiteWulf]

Okay, so it looks like you have 10GbE direct link between the TrueNAS server and your Main Windows PC, and I assume you’re wanting to take advantage of that high speed link, rather than have data traverse the 1GbE links between the TrueNAS server, TPLink AP and router?

First thing, is your TPLink Archer running in AP mode, or acting as a router? If it’s routing (presumably with NAT and firewalling) as well as what your ONT/router is doing, that’s going to complicate matters.

The NICs connecting the PC and TrueNAS server must have static IP addresses configured on them from a different network range to what you’re using on the rest of your LAN, as they can’t communicate with a DHCP server (which I assume is running on your router). Once they have static IP addresses assigned you should be able to explicitly access the TrueNAS server using the IP address on it’s 10GbE NIC, forcing traffic between it and the PC down the high speed link. If you’re using name resolution from your router, you’ll want to either add a static entry for a different hostname specifically for the 10GbE NIC, or else add it to the hosts file on the Windows PC (is hosts file a thing in Windows?)

I notice that your Windows PC is connected to both the wired and wireless networks (via wired connection to the router, and wireless to the TPLink AP). I’d suggest keeping on just one or the other. Same for your HA server; giving both a wired and wireless connection to your AP is complicating matters, imho.

[Protopia]

It might be useful if you were to add the IP addresses on each network interface, and tell us what IP addresses you use for your data transfers so that we can take a guess at how traffic actually flows.

[karl_azytzeen]

Hi oxide, thanks for your fast answer.
Of course a backup task is a must for me, even on the fast drive. Just want know the viability to have a fast drive stand alone with apps, and backup it daily or even weekly. The the boot drive.
But im start thinking is not that simple 'cause configuration in some apps like plex is a real pita. Maybe its just that in truenas software doesent works the way i think?
Right now tring to config plex and i broke the SMB that was working >.<
Excluding the media itself. What are the data that is worth being stored in the fast pool? My thought was, the program and its data in the fast pool, media in a dateset of the storage one.
im wright?
OFC pcie x4 is fast enought for a nvme ssd, just as im limited and i dont see it as necessary, was waiting to know a little more of this system, so i can choose wisely.
For example if i want to use more disks i will need to install an extra caddy mounts in the 5.25 bays, but this is not important or necessary right now.

[karl_azytzeen]

Hi WiteWulf, thanks for your time.
The ONT/router has most of its functions dissabled, just barely working to have the iptv and the computer with acces to the internet, this way i shave a few miliseconds when im playing online respect of connecting to the TPLink.
TP-Link router is fully working with all its capabilities except their paid security crap.
The wireless conection to the TPlink is because im not able to reach what is behind the TPlink from the computer. Probably related to opennbing ports or redirect something, but i didnt want to mess with that, and a few clicks solved it with the wifi. Everything was working wireless perfectly, ethernet came later. The same as home assitant, it started working wireless until i was able to install the wired connection. Will be better to switch the wifi connection to the ont/router for redundancy, but i had all wireless disabled here. both routers are together and found that even on different channels signal quality improved with this swithched off, as i cant move them to he other end of the spectrum cause this will mess with the ZigBee network. And to be honest, the wireless that router the internet provider gives is pure crap. Works ok, but in a very limited range. My house is 250m2 and with the TPlink i get enought signal in the most remote room.

@WiteWulf

The NICs connecting the PC and TrueNAS server must have static IP addresses configured on them from a different network range to what you’re using on the rest of your LAN, as they can’t communicate with a DHCP server (which I assume is running on your router). Once they have static IP addresses assigned you should be able to explicitly access the TrueNAS server using the IP address on it’s 10GbE NIC, forcing traffic between it and the PC down the high speed link. If you’re using name resolution from your router, you’ll want to either add a static entry for a different hostname specifically for the 10GbE NIC, or else add it to the hosts file on the Windows PC (is hosts file a thing in Windows?)

I think its is mostly done. Works as you said, plus the redundacy on 1Gb side, wich i tested it works.
Im lost with what you tell about hosts. I didnt touch anything here.

Anyway, the main problem is that Iperf3 gives good results in the link between windows and TrueNAS 9,7Gb/s and call it day!, but when moving files betweem then it dosent even hits 1Gb/s stable. it usually does it a 65-70 with 90Mb/s peak.
So the problem is on the software side. But how can this be solved? thats what im here for ^^.

Good point @Protopia, just didnt think anyone on a forum would mess tto tha indeep level. Here you are.
Every device has an ip reserved on the router side, so i keep every device with automatic dhcp as router manages that.

Starting from the TPLink side:

• Internet provider router(gateway): 192.168.0.1
• TPLink Router (gateway): 192.168.1.1
• Windows wireless: 192.168.1.11
• Windwos wired: (not used) but .10
• Home assitant wired: .30
• Home assitan wirelss: .31
• Truenas X550 1: .32
• Truenas X550 2: not used cause it s the direct link): .33
• Truenas IPMI: .34

Now the side of the device

• Windows PC wired to IP: DHCP
• Windows PC wireless to TPLINK: 192.168.1.31
• Windows PC wired to TrueNAS: 192.168.2.3
• Truenas X550 to TPLink: DHCP
• Truenas X550 to Windows: 192.168.2.4
• Truenas IPMI: 192.168.1.34
• Home assitant wired to TPLink: DHCP
• Home assitan wirelss: to TPLink: DHCP

If you think you need more data just ask for it! I hope this can helkp you to understand were the problem comes from.

[oxyde]

@karl_azytzeen

Just want know the viability to have a fast drive stand alone with apps, and backup it daily or even weekly. The the boot drive.

Perfectly viable (i have done the same for months, on the old system), but IMHO with the mirror i sleep a bit better
For the boot drive, my point of view Is kinda different: use mirror there Is a disk waste (if you backup regulary your config, replace disk → fresh install → config upload Is literally 15 mins of work); different Is if you can’t phisycally access tò the Nas. Get a look on resource for joe multi report, Is a must have for monitor disks and pool (and automate config backup)

@karl_azytzeen

Excluding the media itself. What are the data that is worth being stored in the fast pool? My thought was, the program and its data in the fast pool, media in a dateset of the storage one.
im wright?

This depend from app to app, and prob on how you are installing them.
Normally, in the install instruction, are always mentioned the various point to mount as pref.
If you lose those data will be a pain (Eg Imagine to lose all Plex metadata), or really a nightmare (Eg lose a Vaultwarden db )

[karl_azytzeen]

Thank you for the info of the update.
Regarding the fans, IPMI only gives me: Optimal, standard, HighIO and Full.
Im running HIghIO so the fans runs continuosly at 1400rpm. any other profile goes below 600rpm and start ramping up and down.

I think i saw the post at some point, but i dont know how to use that info. I will need an explanation like if i was dumb, as i are (regarding this issue). Plus if i recall correctly, they wire most fans to one header. dont know the rate of this board, but they use to be 1Amp. in case i use the Sanyo, each ones requeires .7, in this case will a splitter board with external power work?

[WiteWulf]

It looks like you’ve got this started, and redesigning you LAN architecture is outside the scope of this forum. Just make sure that the direct connected NICs on your TrueNAS and Windows PC with 192.168.2.0/24 addresses do NOT have a default gateway assigned to them. Those NICs should ONLY be used for communicating between the PC and the TrueNAS. And make sure that your Windows PC is connecting to the TrueNAS as 192.168.2.4, NOT 192.168.1.32

(But you also said above that the TrueNAS has a connection to the router using ‘.32’, but that’s not show on your diagram. Confused! )

That said, if you can, disable the router and firewall elements of your TPLink Archer if you can, and run it in pure AP mode. Having two different NAT’ing routers on your network is going to cause you all sorts of headaches.

Either that, or have all your devices connected to the TPLink, and nothing connected to the ONT/Router except your TPLink router. All your devices should have 192.168.1.0/24 addresses.

[karl_azytzeen]

Thank you WiteWulf

@WiteWulf

Just make sure that the direct connected NICs on your TrueNAS and Windows PC with 192.168.2.0/24 addresses do NOT have a default gateway assigned to them

Can you explain me why? Probably its wrong but in old times, in the era when 10/100 connections where top notch and crossed ethernet cables, i remember to use the ip of each other as gateway in direct connections.
But forget a bout this, anyway it is working as you can see

@WiteWulf

But you also said above that the TrueNAS has a connection to the router using ‘.32’, but that’s not show on your diagram. Confused!

To TPLink, not to ONT/router

@WiteWulf

disable the router and firewall elements of your TPLink Archer if you can, and run it in pure AP mode. Having two different NAT’ing routers on your network is going to cause you all sorts of headaches.

The ONT router is pure crap, and also is limited en many ways, i nned to be able to manage the network and i cant do it in the ont one. so everything is done inthe TPLink. If i ddnt have phone iptv and the direct link, that would be only a bridge, just a transciever. Right now, i cant.

@WiteWulf

and redesigning you LAN architecture is outside the scope of this forum

I’m allways open to suggestions. Maybe there is something i didnt ithink, but mostly its like it is for something or to circumvent a wall i cant drill or a cable i cant hide.

[WiteWulf]

Okay, really simply: if a NIC has an IP address and a network mask but no gateway it can only access hosts with IP addresses within that network.

So a NIC configured with 192.168.1.1/24 would only be able to talk to hosts with an address between 192.168.1.0 and 192.168.1.255

When you add a default gateway to a NIC it becomes a candidate for sending traffic to any other network. You want all the traffic for hosts that are NOT your TrueNAS to go out the other interface, so that’s where you add a default gateway.

You want the direct link between your Windows PC and your TrueNAS to only carry traffic for that purpose, so you shouldn’t add a gateway to the NIC configuration. In fact, it wouldn’t work anyway if you did, because there are no routers on that network, just two hosts (the Windows PC and the TrueNAS), neither of which (I hope) are configured to route packets between their interfaces.

[karl_azytzeen]

Its not perfect but i had improved transfer speeds changing some parameters of samba in windows side, cause the problem is here. Newtwork controllers, drivers, and internet settings are all ok. Truenas is ok too.
With this setting i had reached 5-6 Gb/s sutained transfers of large files, peaking at 8 in very short bursts.

Run this in Powershell as admin:

Set-SMBClientConfiguration -EnableMultichannel $true
Set-SMBClientConfiguration -ConnectionCountPerRssNetworkInterface 16
Set-SMBClientConfiguration -EnableBandwidthThrottling $false
Set-SMBClientConfiguration -EnableLoadBalanceScaleOut $false
Set-SMBClientConfiguration -MaxCmds 100
Set-SMBClientConfiguration -MaximumConnectionCountPerServer 64
Set-SMBClientConfiguration -SkipCertificateCheck $false