System Architecture

TrueNAS General
, , ,
1

Hi, all. I’ve been running a TrueNAS Core system here for over a year, and so far things have been running fairly well. My hardware is running beautifully (8x 12TB 12gb/s SAS drives on an LSI 9500-8i HBA, Xeon e5 2687W v4 on an ASUS X99 board with 128GB ECC DDR4 (2) 10GBe Intel x540-T1 nics in LAGG. Serves us pretty well as a small architecture firm. We run all of our file services over SMB, except Nextcloud uses NFS (it was slightly more performant for photo serving in my admittedly anecdotal testing). However, I’ve run into a few issues with my original setup and I’ve been considering some major architectural changes to resolve.

First, for some reason, permissions do not work consistently. As far as I can tell, it’s not an SMB issue, it’s an ACL issue–the system locks out files users have been in on a regular basis. I then have to go in the the dataset and strip ACLs and reapply to unlock the files–it’s not as simple as restarting SMB. It’s bizarre and not at all how it should work. Reading the forums here generally leads me to believe that I set something up incorrectly on the get-go, so I would need to start over on making this right. Maybe I missed something in the SMB setup.

Second, I’m considering moving to Scale because I use the crap out of two main VMs: Nextcloud and Windows Server. Linux has weird issues with the peripherals via bHyve, so having better control over that might make more sense. Plus, I would love to have better control of my backups for the zvols, which has been a pain in the neck on Core. I need to back up the zvols because we store files directly in the VM (I don’t really have a choice on this because Revit Server uses IIS and dumps the critical files on the VM’s main drive).

Third, there are a number of issues I’ve had with commissioning and decommissioning systems in our office, so I’m thinking it’s time to properly set up an AD server to better control our multitudinous software and manage our systems using the unfortunate disease that is Microsoft. As an architecture firm, however, all of our production software runs on MSFT systems only, including their server architecture. Hence my VMs.

Fourth, we’re starting to see a trend towards laser scanning our projects, which is taking up a crap ton of space. More importantly, the file storage is enormous, with an average home project containing around 20GB of database, small, and large (>600MB) files and commercial projects easily 3-5x that, and editing these files requires an insane amount of speed. We can easily saturate an entire 10GbE connection just editing those files. As a result, we’re considering setting up another small SSD-only server that would have much faster performance for those files only and then dump anything we’re not actively working on to the archive on the TrueNAS system.

So here’s the query: In order to maximize performance/reliability/security and minimize the stupidity of this system, I’m considering a major change. If I were to start over with Scale, which of the following options would be best? Or is there another?

  1. VM the Windows setup to serve as our AD server on TrueNAS and attach this new SSD machine via a network connection. Nextcloud is presumed to sit on top of the TrueNAS as a VM in this instance, but could also be on its own hardware.

  2. Have a separate physical AD server, separate physical TrueNAS server, and separate SSD machine. Again, Nextcloud as VM on TrueNAS.

  3. Have a separate physical AD server, combined TrueNAS server and SSD server using the same hardware as listed above but with another SSD HBA? Again, Nextcloud as VM on TrueNAS.

Any advice appreciated!

2

You are asking question that are probably better answered by a consultant or specialist.

iX systems would be one source on what the TrueNAS systems can handle and the issue of separate TrueNAS systems for disk and flash storage.

You also look like you should consider fiber networking from your description of your current setup and not knowing where you are on the small to medium enterprise scale, SME. You talk of mixing database, small and large files and we don’t know if you use different datasets for different data types. Block storage? Current pool size, data growth rate and how full your current sets are now? Expected expansion of the current and future system.

Have you looked at backup solutions like Veeam for your Windows VM (Revit)

Adding link on 10G base T
https://forums.truenas.com/t/10gbase-t-best-to-avoid-it-if-you-can/1594

3

Thanks, SmallBarky. Yeah, I have considered that iX may be the right option on this, but I thought mileage may vary from others with similar experience. Regarding database, small, and large files, those are specific to laser scanning files. They stay on the same dataset and must be in the same folder in order to interoperate appropriately. No, it isn’t as fast as it could be, but it’s unfortunately what must be for where this tech is right now.

Regarding my current pool size, it’s 48TiB on the raidz2. It works great. For backup purposes, it scales down to 3TiB actually used because lz4 is doing a great job with a lot of vector-formatted data. However, my VMs are using quite a bit because I’ve had to redo them a few times, hence part of my desire to want to restart this. They’re taking up around 20TiB. Other than that, we’re expected to grow about 10%-20%in data needs per year.

I haven’t looked into Veeam; will do so. I do backup regularly to a cloud provider for all other datasets, but zvols remain the one thing I struggle with.

Oh, and re: 10G, it’s been working well. Yes, it is somewhat power thirsty, but we ran CAT7A everywhere in our office knowing that we aren’t going to be in this building forever as we keep growing.

4

Not sure if you had read Networking Primer but a link to it
https://www.truenas.com/community/resources/10-gig-networking-primer.42/

I just recently set up Intel X520 10g cards. It works fine for home and the only gotcha was the cards wanting to see ‘Intel’ SFP+ transceivers. I just got some from 10GTek on Amazon that had ‘Intel’ coding. Going fiber is a lot of learning though.

You should be able to at least play with some Veeam software in a test environment and see if that is close to helpful. I am sure there are other options too

5

Don’t virtualize Active Directory on TrueNAS.

You could get away with setting up ProxMox as a hypervisor and then virualizing both the TrueNAS instance and the AD/Windows instance… Seems like you’d want two windows instances… one that has its zvol hosted on TrueNAS… and dependent on it, and another that is separate.

Which leads back to having a separate AD instance.

But that doesn’t mean you need the main windows instance to be separate.

Oh Dear.

1 Like
6

Thanks, Stux. Makes sense not to virtualize AD…honestly I don’t like virtualizing in general for all of the challenges Core has had with my instances.

I think I could also run the AD instance independently and then just backup to the TrueNAS device instead of virtualizing anything Windows at all. When I first set up this server, I considered going full Windows, but I didn’t like being beholden to their filesystem. ZFS FTW!

7

Wouldn’t you still have the same chicken-egg or egg-chicken problem if your separate AD server goes down and your back up is on TrueNAS and TrueNAS needs the AD to boot completely? Am I misunderstanding the difference

8

You would.
BUT - you could then (worst case) rebuild the primary from the secondary and go back to working properly.
For Active Directory - its vital to have a backup / secondary. It doesn’t matter nearly as much where the secondary copy is

9

Why do you run Nextcloud in a VM? It’s just a “LAMP” application. Runs perfectly well with much lower overhead and better access to the actual data in a FreeBSD jail or a container on Linux.