Dragonfish-24.04.2.5
IBM X3100 M4
Intel(R) Xeon(R) CPU E31220 @ 3.10GHz
16GB RAM ECC (gonna try upgrade to 32GB)
2xWD Gold Enterprise Class HDD 6TB, 7200 RPM (Mirror)
Approximately 80 users at the same time.
Everything worked perfectly for 3 years, but with the growth of the company/users, for the past few weeks I’ve been experiencing crashes and slowdowns.
Initially, I believe it’s due to insufficient RAM for the ZFS Cache.
However, I understand very little in general, and I would like to understand if there is anything better that can be done.
Like replacing the HDDs with SSDs or something like that.
My $.02 would be to do as you noted, start with adding more memory. I’m also not certain on this as I’d have to confirm but the drives in a mirror config could be slowing you down on the writes as it has to execute the operation twice. The reads not so much as it could pull from either drive, but a write is the most expensive operation. You could explore adding a few drives to the mix and doing away with the mirror for a better use of ZFS like a RAIDZ1, RAIDZ2 or RAIDZ3.
Thanks for the money haha… When you mention RAIDZ1, would I have to add another 6GB HDD, and theoretically it would be faster, or is it just for security?
I read about using SSDs as Special VDEV, does that make sense?
I didn’t mention it, but it’s an accounting firm, so most of the files are spreadsheets, PDFs, and small files.
Perhaps I’ve misspelled the term, but these “crashes” would refer to the Windows file explorer simply freezing and becoming unresponsive and this happens when you’re just browsing through folders.
Yeah, with memory prices now it’s gonna be tough but believe it’s gonna give you the biggest bang for the buck. 32GB will help but 64-96 is where you’ll see the biggest benefit IMO.
Given you’ve got a MIRROR now I don’t believe you can just add a disk and convert it to a RAIDZ pool. You will need to create a new pool and migrate the data over to it. Then with that done you can add SSD disks for things like logs, cache and such for the pool.
The crashes (or Windows File Explorer going unresponsive) could be a symptom of some SMB configuration tweaks or the system just paging memory and such and causing the delay. I’ve also seen where SMB has issues when there are lots of small files. This again points back to more memory unfortunately.
Also, on the main dashboard of your system, what does the memory utilization breakdown look like when running normal and when your seeing the “crashes”? Running any other services on the system like VMs or Apps?
Yeah, the motherboard is kinda old and only supports up to 32GB, so the most immediate option is to increase the RAM.
And having more time to look for another server to replace it.
I cant post image or links, so here:
prnt.sc /Lsmzp_aWPS7V
prnt.sc /ozUOiAxCEct3
No VM and the only application is Duplicati (and we are doing a specific restore with it right now). That’s why I think the service is using 8GB, as it’s normally less.
I would think about server replacement. Docs show that server tops out at 32GB of RAM. You are also limited in how many drives it can take. Link at bottom.
How many available hard drive ports are there? If you can add two more 6TB or larger HD, You could add another VDEV of a mirror pair for a bit more performance. You would end up with two mirror vdevs of 2 drives wide. 32 GB of RAM would be a good idea. I would stay away from special VDEVs, except L2ARC or SLOG (if you have sync writes). Both are covered in basics section.
One option could be contacting TrueNAS Enterprise and go over the Mini line or Enterprise lines for replacement servers to see what prices and recommendations they would have.
Linking Basics since I don’t know your knowledge level.
BASICS
iX Systems pool layout whitepaper
Special VDEV (sVDEV) Planning, Sizing, and Considerations
SLOG is not a write cache. You only need it for sync writes. So databases, Block storage (iSCSI, zvols for VMs), and NFS.
as already said, add RAM should be easy and cheap (DDR3 ECC, at least on my local market, aren’t growth in price).
You can also evaluate to switch to an E3 1230v2~1240v2, for less than 20€, to gain more CPU power and 1600mhz frequency for RAM (E3 1220 is capable only of 1333, unless you didn’t mean the 1220v2).
These are cheap but quite marginal upgrade IMHO, i don’t see any game changer that can solve all your issue without change platform… and this specific period is not so good to change PC in general
Maybe a chinese X99 platform + Xeon e5 can keep the costs low, but you have to really be carefull on what you buy as motherboard, some of them are really crap ( personally experienced )
And @oxyde I’m very familiar with the X99 platform. I have another server with a Qiyida X99 motherboard for a different purpose, and it’s working very well. And I just bought the RAM online. As you said, they’re DDR3, they’re old and not expensive.
Anyway, thanks everyone for sharing your opinion and expertise.
Personally, since this is for work and 80 users,as noted, more ram for sure.
Are the files they access lots of smaller files, or larger files..
Is bandwidth an issue also , are you only connected with 1Gb or more?
If money is a concern, personally at home I use Dell T5820 towers with Xeon W’s in them or used HP Z6 G4 towers which can take 1st/2nd Gen Xeon scalable, but, they use DDR4 ECC which is more expensive now.
Mostly small files (maybe about 85% of the 1.3TB of files), spreadsheets, PDFs, XML, PNG, and things like that. It’s a accounting company. Bandwidth is 1GB (I’m from Brazil).
So being there it is harder to get used gear a little newer at any decent price! (I lived in Central America for 14 years so I can relate to the pain of getting good hardware at a decent price!)
You should check that your “Enable ATime” settings for each dataset are set to False. If they are set to True, then every read will incur an update of the Last Accessed time for each file placing stress on the system that may not be necessary.
Memory upgrades are probably the lowest-effort win here.
DDR3 ECC is still easy to find and surprisingly cheap in most markets, so bumping RAM isn’t a move that is going to hurt that much. (cost-wise).
You could also squeeze a bit more life out of the box by dropping in an E3-1230v2 or 1240v2. They’re dirt cheap these days and give you a small CPU bump plus faster RAM support at 1600MHz. The 1220 tops out at 1333MHz, unless you’re talking about the v2 specifically.
If I were you, with current hardware prices sheeshhh, now isn’t exactly the ideal moment to jump ship either.
)