I purchased an iXsystems Mini XL+ and 8x WD Red Pro “14 TB” 7200 RPM Drives WD142KFGX (new, on the approved list for the XL+) from Amazon in April. It shipped with Core, and I migrated it to Scale 24.04. All 8 drives are in a single RAIDZ2 vdev. It gets light use, maybe 2-4 hours a day.
Some of the drive temperatures would remain in the high 50s even hours after most usage has stopped, and I’ve seen two drives peak at 60 degrees during a scrub. This is within the 0-65 degrees operating temperature spec https://documents.westerndigital.com/content/dam/doc-library/en_us/assets/public/western-digital/product/internal-drives/wd-red-pro-hdd/product-brief-western-digital-wd-red-pro-hdd.pdf but is concerning. After enabling the HDD Standby feature (drive goes into standby after a 5 minute period of inactivity) and disabling k3s (even without k3s managed apps, k3s itself wrote logs to disk every 5s), temperatures now stabilize below 40 degrees. Would rather keep the drives on all the time, but this felt the lesser of two evils.
I’m looking for advice on how to better cool the drives and otherwise tune the system. I’m considering:
The old miniXL was also a an excellent HDD sous-vide toaster. I replaced it with a Lian Li q26 once the futility of dealing with the airflow inside the old case became apparent. (I upgraded to an industrial Noctua fan, toggled the IPMI settings, etc).
Good luck. Small cases that can hold 8 HDDs (aside the q26) frequently prioritize compactness for cooling performance. Works ok with SSDs since those tend to be more heat tolerant and allow better airflow (less bulky).
Ixsystems has allegedly told another customer that 50+C is in spec. as long as drives stay below 60C. That’s nuts to me but my pools are tiny, sample size is small.
I’m willing to get a new case if I thought it would do significantly better. I do have a preference for smaller cases (apartment living, no rack to mount a big case in) and am not aware of a good option here. I’ll have to do some research and see what I can find. Suggestions welcome.
The post SmallBarky linked suggests tuning fan speeds via IPMI. I’ll definitely give that a shot. I plan on running a series of “experiments” and will share my findings here. For now, I expect to test 1.a baseline/control configuration, 2.my current configuration (HDD Standby enabled), 3.fans running at higher speed tuned via IPMI and 4.fans controlled by hddfancontrol. Possibly more if I come across other ideas. For each test I’ll use rsync to generate enough writes to reach a high temperature and will collect various measurements (starting HDD temperatures, peak temperatures, amount of time to reach peak, amount of time until temperatures cool off to a stable level once load terminates, what that lower stable temperature is, etc.).
After evaluating the proposed options, I have the following findings:
1. Controlling the Rear Fan (FAN A) Speed: A modified version of spinpid2 can be successfully used to control the rear fan speed. However, even with the fan running at 100%, temperatures consistently remain above 40°C. 2. Replacing the Rear Fan: The original rear fan is a Nanoxia Deep Silence 120 mm, 2,000 rpm. This fan performs adequately, and no replacement was found that would significantly improve airflow. 3. Upgrading the BIOS: Since this issue is related to airflow, it is not believed that upgrading the BIOS would yield any significant improvement. 4. Changing BIOS Settings: This option has similar limitations to the BIOS upgrade.
Consideration is being given to other option:
5. Pushing air directly to the drives: This may not be the cleanest or quietest solution, and it is very probable that it will cause problems in the future.
The trick with cooling HDs with fans is to ensure that all the airflow the fan generates results in air flowing passed the drives, ie all sources of alternative air flow to the fan from outside need to be blocked so that the fan draws air through the HD bay holes and around the drives.
FWIW, I replaced the stock fan in my XL with the fan that @Stux references. It made a difference but it didn’t fully fix the problem.
The issues in the XL case are:
pulling air through a tight stack of hard drives creates a lot of static pressure drop
a lot of that air bypasses the stack and leaks from the motherboard area below
the two top drives were always hotter than the rest, suggesting even less airflow up there
a single standard computer case fan is usually not adequate to remove the heat of 8 operating drives.
Before I found the amazing Q26A case (3x120mm in the front, more in the back, less static pressure drop due to more generous drive spacing, etc) I even contemplated manually adding more holes and fans to the top and back of the XL case. But I stopped after looking over the design of the drive tower.
If I was wedded to a XL case and desired normal drive temperatures (especially with air-filled HDDs), I’d fabricate a adapter for the front of the HDD array and mate it to a variable-speed, 6” or 8” axial fan. The adapter would not have to be fancy, just reasonably air tight, forcing most of the air from the axial fan through the tight confines of the HDD array, cooling it.
I built something similar out of cardboard once to allow standard box fan to cool our refrigerator condenser HX while the broken condenser fan motor replacement was on order. Worked well enough for over a week and kept queen bee happy.
You might even be able to use three of those noctua industrial fans to the same effect - ie print a adapter, mount it in front with the three fans in it but I worry that the noctua fans
a) wouldn’t have a favorable performance due to the pressure drop
b) cost more than a quality axial fan like I linked to
Either way, the external fan arrangement kills one of the main reasons why XL drive cases are popular in the first place, ie quick drive removal and replacement. Given the longevity of HDDs under normal circumstances, I’m happy to take a bit more time to get the drive out via my Lian Li case, which also presents the serial number of each drive in the stack once I take the side panel off. It’s foolproof compared to the XL.
The recommendations provided by @Stux effectively resolved the issue.
The main key factor was blocking all external sources of alternative airflow to the fan. This adjustment ensured that, even with the original 2000 RPM fan, the maximum temperature of any drive, even during heavy load, never exceeded 42°C.
Subsequently, the installation of an NF-F12 industrial PPC 3000 RPM fan further reduced the maximum temperature to at most 40°C. Although this fan is quite noisy, it performs its function effectively.
To maintain optimal airflow to keep the hard drives temperature below 40°C and minimize noise levels, I opted for trying the ARCTIC P12 Max.
After three weeks of testing, it proved to be a highly efficient cooling solution.
