My TrueNAS journey - what to do in the future?

TrueNAS General
1

Hi - newbie to the forum.

A bit over a decade ago I purchased a 12 Bay NAS running a proprietary fork of FreeNAS, so I got familiar with the software pretty well. It ran 12x 2TB drives in Z2.

Fast forward until a few months ago, when the NAS died. Long story short, I wasn’t aware FreeNAS was running on an internal USB (the proprietary nature did not allow firmware updates) and to be honest I’d taken my eye of the whole TrueNAS game for quite a while!

Ok, so I have fixed the hardware, upgraded to TrueNAS Scale CE, moving to a mirrored boot off 2 SATA SSD’s in a couple of the bays and loaded the data onto a Z2 pool of now 8 drives. Same drives that were in the previous pool. I had to reload these in TrueNAS core, so Linux could read them (they are hardware SEC encrypted). Again. Another more hrs upgrading.

So all up -This upgrade hasn’t been without a steep learning curve - but I’ve got there. With very little additional capital expense.

The MB (A Supermicro 2012 X9SCL-F) needed the bios upgraded. It already has plenty of RAM 32G. But It’s only capable of SATA II. I’ve sourced a suitable 10GB SFH+ card from eBay. I had to upgrade the Hardware raid controller (that seemed to work with FreeNAS 8.2- but not so great with TrueNAS scale) to IT mode (from IR mode)- which wasn’t straight forward but I got there in the end.

My question is where to from now?

Do I put in new SATA SDD’s for data? Or would this be a waste of (given the mB is limited to SATAII?)

Should I plan another build - like to a M.2 based SATA set up?

Thanks!

2

Well, it depends on your workload. Are you satisfied with your build? If no, why?

IMO, the main advantage of the SSD over the HDD is high IOPS. SATAII is limited at 3Gbps, which is 375MBps. If we divide it by 4KB, we get a 96K IOPS limit. So, unless you want an SSD for higher throughput, 3G SATAII is perfectly fine by me.

EDIT: Actual numbers are a bit different. Yet, my point stays the same.

1 Like
3

Thanks for the reply-

I’m fine with what I’ve done with the MB at the moment(was almost enterprise grade in its day) - with 32 RAM and now the 10gb card it’s great.

It’s more about what storage to roll out, that’s all. The 2TB discs are getting a bit old. Some have failed. A few bad sectors. Then there is the issue of reusing any of the remaining others - given they are hardware encrypted and this has become a PITA with TN Scale.

I was thinking if I upgrade to SSD Sata and stick them in the old bays (with adapters) if/when I migrate the pool (with the whole world it seems moving to solid state storage) it’s going to be a heck of a lot easier migrating SSD’s than discs. Make sense?

But look how quickly the NAS world moved onto M.2 Sata?! In my part of the world a 4TB M.2 2280 NVMe PCIe 4.0 SSD can be sourced actually cheaper than a 2.5 inch SATAIII SSD. And seriously only about $35-40 USD more expensive than an older tech spinner 4TB HDD. Maybe I should move to M.2 Sata right now?! (are there 3.5 inch M2. Sata to SataII adapters?) …. Would this even work with a 13 yr old MB?… get the jist of the question?

4

Perhaps, you mean m.2 NVMe.

Depends on the country. And, IMO, you shouldn’t use QLC SSDs for write-intensive workloads.

I never used such an adapter. I guess it would work. However, it would work only with m.2 SATA drives, not with m.2 NVMe drives. Just a quick reference of differences between the two.

5

That’s only a limitation for SSDs, not for spinning drives.

I hope this really is a HBA and not a RAID controller.

Terminology issue here… M.2 SATA drives are a dying breed.

Where do you want to go? What’s the point of another build now that you’ve spent some time restoring and upgrading the old X9SCL?

1 Like
6

It has an LSI SAS controller- I thought that was a RAID controller. I don’t know the difference between a RAID and an HBA controller- sorry.

I’ve been following Groks advice lol !

Yes. A very very steep learning curve as I said! Heck I even had to buy an old PS/2 keyboard and work out how to hook up an old 9 pin monitor! Lucky I had that still around! And what a bonus having a monitor has been! How many people buy a new NAS these days without an HDMI port - go to load TrueNAS and suddenly realise they can’t do it without a monitor!!! At least I have my 20 yr old one now working- and it was extremely useful when I came to upgrading to TrueNAS!

AI has been incredibly helpful getting me to this point and sorting out the problems with this old motherboard- like I said, it helped me work out what the initial alarms meant, why the chip was overheating, why it was failing to boot (it was booting off a corrupted USB) - what to do about that (boot off an SSD in one of the front drive bays ) how to load TrueNAS- YouTube was awesome working out the difference between CORE and SCALE etc. Back to AI again and a deep dive into the bowels of the Supermicro downloads archives - How to set and upgrade the BIOS and why loading an IT (it had an IR) driver on the LSI SAS controller was so important. That bit wasn’t easy as I said- finding out how to do that on a 15 year old SAs controller

Anyway Like I said a steep learning curve as each point along the way I had to work out why things weren’t working so well :grinning: :grinning:

The last stumbling block were the Hitachi 2TB drives. They are hardware encrypted. They were working fine on the old FreeNAS. How the heck was I supposed to know that wouldn’t be fine on the new Linux based TrueNAS! I haven’t been focused on TrueNAS the last 10 years!

So I’ve learnt so so much tinkering with this rebuild and I’m finally here!!!

Regarding which drives to use moving forward. Sure I’m happy with the old Supermicro set up. It’s awesome- fast. 32 RAM, nothing wrong with power supply etc. Be real shame to send her to the dumpster. But I’m not happy with the drives.

Hence the question what to do?- cause I will eventually need 8 more of them. And I’d like to go 4TB each. That’s not an insignificant investment - I don’t want to rip up $2,000 bucks on the wrong drive choice.

The issue is this. When it comes to retire this old NAS (and that day will come as she is already 15 yrs old) If I stay with HDDs (and the NAS ecosystem moves quickly on to SSD’s - like it appears to already be doing) how the heck do I easily move my Z2 pool with 8 drives onto a non HDD NAS?

Second point. If I go to SSD’s now… and like you say M.2 Sata storage already is looking obsolete- how do I get M.2 NVME storage into the old 3.5 inch Sata II bays and which actual SSD NVME drives do I purchase?

7

0dVNyc0
0dVNyc01280×720 100 KB

Seriously. You can’t connect an nvme drive to a sata port. Even with an adapter.

2 Likes
8

My advise would be to build a new NAS, and use your old NAS as a backup to it.

So, the real question should be do what do you want your new NAS to be.

For example you could use a mini pc with 3 NVMe slots to build a mirrored flash system…

3 Likes
9

Ok!!! Got it! Thanks for all the advice.

I’m going to buy some cheap refurbished 4TB HDDs for this old hardware super cheap only 20bucks a drive! - upgrade my Z2 pool to a bit under 30TB or whatever - then build another NVME based NAS.

Awesome help everyone- appreciate it.

How do ya’ll transfer/copy your pools in a situation like that?

Do you

1/Replicate the pool onto one large HDD- put that in a USB drive dock - attach that to the new SSD build? Or

2/ just copy the data over by mounting both pools SMB etc via the network?

Thanks!

10

I suspect that replication is a preferable method. Never did it myself, so can’t say for sure.

Transferring via external USB drive looks amateurish. Especially when you have a 10Gbps network card.

2 Likes
11

Cool. I agree

Which NVME set up would you recommend?

Something like a ASUSTOR Flashstore?… I notice the current models don’t have a video port…would make upgrading that to TN difficult

Or links to other builds that work?

I’d like to stay using TrueNAS Scale CE both setups. And run around 30TB (bit under) total capacity pools

Cheers

12

If you for some reason specifically asked me, then I can’t really answer. Because I didn’t ever have all SSD NAS. And I don’t plan to have one in the near future[1]. I think you should create a separate topic for all-flash NAS.

I have read the thread about the terramaster F8 plus. From my POV this is bad, subpar or mediocre in almost every aspect except being compact. The same goes for ASUSTOR Flashstor (I didn’t inspect the entire product line).

  1. Well, I had some thoughts about it when first saw the (price of) 16TB D5-5316. But with any reasonable (for me) capacity NAS, I would need at least 4 of them. Then the price goes up to ~5k, only for the drives… So it’s very expensive (for home use), and I’m actually don’t need such high performance. ↩︎

1 Like
13

Sure - no stress -you’re been very vey helpful.

Thanks.

1 Like
14

delving deeper, the Motherboard has ECC RAM. And looks like my HBA - an LSI 9211-8i (now confirmed flashed for IT mode) has 8 direct feeds into 8 of the bays - so I should be able to run SAS drives. Buying a cheap one for 15 bucks to trial out… then the plan is to get 8x 4TB 6Gb/s new SAS drives (“unopened”/unused) for around 65 bucks each… maybe a bit more. That sort of size enterprise seems to be going real cheap ATM… hopefully I can isolate which bays that HBA is connected to, in order to maintain an 8X ZF2 SAS pool… onwards and upwards!

1 Like
15

IMG_6043
IMG_6043480×640 150 KB

HBA left (red arrow) should be able to manage SAS drives. Drive bays connected direct into MB (SATA) right (blue arrow)

16

Ok update : I am nearly there!

1/ Why I am doing this? It proves there is no need to throw out enterprise HW into the dustbin!

2/ I have shown TruesNAS Scale CE is stable on this old MB with ECC RAM.

3/ I am dual booting via mirrored SSD’s directly attached to the SATA bays. TrueNAs will boot off each SSD from either bay (if one dies) : confirmed

3a/ BIOS settings are stable booting in UEFI mode. Bios FW updated to (2.3a) 2021

4/ The HBA is confirmed in IT mode. All Storage HDD’s are attached to HBA controlled bays.

5/ I have confirmed 2x 4TB WD Red Plus drives work well adding to the pool. (I am slowly replacing the old Hitachi drives as they keep getting more sector errors)… gradually updating resilvering until I have a new ZF2 pool (double the size). SAS drives wouldn’t fit the bays. These WD HDD’s are new and should be fine and do the job.

6/ All good! This can be done. And has been a (very cheap) and worthwhile project. I’ve also learnt so much and am confident I can completely rebuild a ZFS based server if the entire thing comes crashing down.

7/ Redundancy. The ZF2 pool is entirely replicated on a separate one stripped large WD drive. This can be removed from the bay and taken to a different site, or be used to quickly rebuild the ZFS data on a different device (if this old NAS dies). Logic here is it is quicker doing this with just the one drive. I am also backing up essential data to AWS S3 deep archive.

Onwards and upwards!

Go TrueNAS Scale!

17

But I continue to get the following input output error trying to add brand new disks. It is very frustrating. it seems to come and go and is not drive bay specific. Am I doing anything wrong?

Traceback (most recent call last):
File “/usr/lib/python3/dist-packages/middlewared/job.py”, line 515, in run
await self.future
File “/usr/lib/python3/dist-packages/middlewared/job.py”, line 560, in run_body
rv = await self.method(*args)
^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/schema/processor.py”, line 174, in nf
return await func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/schema/processor.py”, line 48, in nf
res = await f(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/plugins/pool/replace_disk.py", line 92, in replace
await self.middleware.call(‘pool.format_disks’, job, {
File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 1000, in call
return await self.call(
^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 715, in call
return await methodobj(*prepared_call.args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/plugins/pool/format_disks.py", line 29, in format_disks
await asyncio_map(unlock_and_format_disk, disks.items(), limit=16)
File "/usr/lib/python3/dist-packages/middlewared/utils/asyncio.py", line 19, in asyncio_map
return await asyncio.gather(*futures)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/utils/asyncio.py", line 16, in func
return await real_func(arg)
^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/plugins/pool_/format_disks.py”, line 24, in unlock_and_format_disk
await self.middleware.call(‘disk.format’, disk, config.get(‘size’))
File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 1000, in call
return await self._call(
^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 726, in call
return await self.run_in_executor(prepared_call.executor, methodobj, *prepared_call.args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 619, in run_in_executor
return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3.11/concurrent/futures/thread.py”, line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/plugins/disk/format.py", line 33, in format
self.middleware.call_sync(‘disk.wipe’, disk, ‘QUICK’, False).wait_sync(raise_error=True)
File “/usr/lib/python3/dist-packages/middlewared/job.py”, line 487, in wait_sync
raise CallError(self.error)
middlewared.service_exception.CallError: [EFAULT] [Errno 5] Input/output error

18

It seems to be something with the HBA>?? or firmware for it? As I can initialise the new discs no problem using a vacant bay directly attached to the MB via SATA… (not one of the HBA bays)… Will this be a problem moving forward? say if I move this disc back to an HBA slot after it has been added to the pool? thanks!