FANGTOOTH observations / issues

TrueNAS General
1

I have been using ZFS well pretty much since it came out

for the last almost 10 years… been running NAPPIT / ominos all in one on a ESXI host feeding NFS shares back to ESXI for vm and storage and SMB for NAS duty…

I wanted to try out SCALE in this role and here are first some issues with FANGTOOTH then some observations

1> where did all the performance charts go… ZFS only has an ARC chart… pretty sure there use to be more… like ZIL and ARC stats . not just ARC size… in general I feel that Truenas really does not present enough of its most core function data (ie ZFS) to the user

2> in this all-in-one role, I assign truenas 2 virtual NICS… one for LAN traffic, and one for a software defined internal to ESXI host virtual switch / network for NFS traffic just to ESXI for vm storage and between VMs for fast data … iperf3 reports 15gb/s on this link BUT this second Nic was a bear to set up… took multiple attempts and at least an hour… didnt write down the steps and should have but it was a lot more painful than populating the setup

3> in this all-in-one role or just for stability and performance in general… I wish there were more buttons for stopping un needed services and such from middleware that are not needed… for example… if I just wanted to use ZFS and NAS features… I should be able to have clean interface and not a bunch of deamons and process running for services that are not needed

4> back to Truenas not providing what I consider must have info for a ZFS appliance…

  • when scrubs are running … where the hell is the stats… nowhere to be found without command line. I want to see the typical stats that zpool status would show during a scrub at a minimum
  • disk smart info… non existent without command line… just seeing a pass is not what I want to see if I am looking for disk health… or trending relocated sectors etc…
  • arcstat and zilstat… you have them in the command line… why dont you have these critical stats presented in the gui…
  • Disk charts… labels are verbose and almost useless… I prefer unlike most… to have stats, locations, drives data presented by serial number as the serial number is what I tag the disk slide with… also … no way to indentify a drive in an array from command line for replacement to light up its activity led… gents… have you never heard of dd? in DISKS each drive should have a ID/DD button… press it… array sends dd commands to the drive… lights blink… ah … we found the right drive… clearly these devs have never looked at a wall of drives looking for the one to replace using truenas…

5> nvme is all but useless in the gui… no nvme smart… not even temps… almost everything needs to be done in the command line with nvmetool

In general ZFS has always been a very command line driven storage solution … and most people not even todays DEVs understand how to use it properly…

that said. … nobody had done a great job of a GUI…

Take the solaris storage appliance demo / trainer for a spin… that one is pretty good… especially with some bolt ons from their dev labs… you could actually log and graph latency changes at the drives if loud noise or minor vibrations were present…

Nappit does a ok job of giving more stats… but the '80s called and want their GUI back… its got a dated flavor and kinda klunky … 1 step better than command line… BUT unlike truenas … I dont have to use the command line hardly at all to see the critical ZFS and disk stats that I need.

Truenas devs hate it when we use command line… but their product design demands it for anyone other than a housewife running a NAS for their kids homework storage …

4 Likes
2

Were you using the UI or the console tool?

I too have multiple interfaces, one of which is a bonded pair of 10Gbps SFP+ ports for my LAN. I had no issues setting up either the primary or secondary (DMZ on 1Gbps ethernet) interfaces. BUT … it is different than FreeBSD or Solaris, I found the tutorial here Interface Configurations | TrueNAS Documentation Hub more useful than the UI reference. I probably could have eventually figured it out without the tutorial, but the tutorial gave me the clues I needed to get it done right the first time.

3

Many of the services (Apps/Docker, Instances/Incus, Shares) do not start at boot until you first configure them. If you configure Apps to play with, you can unconfigure and that service will stop and not restart.

Most of the middleware that is running is TrueNAS. It is the layer that provides the services that enable the browser (and now API) based user interfaces, provide logging, provide monitoring, provide notifications, and much more.

4

In the Storage screen the ZFS Health brick for the zpool in question will display estimated time to completion. I agree that having all of the underlying information would be nice for power users like us, but for the general purpose users (who greatly outnumber us) they need to keep the UI simple.

I am hopeful that one day there will be a global switch to enable power user mode :slight_smile:

5

Paul

thanks for taking the time to reply… and I know who you are so… thanks for the engagement…

I was doing as much as possible during provision from the gui… obviously some steps were likely done initially from the initial iso boot interview and setup …

adding the second ‘virtual’ Nic for the internal network … the vm was provisioned with 2 nics during ISO install… and the setup I believe I was trying to do from they 25.01 release gui… not the beta…

as for ‘power user’ mode…
the power of ZFS rests in 2 things… the unbelievable flexibility of provisioning options, monitoring… and lastly absolute data integrity . or at least that last one was the SUN/Oracle focus until openzfs came along and it became a race of feature additions and keep-up with the joneses (oracle)… oracle ZFS is still the gold standard… but since they locked it up… openzfs is the only game left in town … the ominos guys are slow… but methodical … but always behind in zfs feature implementation …and even drivers… but they focus on reliability

anywho… back to ZFS monitoring … it should not be a power feature or very hard to show during a scrub… the basic stats as that is the first giveaway of an issue… and I can see faults as they are piling up if there is an issue long before the scrub completes… maybe if there are enough issues on a bit pool right at the start… I might have to cancel the scrub thrashing the drives… and take steps to remediate using the last remaining life of those drives… also … plenty of gui real estate your not using … should not have to go to the reports page… if I pull up a dataset… you could have some live stats right there…

also you didnt mention were the zfs graphs are reports went in fangtooth… are you brining these ZFS charts back… just having an arc size chart is well… useless… as the arc typically will just grow to its defined size and stay there… but again… tons of ZFS stats that should be historically available by chart to tune and monitor

also … no NFS charts any more… or other NFS or SMB stats…

while I am at it… this is a BIG one…

you guys ever going to get around to replacing or at least giving the option of using KSMBD to replace samba… as KSMBD performance is WAY better being run in kernel space just like the original SOLARIS ZFS intended …

6

If you are new to TrueNAS, there’s been some changes.

Netdata reporting was moved to an App for security reasons: Netdata | TrueNAS Apps Market
Try it out and provide your feedback.

The WebUI only uses the TrueNAS API which was REST and rather static. Netdata bypasses that API.

Fangtooth enables an improved websocket API. That API is better for relaying real-time info and will enable improved integrated UI in the future. api.truenas.com Power users are welcome to use this API for super cool things (please share).

Future versions of TrueNAS can improve the WebUI with that. Any feedback can be made on Feature Requests. category.

7

@duecedriver
You make quite a few good points which I do agree with. I too would like to see the stats and in a better (what I would like) layout.

My little script, which is not part of TrueNAS, does perform NVMe tests, and a few other things. Look for Multi-Report in the resources. But it does not address a great deal of what you are talking about.

As @Captain_Morgan has said, there is an API which those advanced users can utilize to create a new personal interface/GUI if you desire and a lot more. I’m not a programmer so that is out of my reach for now. Also there has been a lot of hardening of TrueNAS for security. It has bit me as well but I’d rather have a more secure system than one a hacker could easily infiltrate.

If you are able and willing to create some new GUI interface or add-ons, please do share. There are a lot of us out here who love data that allows us to make smarter decisions. That wall of drives part you mentioned, I cannot imagine but it does not sound like fun.

8

truenas is an APPLIANCE / platform

it is suppose to take the advanced and complex nature of ZFS and distil it and make it more user friendly

I dont want to run a front end … that takes up resources and introduces potential issues… then only to have to build my own scripts and API calls … and resort to other 3rd party tool like netadata…

I would just load up a basic terminal based install of solaris or ominos and control it from the command line like I mostly have been doing for nearly 2 decades…

after the length of time truenas has been in development… it should be doing more… not with fancy bolt on features … but with its CORE function of being a front end to the ZFS file system first and foremost…

1 Like
9

We don’t disagree with doing more… The new API enables us to do that.

We don’t see TrueNAS as a front-end to ZFS… we see it as a flexible NAS that uses ZFS.

There are three choices for each user:

CLI only
WebUI + CLI
WebUI + CLI + Netdata

If you need more , Feature Requests is the process… we then count votes.

We’re always looking for developers that want to contribute.

10

well that is my point now isn’t it…

your an ‘enterprise’ NAS with ZFS backing

its all about the DATA… and its stored on ZFS

if a storage admin has to keep dropping into CLI for the basics or monitoring and health let alone turn basic knobs… what’s the point. again… I would set up a ZFS based CLI distro and forego the expense, stability, and headache of using a front end… that rather than making me productive, slows me down as I jump from gui to cli to api to 3rd party tool and back again

2 Likes
11

We have record revenues on a Enterprise appliances and storage admins don’t have to do that… its just works. IT includes HA and Vmware/Veam certifications.

The people who want the visibility you are requesting are ZFS experts… most TrueNAS users are not ZFS experts. They are comfortable with the CLI.

12

record revenues for ixsystems perhaps…

but spouting revenue over customer satisfaction is why this in not in any production environment us ‘zfs experts’ or ‘storage experts’ would admin…

2024 revenue
ix systems… what 24 million
NetApp – Net 6.7 billion for fiscal year 2024
Oracle Total Revenue $53.0 billion
hell QNAP is over 300 million

so even if you dont care about building the BEST ZFS solution …
your not the highest grossing either… likely because you dont have the feature set us ‘experts’ billion dollar global multinational employers want to spend big dollars on…

you want to focus on SMB (small business) with no storage admins… fine… home labs … go for it… I will check back in another couple years to test drive it again…

I bill out at over 250 an hour… so while you might be looking for devs you likely cant afford me… but

if my free advice isn’t worth that cost… perhaps you cant recognize value …

1 Like
13

Happy to have the advice… the right channel is the Feature Requests.

Generally, we handle bug reports and feature requests. To recommend a change in business strategy there is a lot more information that is not available to you.

We are not as big as NetApp and Oracle… but out storage business is growing faster than theirs. Your revenue estimates were way low for us.

We also have a higher Gartner Peer Insights rating with 4.9 out of 5.
https://www.gartner.com/reviews/market/primary-storage-platforms/vendor/truenas/product/truenas-enterprise

You have a lot of expertise and experience…if TrueNAS doesn’t meet your needs, OK. If it does, we’d love your advice on what’s most important to do next.

Replacing Netdata wasn’t the highest on our list…but that’s the goal of Feature Requests to gather data on what people view as important. We get the same data from our customers.

14

not looking to get into a pissing contest… but the rev figures for IX were per a statement by your own CEO…

“Michael Lauth shares how iXsystems grew to $23.7M over the past 23 years.”

and with just shy of what 200 employees … that’s $120,000 in revenue per employee, not exactly burning down the house … so dont brag to me about how you dont need to hear about how your product can deliver greater service to your users because you have great revenue… and then tell me your not as big as so and so… who by the way do it better…

I have a BS in software engineering, masters in aerospace and physiology which also makes me a subject matter expert on human factors and machine human interfaces .

coupled with over 40 years of real world behind the keyboard time and I can assure you I know of what I speak

in the datacenter like in golf… the old saying drive for show… put for dough is replaced with compute is easy … data is business critical… every C suite I brief… you dont bring compute to a ransomware fight… your go-to guy better be your storage fabric and engineer… even wyatt erp knew in a real gunfight… it was better to have doc holiday skin that pistol… with a proper ZFS provisioning … I could leave a firewall wide open… burn the building down after terrorist run amok pulling hard drives out… and in very short order have you back up and running … loose some compute… slows you down… loose just a bit of data… and it could wipe the company out… data is hard… data is the lifeblood of a company… they want rock solid reliable. and when it comes to data… people who go cheap go home unhappy… and the guys that keep that data online with 5 nine availability or higher… dont want the appliance interface to get in the way… they want it to assist in getting the job done… should not have to go to multiple tabs, windows and charts to see what could be presented on one page… and the data that is presented should be presented based on its value on getting insight into what is not right quickly… where the bottle necks, inefficiencies, to drive tuning and optimization exist … etc…

simply … while the truenas is pretty… its not practical… its like dating a supermodel… sure she turns heads… just dont get into a conversation with her… its an empty shell…

if you guys put 10% of the engineering into analytics vs what you spent on the move to incus… you would be moving in the right direction quickly …

1 Like
15

You’d have to share your source for that and date. It nots correct. I think there’s a bad source on the internet (what a surprise).

Latest announcement was here:

https://ktla.com/business/press-releases/ein-presswire/615818533/ixsystems-surpasses-100m-milestone-in-2022-as-enterprise-adoption-of-truenas-open-storage-accelerates/

Again, I’m ok with with the request for more analytics. You are unusual in not liking netdata, but that is your opinion.

However, you don’t understand our business and don’t have the data needed.

16

Couple of points.

As per the link just above - those of us who work in the corporate world know that 100mm of ”bookings” can mean just about anything, depending on the length of said contracts or for that matter whether they are even contractually binding. Relative to hard turnover it’s a different metric altogether so neither supports or contradicts @duecedriver’s quote.

Second, I think the discussion above again exemplifies that the target market as well as direction of TrueNAS is unclear (at least to me). Which, with my professional/corporate hat on, would still rule out bringing it into my workplace. @duecedriver has a view and therefore an expectation on the product, which is unmet - is that because TrueNAS is actually trying to go into a different direction/market, or more a case of relative priorities and execution? Unclear.

Thirdly, @duecedriver if you were called in for a ”briefing” where I work, with that language and tone, it would only happen once for sure. :wink: I think you have some valid points but they are delivered through rants and incomplete sentences… With all due respect.

17

I think it’s fair to say that TrueNAS is going through an identity crisis and has been doing since its conception. I don’t mean that necessarily in a bad way as it’s almost a victim of its own success as it has the ability to be all things to all people. Ask 100 people on this forum how they use TN and whats important to them and you’d probably get 100 different answers. I’ve used TN in Enterprise for the last 10 years and sure there are features I would like to see but appreciate I’m just one of many voices and it’s hard to complain too much when you consider we get this stuff for free. I also used Omni OS / Nappit about 10 years ago and decided FreeNAS worked better for my needs and have stuck with it.

18

The biggest challenge imo. And crowdsourcing product roadmap via the standard answer ”raise a feature request” isn’t going to resolve it either, because that will drive in the direction of those inclined to submit such requests - presumably mainly the homelab/enthusiasts crowd? Is that where iX sees the future of the product and future revenue streams? And if not, then where are the actual paying customers of tomorrow, and where do those needs overlap with the enthusiast crowd? Leading to questions like, on what basis will iX accept or reject feature requests, i.e. what are the higher level objectives that the detail feature requests need to support?

Underlying technology is important of course, but only if it ultimately serves its purpose of solving real problems for hopefully paying customers. Until then it’s just implementation detail and at worst engineering time spent in the wrong areas.

19

Especially when it seems entirely arbitrary which FRs iX will pay attention to. Here’s one with 72 votes that has zero input/response from anyone at iX:

…and this one, with one vote, is accepted:

Obviously they’re very different requests, but 70+ votes is kind of a lot to have zero response from iX–makes it looks like the votes really don’t matter very much.

3 Likes
20

its not that I dont like netdata… nor know how to use it or set it up…
but if I have to use netdata… why use truenas…
moreover… ixsystems removed netdata from fangtooth in order to pass security audits…
again. its not like truenas lacks the arcstat or zilstat commands or the ability to generate the analytics… its the laziness of the gui dev team to incorporate meaningful and prescent analytics and present them in the truenas interface in a way that a zfs admin would find useful and in a workflow that enhances productivity … and not a bunch of clicking around to multiple pages and dropping in and out of shell … which doest remember what you were doing which means I really need to have an ssh shell going next to the gui… its a mess…