Clearing the Air on Build Scripts

Announcements
42

They could publish an HCL that would be a prerequisite to buy “Enterprise as software” from iX.

E.g.

  • me
  • my friends who run TrueNAS because I recommended it
  • my company which runs TrueNAS because it’s operationally superior to anything else

are all running on Supermicro server grade hardware. Easy peasy - no hardware related issues whatsoever. Seriously. Not one problem anywhere in more than a decade.

Yes, that means my central home lab server still running CORE clocks in at four figures €/$ without storage. It works. That saves me a heck of worries and effort.

But then of course given the speed at which our industry tends to move that HCL will be outdated the day it’s published, certified systems will be hard to purchase, people running home labs will still complain … so what’s to gain for iX?

I perfectly understand them not going to do it.

3 Likes
43

This feels like the type of quote that could be lifted straight out of a certain book. “We don’t sell you things to own, we sell you an experience! Depend on us for the experience to keep flowing!”

:roll_eyes:

2 Likes
44

It’d be easy to read these and conclude that TrueNAS only contribute back to open-source software when they want or need something to support a headline feature in the release notes. Sure, they do, but there’s more to it than that.

I’m an OpenZFS dev (something like a “core” developer, though we don’t have that kind of structure exactly). I recently started working at TrueNAS. While there are some specific internal projects that I’m involved in, I’m also here to make sure OpenZFS keeps ticking along, bugs get fixed, things work nicely with new hardware (SHA512 acceleration on Arrow Lake) and new kernels (Linux 7.0!), and so on.

By hiring me and spending proper decent tech-salary money and not tip-jar money, TrueNAS are making a real, active investment in OpenZFS. Obviously that’s good business sense; TrueNAS needs OpenZFS to continue and will directly benefit from that work happening, but so does every other OpenZFS user out there.

But, for the last three years I’ve been doing the same work as a contractor, and let me tell you, it is tournament-level difficult to get companies to pay for long-term ongoing maintenance of the open-source software they build their business on (and some of my clients were household names with multi-billion-dollar market caps, so they certainly weren’t short on cash). So when I was talking to Kris and the team, I said something like “I want to work on these projects with you, but I want to do it the right way, properly integrated into upstream OpenZFS, and I want to make sure we’re doing the long-term work to make sure OpenZFS continues well long into the future”. From recent experience, I was expecting a negotiation. Instead I got baffled stares and “uhh, ok?”, like I’d suggested we should all breathe air or drink water. At that point I was pretty much sold.

Of course, I’m not the only OpenZFS contributor on staff, and OpenZFS is not the only open-source project that TrueNAS staff are contributing to regularly. And I’m not trying to claim that they (“we”, I guess! :sweat_smile:) shouldn’t be called out if something seems off.

But, at least, I feel confident that the management are genuinely trying in good faith to balance everything in a sustainable way - yes, bringing on more customers and making money, but also making sure staff are taken care of and making sure the open-source projects and communities we rely on continue to thrive. I don’t envy them that task, but they’re not giving up even in the face of a hell of a lot of chaos right now (no joke: it is not an easy time to be selling computers).

I know that most of you don’t know me at all, so there’s no reason to think I’m anything other than a company shill (but I swear I did not tell anyone I was writing this, and like, I’m replying directly to my Big Boss, so if I’ve screwed up I’m about to find out!). And trust has to be earned, and it has to be maintained, so I’m not suggesting you shouldn’t be watchful. On the other hand, it has to be possible for that trust to be earned, and its not real trust if even the idea of a misstep is enough to cost you any goodwill you’ve accumulated.

23 Likes
45

Your comments are very well written and do help share a perspective I think hasn’t been spread as much lately as it should, so thanks for speaking up.

That said - on the theme of trust - I’ve found more reasons lately to (slowly) begin revoking trust. Not all at once, but the following question is one that led me to resign at a previous employer and make the leap to the next one:

“Can I name a single intelligent decision from the last six months from management?”

The answer was “No”, and that made me leave for better pastures.

Right now I can’t name a single intelligent decision from iXsystems with regard to TN CE in recent memory. The trust can be reissued on demonstrable actions well documented elsewhere. But right now? All I need is a better offer and TN is like my past employer.

1 Like
46

Can you consider partially reversing the decision of not having any sort of open build system and release if not a full build stack and CI but some sort of bare bones scripts that can only build the CE edition, without any sort of bells and whistles like Secure Boot? Surely with LLMs being pretty good these days at boilerplate like that, some time can be spared for it.
I think the decision to close off build system has created a lot of animosity towards TrueNAS in the community, and a lot of people in the community have a strong belief in the idea of open source. TrueNAS being technically open source but only on paper because you can’t really feasibly build it without writing your own system goes against the spirit of open source, and especially the GPL.
This could backfire in indirect ways like people being less likely to engage with TrueNAS, like go out of their way to test betas and report bugs, contribute useful stuff to it (Proxmox plugin comes to mind), and direct ways like the unofficial ARM version.

1 Like
47

I agree with this.

I love what TrueNAS CE offers, and I think it’s a great system delivering wonderful features. That being said, it’s slowly feeling like TrueNAS/ix is moving further and further away from the open source landscape that they built their roots on, and is consistently not listening to the community, even when they blatantly tell you they want something. A lot of decisions lately feel like “this is the way it is, deal with it”, rather than listening and considering the things that their community user base actually wants.

I understand it is a tough time economically right now, and if the business becomes unsustainable, there would be no TrueNAS for us to enjoy at all. That being said, it’s a shame to see this project become more and more of a “big tech” software feel where they don’t listen to their users and just do what they want, rather than the open source roots we all know and love. (For the record, I know you all DO still listen to users in many ways, but it does feel like this is becoming less and less of the case as time goes on.)

I do not currently plan on leaving, but I can imagine a lot of people are currently considering it, and I hope ix considers that it may be time to further listen to the community they have built themselves on from the start.

4 Likes
48

That’s probably too harsh to put it and it is their money and goodwill, after all, we are riding for free, but the fact is, as a newcomer, I was met with things usually not seen in a truly open-source spirited environment. For example

  • cloud backups limited to a single provider they obviously get money from, although the technology inside (restic) is more that happy to support a whole lot of them, with a nonsensical explanation
  • refusal to document the script API for ACME DNS authenticator, also with a nonsensical explanation. I’d be more than happy to contribute support for the one that I am using, but with the approach seen, I have better things to do, especially if I have to convince someone that it is enterprise relevant first, and now have to fight the build environment as well and lose the possibility to test end-to-end
  • tightening the screws in the API, breaking stuff people were doing with legitimate expectations (I cannot add my user to a built-in group such as docker anymore), taking stuff away (scheduling of the SMART tests), …
  • putting features behind TrueNAS Connect just because - I can understand not having a file manager in the GUI because of the development effort, but I don’t understand planning to offer one, but only with the dependence on some cloud
  • now this closed-sourcing the build environment

Some of the discrepancies in the expectations obviously come from the fact that most of guys disputing those don’t in fact want an appliance, and would like to see a free-to-tinker box instead. This is a different paradigm, I get it. The problem here is, that folks have the feeling that the line between the open and closed is shifting away from them one release at the time.

I am a techie enough and my homelab is non-critical enough to always have the option to set up a Debian box with ZFS, write a few ansible scripts and off we go, so no big deal. Without that option it is probably hard to commit to running TrueNAS long-term (although if it is doing its job today, it will be doing it in a year as well… modulo security patches that will stop arriving at some point).

Just my 2 cents

9 Likes
49

Thanks for your write-up. I get what you are saying and am happy for you (and all of us) that your work is now properly paid and supported. This doesn’t change the feeling of many in the community who have been loyal since the FreeNAS days that the name change was more than cosmetic.

Promising features and then changing the scope and telling the community that one was just being cheeky is not a good look. Closing tickets without even seeming to read them if they are not containing certain trigger words is not a good look. Offering a Feature Request section in the Community Forum(!) and then rejecting most request with “no business use” is not a good look. Changing rules regarding build scripts or other parts of the software with no information upfront is not a good look. Does iX have any obligation to the community? Not really. But if they act like that they shouldn’t be surprised if the community is disappointed and not feeling heard or valued.

I understand the frustration in many community members when features are (accidentally) broken and then left that way because it would be “(…) too much development effort (…)” to fix it and business doesn’t need it. Just as an example. (The question always with such things is, does business not need it or do your current customers not need it and potential ones are not considering you because you don’t have it.)

Well, why then insist on only selling appliances and not the software itself to business customers? There are a lot of businesses right now that are considering repurposing older HW for a different use-case to sit the situation out, if possible. A backup NAS? Can be on an old server. Those are customers who are now forced to use CE since their use case is not supported. Those might be willing to pay for the enterprise features but have no way of doing so without buying HW they don’t need or can defend right now. This, in my humble opinion, is money left on the table. And might get worse if prices continue to rise.

See it this way: Customers small enough to not buy HW right now will also not buy iX devices right now. Customers big enough that they get the HW much cheaper than iX can might consider a different solution if iX is getting too expensive.

Vates with the Xen Project and XCPng seems to mange Open Source and business quite well and they are only doing software.

Also, there is a huge push from within the EU to get more independent software-wise. This means - for now - that Open Source is fine, even from outside the EU but closed source or single seller, especially from the US, is an issue. That is, again, something that might come back to bite iX in the future.

And those are only the business relevant thoughts right now and not all of them are linked to the original post but since the discussion seemed to go in that direction, I thought I’d share them.

That all being said, I’m still grateful to have such a powerful software available to me for free and I hope that iX knows that most criticism from the community comes because we care about TrueNAS, sometimes feel like part of its history and want it to be better. Accepting feedback isn’t easy, especially if one feels attacked by it.

3 Likes
50

If you are going to say something like this, please cite actual examples of the claims (with links!), so that we can better understand what you are thinking of.

To your general point about offering a feature request section yet not accepting all requests, I don’t know what to tell you other than that your expectations sound unreasonable - they run a business, every choice they make is with the survival of their business in mind, or it should be at least.

3 Likes
51

I think its worth taking a moment to read the guidelines I posted recently on how we are evaluating feature requests these days. I’ve been saying this in various places now for quite some time, but this write-up helps explain our stance pretty clearly. We love feature requests. If money, time and developers were unlimited there’s a million things I’d love to also add to the product, even just for my own selfish reasons. But since thats not reality, we have to be pretty pragmatic with what we pick and chose to go implement. And what the opportunity cost is if we pick X instead of Y to do.

In response to the comments about us being restrictive on what settings or features we expose.. I’d also like to make the point (again) about TrueNAS being an appliance. Not a Linux distro. Yes, we sometimes have restrictions on features, configuration settings, API calls, etc. Also this isn’t some generic Linux install where you can go tweak systemd or change conf files in /etc at will. We have to test almost every single combination of knobs (of which there a LOT) and guarantee that on each upgrade your system comes back healthy and things don’t break. There’s always going to be some tension between the super experienced folks who want to get under the hood and twist all the knobs on their own and those who want a product “that just works”. Because we’re an appliance, we skew towards the “it just works” end of the spectrum as a default. We expose the more advanced things only when and where it makes the complete sense to do so. Where we aren’t going to be pointing a loaded gun at somebodies foot in the process. We did that far too often in the past, usually to make the small, but vocal, uber technical minority happy. Often at the expense of too many others who then inadvertently broke things because they didn’t have that technical depth to know how to do so safely.

1 Like
52

I hadn’t thought of this option, but honestly it makes more sense if providing any kind of support to home-lab and other DIY’ers is too far out of reach. Making an Enterprise-lite edition available opens the door enough to satisfy some higher-end appetites, particularly those whose job it is to evaluate solutions for their clients or employers.

Quick story:

At a previous job, we had Netapp shelves for our VMware cluster, but, after several rounds of testing, we settled on a fully-redundant iXSystems TrueNAS box for our TV CDN store (this was 2015). It was the only solution we had tested that could simultaneously handle 1) recording 30 minutes of live TV for 180+ channels (roughly 3Gbps non-stop), 2) recording individual user’s movie and TV show requests (network DVR), and 3) playing back live TV or recordings for hundreds of users, particularly at peak time. It was the perfect use case for ZFS with a high spindle count, a flash ZIL/SLOG, and oodles of ARC.

In this case, it was the positive experience with a good piece of kit that got me using FreeNAS/TrueNAS for my own subsequent projects and recommending it to others.

2 Likes
53

And there you go:

54

Would be nicer if I had some straight-forward pat-answer formula to spit out for this one, but reality is a bit more complex than that :slight_smile:

Oh I understand completely and this is actually exactly what I was hoping for — it’s just when say something general like “features” it leaves space for people to assume the worst, but the ZFS examples really help to clarify!

1 Like
55

Coming late to the battle, but I have some honest question on the feature aspect. iX has invested in HexOS, and that one is much more dedicated towards home consumers.

This would need Truenas to have some dedicated consumer features to be added. I think about ZFS extend, and ZFS hard drive mix and match, but you could also have Disk Spin down for home users not using the server as much, or a more basic tiering of drives (caching on SSD for example).
Is this also part of the decision which is going to be made for adding features for the CE ?

This is again, a very honest question. I see a lot of aspects discussed about the pro users, and this is fine to me, devs need to eat. But Truenas is also used for HexOS. And I would definitely understand (expect, given iX’s implications ?) that this would also be considered.

56

I seem to have caused some confusion as to what I meant. Sorry, if I left too much room for interpretation or it seemed hostile. Quite the opposite was my intention. Maybe I can clear it up a bit.

What exactly do you mean? Or do you mean the whole paragraph? Some examples have been given in this thread already but if you tell me what you want me to elaborate on, I gladly will.

That wasn’t the point. See below.

Thanks, I read those when they were posted before and think it explains quite well what you expect and that is fair. The point I was trying to make was not that every feature has to be accepted - that is utterly unrealistic even with best intentions and money to throw around - but how it looks and sounds.

This might even be a good example: Such a post or guideline before establishing a Feature Request section would have tempered expectations and made it more clear for everyone. From what I read, many disappointed or “What even do we have this section for if…” posts were from users who had different expectations and were irritated by the dissonance of Community (Edition) Feature Request section and the business answer. I don’t argue with iXs argument itself there, only with how (clearly) it was communicated beforehand.

Most of the issues are about “proactive” communication or communication and wording as a whole. Communicating changes after the fact or not really at all sometimes is what many here seems to have an issue with, not necessarily the changes themselves.

Again, that’s only my observation. Maybe I’m completely wrong about all of that. I can accept that. Also, my last point stands:

1 Like
57

This will most likely not be what I talked about, especially since it seems to be closed source, reliant on iXs cloud, collecting quite a lot of data (according to the privacy policy), and includes forced arbitration, even if this would not be applicable to me due to my location. But maybe others will feel differently about that.

58

The key takeaway about the request guidelines and the “Enterprise Focus” isn’t “no consumer features will ever be added again” but “we want to be sure it won’t negatively impact Enterprise customers.”

For example, disk spindown is definitely not something that the Enterprise group is concerned with. The challenge would be “how do we implement it in a way that the CE users can deploy” but also without causing a CE to Enterprise user to see it as “removing a feature” if it’s not available there.

2 Likes
59

Seems fair. I see your point, and I understand the limit of having your reputation preventing you to add an option with an Astérix saying “pro user, use it at your onward risk”.

Thanks

60

This I don’t really get. I am 100% with you on no[1] enterprise customer wanting spin-down. But keeping with that example: Why would they miss it then and see it as removing a feature when going from CE to Enterprise? This is an honest question out of curiosity. And sorry for veering a bit more off-topic.

  1. Most likely almost no-one. But that’s not the point ↩︎

1 Like
61

Perception challenges. People see “Enterprise” as purely a superset of a “lower tier” rather than a focused release that might not have some of the newer features that could still be rough around the edges, or open up potential edge cases that might be acceptable for a home user but not in the Enterprise.

We touch a little on it in the blog linked in post #53 about a “bridge” version between CE and Enterprise.