I also wrote “as far I am concerned”.
My test SCALE server will be decommissioned this week-end. I’m out of SCALE/CE. The next experiment will be zVault, and my storage remains on CORE 13.0-U6.7 indefinitely. (I do not use Windows, so need not upgrade to U6.8.)
How is it going worse than that?
6 Likes
Maybe the elephant in the room, I don’t know… and it may be rude to mention a competitor here but I will anyway. I agree totally with the above and the main problem seems to be that someone else has already done it… What Kris describes in the snippet above (templates and click-and-go instances of various pre-cooked environments) has already been done in Proxmox, which is also open-source and Debian-based and in addition does clustering, HA and live migration, remote management of VMs and containers, both ZFS and Ceph integration and more. And has been working on that since 2005 and is very robust. It would seem unwise for TrueNAS to try to do the same thing. So where is the product going? I agree this is probably the most fundamental question here.
2 Likes
iX: You’re killing me here.
Is there a migration paths for Incus VMs or not? Yes/No answers only.
Enterprise customers, at least those I interacted with in the past, have requirements that are very different from the virtualization TrueNAS provides. In fact, not even XCP-ng, Proxmox, etc. are alternatives (at least short-term). There is a reason why organizations are keeping VMware despite of what Broadcom is doing.
VMware has features that are not available on other product. And even if all of a sudden e.g. Nutanix pulled this off, the cost and risk of switching is enormous. If you are an organization with 200k+ staff and billions of revenue per month, saving 10 million dollars on virtualization license costs is not a valid business case.
One has to realize that switching virtualization would be a multi-year project and cost way more than just 10 million. And that assumes nothing goes wrong.
Hyper-converged is nice for edge deployments or the SMB market. But IMHO it is not mainstream. So there is not much of a market anyway.
Last, but certainly not least, trust is critical for enterprise customers. Depending on their size relative to the vendor they expect (and get it) to be involved in roadmap development. If I am about to spend 50+ million over the next 5-10 years, I want to be certain that there are no surpises. Because in that case my head is on the chopping block and I can kiss my career goodbye.
I never understood why iXsystems went the Scale route. In terms of paying customers they should have stayed on FreeBSD.
6 Likes
Regarding trust, I have two interesting links to submit from another well-known forum:
Exhibit A
Proxmox VE
Great hypervisor, solid GUI, ZFS support, well-maintained.
But: Docker needs to run in a VM, which adds layers. The interface has small, hard-to-find details.
Feels like a patchwork of good tools rather than a cohesive platform — and becomes a bit of a sprawl over time.TrueNAS SCALE
Love the ZFS base, and the Linux pivot is promising.
But: Still young on VM/container management. App system is clunky.
Performance tuning often feels like chasing ghosts. Some of the same sprawl issues as Proxmox.
(I highly recommend reading the thread in full for feedback from other users and explanation of the “sprawl” issue. OP likely set the bar too high by wanting an all-in-one which does everything perfectly… but this is what TrueNAS apparently tries to achieve, and the OP’s wish list for storage reads exactly on TrueNAS. And then everything falls apart when we come to apps and virtualisation.)
Exhibit B
answer
TrueNAS is OK, and I have been using it for years now. […] I like it well enough for its storage server capabilities, but I don’t use it to run docker containers or VMs. It is really BAD at containers and VMs. Its also really weird with file permissions. Also you need to know going in, all of the newest versions are considered to be in beta. If you download the newest versions you are agreeing to be a tester. You need to be running off of something like 3 or 4 releases old to be on something stable enough for production. Software Status - TrueNAS Roadmap - Open Source NAS Software
The double “debacle” from k3s/docker and now Incus is something TrueNAS will have a hard time recovering from.
12 Likes
Don’t forget in the k3s debacle, they pushed TrueCharts too and ended up stonewalling them too. It became a nightmare. I moved from those version of apps to jailmaker because of those debacles. It has been stable ever since for me, including my own builds of containers in Incus.
5 Likes
The alleged reason was that they had to spend considerable resources adopting ZFS to still run on FreeBSD despite most of the ZFS development happening in Linuxland, IIRC?
If true, that seems like a valid reason - ie there are only so many resources a business can deploy and FreeBSD doesn’t enjoy the same level of ongoing OS development as Debian Linux, etc. does. This becomes pretty relevant once proper drivers for new network cards are no longer supported in FreeBSD, etc. (See Aquantia on SCALE vs. Core, IIRC)
So I can see the reason for the OS shift. But promised features (i.e. resuming failed replication attempts from the point of failure, not starting over) do not seem to have been implemented either. So I can see why people are frustrated with the Apps and VM changes that result in a lot of extra new work just because an OS revision was done.
OpenZFS is fully supported on FreeBSD now.
3 Likes
If recent history is anything to go by, this thread will be shortly closed and locked by iX as off-topic.
But I sincerely hope that, whether they do so or not (this being their forums, after all), that someone there does pay real heed to the strength of community concerns raised here.
To be clear - not necessary the specific choice of container/VM solution, but the impact of and loss of trust brought about by the lurching, sudden changes in direction of the product. It seems unavoidable that this has been intensified by the (self-imposed?) six-monthly release schedule.
I just hope someone at the company is taking this on board.
6 Likes
Possibly not, because this is not an “Announcement” thread initiated by iX and then discussing the release; this is users chatting between themselves. But, for this reason, it is quite possibly not read and/or not monitored by iX staff.
I really do not get why they bother to provide VMs, jails, containers, … and based on that “plugins” or “apps” at all.
The platform would be perfectly fine with just storage features.
But if you offer “plugins” you have to own that feature and that includes effin’ life cycle management, i.e. regular updates among other things.
Pushing a one click Nextcloud install out the door and than not caring about how users are ever going to update that is downright evil because Nextcloud is intended to be connected to the public Internet.
Just to name one popular application.
Or make sure the jail subsystem is robust, keep your base FreeBSD version up to date with upstream and let users figure out how to deploy applications in a jail - another possible and sane approach.
sigh
Patrick
8 Likes
But ZFS integration in Proxmox is poor. Generally all kind of storage management. There is no UI way to partition disks, create zpools to your liking, allocate swap space, etc.
Then there is no snapshot and replication in the UI.
Etc.
So there is enough room for TrueNAS USPs.
I would be perfectly happy with exactly the latest CORE feature set, fewer bugs (VNC for VMs, cough) and a FreeBSD 14.3 base. I am not missing anything feature-wise in CORE. Only that it’s not maintained, anymore.
EDIT:
Even the CORE UI is in some areas way superior to Proxmox:
LLDP?
- TN CORE: switch on the service, done.
- Proxmox: install from CLI and edit config files or configure systemd.
SNMP?
- TN CORE: switch on the service, done.
- Proxmox: install from CLI and edit config files or configure systemd.
SNMP and LLDP are enterprise features - just saying 
5 Likes
Imagine a product that uses and updates upstream FreeBSD and OpenZFS as its base, with a NAS middleware and set of configurations that run on top of the operation system.
How awesome would that be? 
3 Likes
I actually use Truenas precisely because they offer storage and VM/docker in single package.
I don’t want to use one server for VMs and second as NAS. I want all in one solution because for homelab it’s simpler and cheaper.
8 Likes
Agree - good thread to read. But you will always have the “sprawl” issue.
The only remedy for this, that I know is: infrastructure as code.
Ansible, Puppet, Chef, … whatever you prefer. Maintain via git, e.g. with self hosted Gitea. Done.
Not quite entirely, IMHO. Yes, the FreeBSD and Linux ZFS code bases diverged. But while there was admittedly more participation and development in Linuxland, you must take into consideration that some of the most experienced and knowledgable developers for ZFS are still with a variant of Solaris or with FreeBSD.
And the problem was solved upstream! OpenZFS is a single repo with CI in place and automatic checks for every commit, and they must pass on both FreeBSD and Linux, so for once developers from both camps teamed up and solved the problem.
No extra workload for iX whatsoever. FreeBSD and Linux are both first class citizens as far as OpenZFS is concerned.
Solaris is still struggling to jump on the bandwaggon, unfortunately, but I’m positive they will get there.
Heck, even my Mac runs a current version of OpenZFS.
There is no - absolutely no - need to manually backport anything from Linux to FreeBSD as far as ZFS is concerned. If you stay on a current and supported release of FreeBSD you get it all for free.
6 Likes
Oh… don’t be so dramatic! There are plenty of (top) managers who messed up the things and are ok with their careers. Sometimes to the point when they are messing up the things again (at the new company).
That doesn’t necessarily work – many modern work flows requiring storage break down when storage is disaggregated from the application – it’s just too slow/has too much latency, e.g.:
– from the ElastiFlow Slack channel discussing the unsuitability of Ceph storage backends
Having an app layer on top of your storage platform – especially a performant, and manageable/automatable form like Docker or LXC – is important. Even VMs, which have to go through the network stack to access storage, are too slow.
I don’t think TrueNAS could replace enterprise vSphere – not short of some serious development of software defined networking functions/distributed virtual switches/etc – but I’m thinking (and prototyping) some specific applications stacks (like ElastiFlow above) in either Docker Compose or LXC that could make a great pair with TrueNAS. Plus, with the fremium angle, clients can prove the solution on a free platform, and if happy enough, they can look to an Enterprise TrueNAS solution with support.
Fangtooth was delivered. Incus in it is awesome, so far. The only problem I saw was some people bitching they couldn’t migrate a few VMs, or gripes that there was a new hidden store for data. What exactly was the big problem, aside from negative gripes can be utterly contagious online?
Suffering from aggravation? Take a deep breath, maybe make yourself a cup of tea, and try to relax – making things work in different ways from time to time is a constant of system administration. If it’s a big problem, you may have accumulated significant technical debt because you haven’t worked on disaster recovery, infrastructure as code, automation, etc., in a long time. Now, is a great time to finally tackle the things you’ve been neglecting — or not, and just stay on the old platform.
I’d sooner say that if there was any mistake, it was not simply saying, “this is the new platform that’s taking us to the future. The old platform is over. Stay on the old version for as long as you need to run your old systems. Try asking in the forums for help migrating/rebuilding systems for the new platform – obviously, we can’t support ironing out the kinks in everyone VM deployments.”
At any rate, I’m skeptical most of the complaints are valid anyway. How many VMs did the migration “force” you to rebuild? How many VMs do you typically run simultaneously/continuously on your TrueNAS box(es)? How many CPU cores and how much RAM do your TrueNAS systems have to power all those VMs? Why do you use (the threadbare) virtualization functionality of TrueNAS so extensively as a hypervisor, instead of using one of the more powerful, fully featured dedicated (even free) hypervisors on the market?
These seemed like the more reasonable takes on the old VM system:
Kris Moore:
…of course now we have folks in the community using VMs for many years,
even though it’s never been an enterprise thing, for us – it’s always been
something we’ve treated as experimental internally, just never called it that…
– https://www.youtube.com/watch?v=SIs0e9NepbM&t=1m44s
I have trouble seeing this lament over the retirement of the old system as much more than a snowball effect of complaints, where some people get a little too wound up on the thrill of complaining and hurling blame around. The future is much more interesting.
2 Likes
Proxmox is great for containers and virtualization, not so much so for Docker, pretty bad for ZFS management (really, all it can do in the GUI is create a pool), and has zero NAS features.
When SCALE was announced, I foolishly thought it had potential to be a Proxmox killer. Let’s just say I no longer hold that view.
Because FreeBSD is a dead-end OS.
Because all their competitors provide plugins/apps, so they feel the need to be seen as providing them too. What they clearly don’t see the need to do is to actually resource that requirement.
3 Likes
Nope.
1 Like