Jails in my setup had their own IP’s, they were like little virtual machines, and they’re just now getting around to replicating this in Scale. I’ll probably migrate in June once all these dockerized apps support using an IP alias which is assigned to one of the host interfaces. Some apps already do (mariadb yet another I have discovered recently) but most don’t, and there’s a finish line date so I’ve planned my move to scale after that.
1 Like
Also, props to the devs for making this happen. Us Core folks are a rare breed and it’s nice to still get a little bugfix update like this.
1 Like
Ahhh yes, sorry when I said Scale, I actually meant CE, CE is just scale rebranded isn’t it ?
By the time I get around to moving over to the latest and greatest they may have released 25.10…
So you mean take my install and edit /usr/local/lib/python3.9/site-packages/middlewared/plugins/pool.py in this case, replacing this one line?
Just want to make sure I understand how I apply this myself here.
1 Like
If that’s all it takes, I’ll do it on my own system as well.
+1
2 Likes
That’s exactly what I did a while ago, and then restart either nginx and/or some other service that I’m not quite remembering right now (middlewared, perhaps?), or reboot, for the patch to take effect.
I got the feeling long ago that iX was going to abandon CORE earlier than expected and simply not ever release 13.3-U2, and I didn’t want my boot drive getting unnecessarily worn down by daily scrubs, so I didn’t wait for them to correct the problem. Patch has been working fine ever since!
1 Like
Obviously a massive change that would have required too much resources to be validated and included in the final update… 
4 Likes
Much too risky for all the enterprise customers running Core 13.3. 
1 Like
Then that means it survives reboots? 
Did the patch also fix the scrub schedules bug as well?
Absolutely survives reboots, because the patch is applied directly to the executable Python code, and that doesn’t change or “reset” in any way across reboots. And if you don’t restart the services, or reboot the box instead, the patched code will not be picked up and you’ll still experience the problem.
In my case, I applied the patch only to fix the scrub schedules bug, that and nothing else, and it worked well for that purpose. But it sounds like you might also be referring to some other problem as well, so I won’t speak to that.
2 Likes
There are two bugs, but likely related to the same line of code.
- The boot-pool scrubs every day, no matter what you set it to.
- The scheduled scrubs (Scrub Tasks) initiate on every trigger, ignoring the threshold days. Instead of monthly scrubs for the storage pools, you end up with weekly scrubs.
I’m going to apply this fix now and see what happens by tomorrow.
Please tell me I’m wrong. I hope I’m wrong. Do I have this right?
- TrueNAS SCALE and Core used to initiate scrubs in the standard method, but then later switched to using
py-libzfs.
Here is the proof of that.
2024-07-07.03:45:00 zpool scrub boot-pool
2024-07-13.03:45:01 zpool scrub boot-pool
2024-07-20.03:45:01 zpool scrub boot-pool
2024-07-27.03:45:02 zpool scrub boot-pool
2024-08-03.03:45:00 zpool scrub boot-pool
2024-08-09.03:45:00 zpool scrub boot-pool
< --- after this, the scrubs happen daily --- >
2024-08-15.03:45:02 py-libzfs: zpool scrub boot-pool
2024-08-16.03:45:00 py-libzfs: zpool scrub boot-pool
2024-08-17.03:45:00 py-libzfs: zpool scrub boot-pool
2024-08-18.03:45:01 py-libzfs: zpool scrub boot-pool
2024-08-19.03:45:00 py-libzfs: zpool scrub boot-pool
2024-08-20.03:45:00 py-libzfs: zpool scrub boot-pool
< --- it continues forever like this --->
You can see that after I upgraded from Core 13.0 → 13.3, zpool history | grep scrub shows my scrubs are being initiated via py-libzfs. You’ll also notice that my boot-pool went from weekly scrubs to daily scrubs. (This is a bug.)
What also happened is that my storage pool scrub tasks were ignoring the threshold days. (Also a bug.)
-
To complement this switch to py-libzfs, there was a single-line change in code back in May 2023. Now when checking for past scrubs,
py-libzfsinitiated scrubs are also checked. (Makes sense.) -
Somehow, while both changes were properly implemented in SCALE, only half was included for Core 13.3. Core 13.3 now uses
py-libzfsto initiate scrubs, yet it does not check for py-libzfs initiated scrubs for the boot-pool or any scrub tasks. -
This obviously manifests as a bug, which users noticed and reported back in September 2024.
-
The fix was simple. All that needs to be done is to backport the code that had already been written. (The single-line of code to properly check for
py-libzfsinitiated scrubs.) This was committed in September 2024, and targeted for Core 13.3-U2. -
Almost nine months later, Core 13.3-U1.2 is released, which does not include this simple fix. (A bug that was known for nine months. A bug that was easily fixed with backported code.)
-
Today, we’re being told that this simple (backported) fix could not be included because of extensive testing costs and required resources?
I’m sorry… Can someone tell me what I got wrong? I want to be wrong. This doesn’t sit well with me…
1 Like
You’re missing that iX just don’t care. They’re done with CORE, other than to file copyright notices against zVault (because apparently TrueNAS isn’t as open-source as they’ve claimed).
1 Like
“Don’t care” is too strong I’d argue. As discussed previously, when you have a train of software with less than a quarter of the users of an obsolete version from 5 years ago (11.3) its hard to justify spending ANY time on it at all. We did the minimum required to not leave it hanging in that potential data-corrupting state. We opted not to go start grabbing other patches, because once you open that can of worms things get to spiral out of control quickly. We then looked at the slate for U2 and decided it wasn’t worth another release & test cycle, especially as we are unifying things on CE going forward.
As for the open-source dig, everything in the Free/Community Editions of the software is open, nothing has changed there. The enterprise specific bits (which you don’t see without our enterprise hardware / license) are “source available”. So zVault can easily remove (which they did) without any real impact to functionality folks on the free side sees. What they do beyond that is on them 
You’re welcome to complain about that, but that’s how FreeNAS/TrueNAS have been for a decade plus now, nothing new here.
1 Like
For applicable values of “easily”. While the core modules with an iX copyright (rightfully so, not debating!) are only HA and enclosure management, the functions in these modules are referenced in >50 source files.
So if you are wondering what the zVault team is up to and when there will be a new binary/installer release - it’s going to take a while.
I for one will encourage my “end user/customer” friends to move to SCALE/CE, I am running a CE system with productive apps, but the core file sharing and jail services will remain on 13.3 and switch to zVault eventually. In case that project fails it will be FreeBSD and Ansible and no UI.
I intend to support, hack, evangelize, … FreeBSD to my retirement and beyond.
Take care,
Patrick
1 Like
Of course you would, but actions speak louder than words. And it’s telling that you’re blaming users for a situation you yourselves created–you did everything you could to discourage users from upgrading(?) to 13.3, so it’s no surprise that few did. But that’s now your justification for refusing to fix bugs in that code, even when you’ve had the fix committed for almost nine months.
IB4TL
2 Likes
As is your right and freedom to do so, in the best traditions of open source
I think competition and choices are good overall and wish the BSD folks the best of luck in that endeavor of competing with Linux. I personally think that ship has sailed and the next real evolution of Linux will be something NEW, not a revival of an older legacy system personally. But my crystal ball may not show the same future as others see in theirs.
1 Like
I mean sure, what benefit to us would it be to advocate for users to stay on a train we’ve already been messaging was not proceeding into the future. The 13.3 release in many ways was chance to test and see how many “die-hard” BSD folks would rather use an effectively defunct, but more updated FreeBSD build, mainly due to jail requirements. The answer was a paltry few, which shocked me as well. I thought I was low-balling the guess and it was still 5x smaller than what I thought at a bare minimum. Hindsight being 20/20, might be that the 13.3 train was unnecessary, but still glad we did it, since we were able to make that sunset decision with data and not just our assumption.
But I feel like we’re all just rehashing the same old argument now all over again. (Does this place have a bad echo?). Anyway, we’ve diverged a bit from the main announcement topic here. If folks want to repeat this in another thread feel free!
1 Like
Seems like some frustration goes from unresolved bugs (which actually already have a fix in the codebase).
Over a decade ago AMD dropped support for one line of their GPU. Just before the Windows 8.1 release. Perhaps for the same reason as the testing cost you’ve described above. So this GPU line remained without Win8.1 support.
However, there was a beta driver with Win8.1 support deep, deep in their site. Long story short – I’ve partly successfully used this very driver even with Win10. And tried to avoid AMD ever since.
The `great` finale of the AMD GPU story
After some of the major updates, windows had been switching this “unnoficial” driver to the Microsoft basic driver. This brings temperatures very high, but after manual update of the driver back to beta, all becomes clear for a while (minor updates went without a hitch).
During one of those major updates I was afk. And without swift switching to the beta driver, the 7-year-old laptop eventually toasted itself. Good job, AMD 
!
This laptop could even be my first truenas!
I think you should consider making a download link for the 13.3-U2-alpha somewhere deep, deep in your site. Because encouraging your users to fix the bug themselves looks very similar to mockery. Even if it is really not.
P.s. I’ve never used a CORE version.
1 Like