Cannot sync application catalogues since upgrade to 24.04.1

Really need some help here. I don’t know where to look next. I am reposting here what I created as a bug report a week ago. Here is a link if anyone has access: Log in with Atlassian account

Any more info requred please let me know. My network topology is two PFSense firewalls in High Availability mode. The LAN switch is a Cisco 3750 with 10gb fibre to both firewalls Both servers have two physical network connections to the switch - those ports are configured to VLANs 10 (network VLAN) and 25 (Server VLAN) and share them to internall apps and VMs via bridges. They also share a dedicated network card for inter-server communication

FIRST

Both servers cannot seem to sync the Truenas or Truecharts application catalogues anymore. I have checked the system time and it looks fine. BIOS time is set to UTC time and the server Local time is reporting correctly. The system also automatically resets the BIOS time to UTC time upon every reboot even if I change it manually to something else.

SECOND

Windows 10 VMs on 24.04.1 are having multiple (maybe related) issues. Both VMs are running the latest stable version of the windows VirtIO drivers (0.1.240). One system formerly had a VPN installed on it (I removed it for testing) and the other has only been used for testing and has NEVER had a VPN installed on it.

  1. Download speeds from the internet struggle to hit 1Mbps. SMB fileshare transfers from VMs to SMB shares fail every time but VMs to either server seem unaffected. I am attaching some screenshots from 2 VMs showing bandwidth testing using Ookla. No other devices or PCs on site are showing any issues with internet speeds.

  2. File transfer uploads to the server SMB shares fail but downloads to the VM work fine.

  3. I was able to get one VM working after rolling back the Truenas OS to a previous version. Clones of this one working VM also work fine but any new VMs created have the same networking issue - downloads struggle to reach 1MBPS but uploads are uncapped.

As an update to this - I just rebuilt my test server with a freshly downloaded copy of 24.04.1.1 and reloaded my configuration file and the issue still remains - I keep getting errors trying to sync application catalogues and newly created VM’s still have thier download speeds throttled.

Still getting these errors:

Critical

Failed to sync TRUECHARTS catalog: [EFAULT] Failed to clone ‘GitHub - truecharts/catalog: This is JUST the catalog, please refer to truecharts/apps for the actuall app code!’ repository at ‘/mnt/BlackPool01/ix-applications/catalogs/github_com_truecharts_catalog_main’ destination: [EFAULT] Failed to clone ‘GitHub - truecharts/catalog: This is JUST the catalog, please refer to truecharts/apps for the actuall app code!’ repository at ‘/mnt/BlackPool01/ix-applications/catalogs/github_com_truecharts_catalog_main’ destination: Cloning into '/mnt/BlackPool01/ix-…

2024-06-11 13:11:42 (America/Detroit)

Are you able to sync the apps catalog from iX? or is this impacted the same as truecharts?

What version does work worth with if you rollback?

Both app catalogues on both servers cannot sync. The only network change within the last couple months I made was to add some rules to reduce “Bufferbloat” which I currently have disabled while I figure out what is going on. I also set my pi-hole servers to stop blocking traffic while the updates try and run.

As for rollbacks - I rolled back to 24.04.0 and then to 24.04-RC.1. The first rollback fixed the network issues I had with one of the 2 Windows VMs I was running but not both. If I try and create a new VM now, the network is throttled as I described.

I rolled back to the previous version and that resolved the network issue for one of the VMs. A further rollback did not improve anything

24.04.1.x uses a different kernel to 24.04.0.

The 24.04.2 nightlies have 6.6.32 (iirc) and it may be worth testing to see if your issues go away.

(Some of mine did)

Thanks for the tip - I will look into trying out the nightlies.

That kernel upgrade is exactly what I initially thought may be behind the issues when they didn’t fully go away after rolling back.

OK test system is now on TrueNAS-SCALE-24.04.2-MASTER-20240611-013909.update

Application catalogue updates. will update after testing

On the newly rebuilt server I keep getting this service failing:

● logrotate.service loaded failed failed Rotate log files

and the other server keeps getting these:

● apt-daily.service loaded failed failed Daily apt download activities
● console-setup.service loaded failed failed Set console font and keymap
● logrotate.service loaded failed failed Rotate log files
● serial-getty@ttyS0.service loaded failed failed Serial Getty on ttyS0

Is there some log files I can look into?

Test server has a failed service today:

● apt-daily-upgrade.service loaded failed failed Daily apt upgrade and clean activities