Updated to 24.04.2 now VEEAM backups don't work

TrueNAS General
,
1

I have been using VEEAM Backup & Replication with no issues for the past couple months running on TrueNAS 23.10.1.1. Yesterday I updated to TrueNAS 24.04.2 and now my NFS config backups from VEEAM started to fail. Were there any changes between TrueNAS 23->24 that would cause this issue? The strange issue is that only the configuration backups fails. My scheduled backup of VMs completed successfully. Some things I’ve tried

  • Rebooting both Truenas/Veeam server
  • Creating new NFS mount of empty folder
  • Using NFSv3(works)
  • Reading nfsd logs with rpcdebug -m nfsd -s all
  • Creating a fresh TrueNAS install with a new drive
  • Querying the NFS share with nfs-ls(I am able to browse and add/delete files as well as mount on other systems)

Below is a log output of the Veeam configuration backup error. There are no errors on the TrueNAS side as far as I can see(systemd, systemctl).

[13.07.2024 11:43:15.658]    <17>   Error (3)    [RPC] Failed to get a file attribute from the /mnt/FILE-SERV/Backups/VeeamConfigBackup/VEEAM/VEEAM_2024-07-13_11-43-01.bco
[13.07.2024 11:43:15.658]    <17>   Error (3)    [RPC] --tr:Error code: 0x00000000
[13.07.2024 11:43:15.658]    <17>   Error (3)    [RPC] NFS4.1 error.
[13.07.2024 11:43:15.658]    <17>   Error (3)    [RPC] --tr:Failed to get search 20. Session: 0
[13.07.2024 11:43:15.658]    <17>   Error (3)    [RPC] --tr:Failed to complete RPC request. Method: [NfsFindNext], In XML: [<InputArguments><Link value="nfs41://10.10.10.10:/mnt/FILE-SERV/Backups/VeeamConfigBackup/VEEAM" /><Count value="128" /><Handle value="20" /><AclMode value="0" /></InputArguments>]
[13.07.2024 11:43:15.658]    <17>   Error (3)    [RPC] --tr:Failed to invoke RPC request
[13.07.2024 11:43:15.658]    <17>   Error (3)    [RPC] --tr:Failed to call DoRpc. CmdName: [DoSerialRpc] inParam: [<InputArguments/>].
[13.07.2024 11:43:15.658]    <17>   Error (3)    [RPC] --tr:Failed to call DoRpcWithBinary. Method: [DoSerialRpc], In xml: [<InputArguments/>]
[13.07.2024 11:43:15.658]    <17>   Error (3)    [RPC] --tr:Failed to call DoRpcWithBinary
[13.07.2024 11:43:15.658]    <17>   Error (3)    [RPC] --tr:Failed to do RPC call with serialized data, Strategy type: DefaultStrategy
[13.07.2024 11:43:15.658]    <17>   Error (3)    [RPC] --tr:Failed to call [DoSerialRpc], session id: {985ef0a1-02ca-44cd-89ca-b302d55662f2}
[13.07.2024 11:43:15.658]    <17>   Error (3)    [RPC] --tr:Failed to call [NfsFindNext]
[13.07.2024 11:43:15.658]    <01>   Error (3)    Retrieved incorrect string: do rpc - The operation completed successfully.
[13.07.2024 11:43:15.658]    <01>   Error (3)    Failed to get a file attribute from the /mnt/FILE-SERV/Backups/VeeamConfigBackup/VEEAM/VEEAM_2024-07-13_11-43-01.bco
[13.07.2024 11:43:15.658]    <01>   Error (3)    --tr:Error code: 0x00000000
[13.07.2024 11:43:15.658]    <01>   Error (3)    NFS4.1 error.
[13.07.2024 11:43:15.658]    <01>   Error (3)    --tr:Failed to get search 20. Session: 0
[13.07.2024 11:43:15.658]    <01>   Error (3)    --tr:Failed to complete RPC request. Method: [NfsFindNext], In XML: [<InputArguments><Link value="nfs41://10.10.10.10:/mnt/FILE-SERV/Backups/VeeamConfigBackup/VEEAM" /><Count value="128" /><Handle value="20" /><AclMode value="0" /></InputArguments>]
[13.07.2024 11:43:15.658]    <01>   Error (3)    --tr:Failed to invoke RPC request
[13.07.2024 11:43:15.658]    <01>   Error (3)    --tr:Failed to call DoRpc. CmdName: [DoSerialRpc] inParam: [<InputArguments/>].
[13.07.2024 11:43:15.658]    <01>   Error (3)    --tr:Failed to call DoRpcWithBinary. Method: [DoSerialRpc], In xml: [<InputArguments/>]
[13.07.2024 11:43:15.658]    <01>   Error (3)    --tr:Failed to call DoRpcWithBinary
[13.07.2024 11:43:15.658]    <01>   Error (3)    --tr:Failed to do RPC call with serialized data, Strategy type: DefaultStrategy
[13.07.2024 11:43:15.658]    <01>   Error (3)    --tr:Failed to call [DoSerialRpc], session id: {985ef0a1-02ca-44cd-89ca-b302d55662f2}
[13.07.2024 11:43:15.658]    <01>   Error (3)    --tr:Failed to call [NfsFindNext]
[13.07.2024 11:43:15.658]    <01>   Error (3)    [CProxyRpcInvoker] Failed to call RPC function 'NfsFindNext': The operation completed successfully. Failed to get a file attribute from the /mnt/FILE-SERV/Backups/VeeamConfigBackup/VEEAM/VEEAM_2024-07-13_11-43-01.bco. NFS4.1 error.. InParams: '<InputArguments><Link value="nfs41://10.10.10.10:/mnt/FILE-SERV/Backups/VeeamConfigBackup/VEEAM" /><Count value="128" /><Handle value="20" /><AclMode value="0" /></InputArguments>'.
[13.07.2024 11:43:15.658]    <01>   Error (3)    The operation completed successfully.
[13.07.2024 11:43:15.658]    <01>   Error (3)    Failed to get a file attribute from the /mnt/FILE-SERV/Backups/VeeamConfigBackup/VEEAM/VEEAM_2024-07-13_11-43-01.bco
[13.07.2024 11:43:15.658]    <01>   Error (3)    NFS4.1 error.
[13.07.2024 11:43:15.658]    <01>   Error (3)     (System.ApplicationException)
[13.07.2024 11:43:15.658]    <01>   Error (3)       at Veeam.Backup.ProxyProvider.CNativeAccessor.GenerateExceptionWithoutTrace(Exception ex)
[13.07.2024 11:43:15.658]    <01>   Error (3)       at Veeam.Backup.ProxyProvider.CNativeAccessor.CheckString(String value, String message)
[13.07.2024 11:43:15.658]    <01>   Error (3)       at Veeam.Backup.ProxyProvider.CProxyRpcInvoker.Call(String methodName, CProxyInvokeInputArg inputArgs, Boolean secureCall)
2

So you think its an NFSv4.1 change?
Linux kernel provides that. We might have to dig around there.

Is there any documentation of how you set-up your NFS 4.1 share on the Veeam side.

3

I was able to resolve the issue by appending the following to /etc/sysctl.conf. It seems like the VEEAM client does not know how to handle file delegations, or there is a bug with how VEEAM handles it.

fs.leases-enable=0

Is there any way to set this option from the GUI, or will I need to edit this file if it gets overwritten? Also will this affect anything else running in TrueNAS?

1 Like
4

From GUI try System, Advanced, Sysctl section.

5

Well done!
It seems like there is a change from Cobia.
If you want to report it as a bug, we can review why and see how best to handle. Obviously, we’d prefer that Veeam works well by default

1 Like