[Errno 22] Invalid argument

TrueNAS General
,
1

Dear TrueNAS community.
I have a somewhat beefy server 512GB RAM with AMD EPYC 7662 64-Core Processor. My OS version is TrueNAS-13.0-U2

The dashboard shows Services 116.5GB, ZFS Cache 14.9GB and Free 380GB. I have a few VMs totaling 76GB and in order not to worry about memory, I’ve set the vfs.zfs.arc_max to 372GB well below the suggested level. I arrived at this number by giving myself a 16GB buffer instead of 4GB on an allowance of up to 124GB in VMs if i wanted too which i never reached.

Whilst trying to create a new baby VM 4GB I’m getting below error which i read as memory overcommit.

Your kind assistance is greatly appreciated.

Thanks in advance.

Error: Traceback (most recent call last):
File “/usr/local/lib/python3.9/site-packages/middlewared/main.py”, line 139, in call_method
result = await self.middleware._call(message[‘method’], serviceobj, methodobj, params, app=self)
File “/usr/local/lib/python3.9/site-packages/middlewared/main.py”, line 1246, in _call
return await self.run_in_executor(prepared_call.executor, methodobj, *prepared_call.args)
File “/usr/local/lib/python3.9/site-packages/middlewared/main.py”, line 1151, in run_in_executor
return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
File “/usr/local/lib/python3.9/concurrent/futures/thread.py”, line 58, in run
result = self.fn(*self.args, **self.kwargs)
File “/usr/local/lib/python3.9/site-packages/middlewared/schema.py”, line 979, in nf
return f(*args, **kwargs)
File “/usr/local/lib/python3.9/site-packages/middlewared/plugins/vm.py”, line 1594, in start
self.middleware.call_sync(‘vm.init_guest_vmemory’, vm, options[‘overcommit’])
File “/usr/local/lib/python3.9/site-packages/middlewared/main.py”, line 1294, in call_sync
return self.run_coroutine(methodobj(*prepared_call.args))
File “/usr/local/lib/python3.9/site-packages/middlewared/main.py”, line 1334, in run_coroutine
return fut.result()
File “/usr/local/lib/python3.9/concurrent/futures/_base.py”, line 438, in result
return self.__get_result()
File “/usr/local/lib/python3.9/concurrent/futures/_base.py”, line 390, in __get_result
raise self._exception
File “/usr/local/lib/python3.9/site-packages/middlewared/plugins/vm.py”, line 1227, in init_guest_vmemory
setvmem = await self.__set_guest_vmemory(guest_memory, overcommit)
File “/usr/local/lib/python3.9/site-packages/middlewared/plugins/vm.py”, line 1219, in __set_guest_vmemory
sysctl.filter(‘vfs.zfs.arc.max’)[0].value = new_arc_max
OSError: [Errno 22] Invalid argument

2

I would suggest updating to U6… there was some ARC fiddling. See if that helps. Also, you should not touch the zfs.arc_max since it’s dynamically allocated: if your system needs more RAM for VM use, the memory allocated to the caching will shrink.

3

Dear Davvo,

Thank you for your reply. I will try to update to U6 and advise shortly on the outcome. Can you also please confirm, i’ve seen in previous versions of TrueNAS that vfs.zfs.arc_max was a strongly recommended tuning parameter to ensure sufficient memory is reserved for VMs. Is this no longer the case in Version 13 ?

Thanks again for your prompt reply.

4

According to setting vfs.zfs.arc_max Invalid Arguement (trying to benchmark) | TrueNAS Community your issue might be due to an incorrect multiple inserted in the value.

ARC workings has been revamped in a recent ZFS update, and I remember quite a few tunables becoming useless/not functional… but should not affect this specific tunable.

Searching the forums prompts a “yes it is still the case” to your question, though from old threads; I am not sure myself. More experienced users will surely chime in about this, but you can try searching the old forum in the meantime. I also suggest adding the ZFS tag to the topic.

@pmh iirc bhyve does not need this anymore, am I wrong?

1 Like
5

I generally recommend to remove all old autotune values for TN 13.0 and above.

2 Likes
6

:point_up:

*I agree with this and also want to emphasize that others should heed this message.

2 Likes
7

Dear Davvo, pmh, and winnielinnie,

Thank you all for your prompt replies. As suggested I have added the ZFS tag to this thread.
Fist up, i removed the tunable however still received the same error. I then proceeded to upgrade to TrueNAS-13.0-U6.1 (with not legacy tunables) and can confirm all VMs including the new one started.

Additionally, looking at the dashboard, i can now see Services 46.2GB, ZFS Cache 7.9GB and Free 457.7GB. Previously these values were Services 116.5GB, ZFS Cache 14.9GB and Free 380GB.

It was my assumption defining a VM of certain memory removes that amount from Free and includes it in Services however it appears that services only looks at actual memory utilized?

I will assume that for now this issue is resolved.

Thank you once again for your support.
It is truly greatly appreciated.

1 Like
8

In ZFS, the ARC is dynamically allocated: if something needs RAM it will take it from the available first and then, if there is no more free, from the memory allocated to caching; since caching is designed to grow and utilize all available memory, services will eventually take RAM from evicting part of the cached data. All of this is automatically managed by ZFS. As far as I understand when you select a certain amount of RAM for a VM in CORE, you actually specify its maximum RAM usage: if it doesn’t require that much, TN will not allocate such memory to the services.

You can select a solution, changing this way the topic’s status.

Glad to have been of help.

1 Like
9

Thank you Davvo,
Greatly appreciated.

1 Like