So with NFS the processes middlewared and python are very busy. What is the relation to NFS service that it is triggering middlewared and python so hard? I see asyncio_loop is the main proces for the middlewared and python as subprocesses.
@kris What can we do more to narrow down the cause and fix this issue?
I don’t want to jump on @awalkerix toes here, if he’s already troubleshooting this behind the scenes. But I would assume this attached to a bug ticket and debug file for us to inspect configuration would be a start 
I am currently busy with other tickets / features for 25.04. I for some reason misremembered this being CPU bound and thought it was quick candidate for profiling.
Filing bug ticket would probably be a start if system exposing issue:
- is not virtualized
- has not had base OS modified.
Ok, opened this ticket: Jira
EDIT: it got automatically closed, can you check?
Anyway:
OS is not virtualized
OS is not modified
I tried to create bug report. But during the creation I received below the following traceback. And ticket NAS-133208 is in Jira closed directly by Bug Clerk. Not really helpful this way:
`Traceback (most recent call last):
File “/usr/lib/python3/dist-packages/requests/models.py”, line 971, in json
return complexjson.loads(self.text, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3.11/json/init.py”, line 346, in loads
return _default_decoder.decode(s)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3.11/json/decoder.py”, line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3.11/json/decoder.py”, line 355, in raw_decode
raise JSONDecodeError(“Expecting value”, s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File “/usr/lib/python3/dist-packages/middlewared/plugins/support.py”, line 381, in attach_ticket
data = r.json()
^^^^^^^^
File “/usr/lib/python3/dist-packages/requests/models.py”, line 975, in json
raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File “/usr/lib/python3/dist-packages/middlewared/job.py”, line 509, in run
await self.future
File “/usr/lib/python3/dist-packages/middlewared/job.py”, line 556, in __run_body
rv = await self.middleware.run_in_thread(self.method, *args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 1367, in run_in_thread
return await self.run_in_executor(io_thread_pool_executor, method, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 1364, in run_in_executor
return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3.11/concurrent/futures/thread.py”, line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/schema/processor.py”, line 183, in nf
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/schema/processor.py”, line 55, in nf
res = f(*args, **kwargs)
^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/plugins/support.py”, line 384, in attach_ticket
raise CallError(‘Invalid proxy server response’, errno.EBADMSG)
middlewared.service_exception.CallError: [EBADMSG] Invalid proxy server response
`
Our system is virtualized. But that was also with TN Core where is no NFS performance issue at all.
Also OS is not modified. Same hardware/disks/virtualization. Disk controller (SAS) is passthrough PCIe. TrueNAS has fully control over the disks.
Debug file can not uploaded to Jira ticket due closed by Bug Clerk.
Andrew Walker has reopened my ticket and I uploaded the debug file. Hopefully the cause of this issue can be found and fixed soon.
1 Like
@urgali Can you do this test as mentioned by Mark Grimes in my NS-133208 Jira ticket also?
I do not have the option to roll back to Core to do test compare.
i don’t have access to the jira ticket, can you wirte commands here?
@urgali I just sent you a DM.
Just for info for everyone, the issue is being checked on @Hunter jira ticket
we will update here once there is a fix
After some interaction with TrueNAS team, they can just simple not reproduce our issue and just close the ticket.
We almost ordered a hardware enterprise TrueNAS, but now they never will see the order at all. I’m finished with their arrogant way of handling this ticket. We could still do some more troubleshooting actions. But they are not interested.
We going to dump TrueNAS now and move to a different solution.
That is no doubt frustrating @Hunter. I’ll do a little internal digging to see what the situation was here.
However, we have to acknowledge the fact that we only have so many developers to go around. If we spent all day investigating every issue that isn’t reproducible on our own hardware, in as similar of a lab setup as we can manage, we’d hardly ever get anything done on the product. We typically have to prioritize issues we can reproduce on our own gear or paid enterprise customers environment.
That said, we absolutely want to do better in how we handle these cases, so I will check in on it.
2 Likes
My guess is that the problem is not reproducible, because the problem is somehow connected to PBS.
PBS introduces a lot of additional bottlenecks and adds a lot of complexity.
Garbage collection is mostly reading metadata.
So first thing I would test is if metadata read performance is worse with SCALE than CORE.
I don’t think so, but you never know.
My next paragraph is not to defend TrueNAS. It really could be the case that something in NFS got worse. But your setup is extremely poor for multiple reasons.
- PBS recommends mirror, you use RAIDZ1
- With RAIDZ1 only one drive can fail, but rebuild times are high with 9TB drives. This is pretty dangerous.
- RAIDZ1 with 7 drives is not optimal in case of pool geometry and could lead to worse storage efficiency than you expect
- RAIDZ could become very fragmented and performance could tank in the future.
- 32GB RAM is very poor
- adding an L2ARC makes the 32GB RAM even worse!
- Having PBS in a VM and accessing it over NFS is an added layer of complexity and possible bottleneck
What I would change:
A: If you use that system only for PBS, I would probably not use ZFS to begin with. PBS has integriy checks on its own. Your performance could be better with ext4 and your integrated RAID controller.
B: On the other hand, since you already have NVME drives, using them as special vdev instead of L2ARC could do wonders for metadata performance. That is also the reason why PBS really pressures you to use special vdev if you insist on HDDs instead of pure flash storage.
C: Use mirrors instead of RAIDZ
D: Whatever you do, run PBS bare metal with local storage
E: Read the manual of PBS. You haven’t. Otherwise you would have chosen special vdev instead of L2ARC. They go into performance considerations in the docs. They also offer a great performance benchmark tool to test for potential bottlenecks.
I love ghetto builds and I am a fan of “fu** around and find out”.
But I would not blame on TrueNAS, just because my ghetto build went (as expected) sideways.
If I don’t follow any recommendations, add multiple layers of complexity and can’t even reproduce my issue on another system, no company in this world will provide me any support.
Unless I pay per minute 
1 Like
Your description of my system is not even close at all.
Second, PBS is not the issue. It is already ruled out.
The only changed factor is TrueNAS core to Scale upgrade.
And with the upcoming Core+Scale->community edition migration, I guess more people will facing this bad performance issue.
I wasn’t talking about your system but OP.
As stated on previous posts, scenario is not optimal and we know it.
The question that no one was able to answer (everyone keeps saying we have to change this and that) is: why on CORE 12 it was super fast while on SCALE 24.04/24.10/24.10.1 is super slow with exact same setup/scenario?
Why we get better performances over SMB while NFS is bad?
There is obviosuly something problematic on SCALE NFS and, as reported above, I’m not the only one having this issue.
Also, good @Hunter has exact same issues with different setup.
So please, everyone, stop saying what/how to optimize, this is completely out of topic.
Thanks
1 Like
Even CORE 13.x has super fast NFS. Scale 24.x and 25.04 (Fangtooth) Nightly not!
Don‘t disagree, that is why I wrote that I can imagine NFS performance regression. This was more to help you setup a sane system. Because even if they can find the NFS problem, you have a way worse performance with that setup compared to what you could achieve even with the same hardware.
If this is like you claim a NFS on SCALE only problem, we should not mix PBS into it and simply test NFS with something like fio or run ls and see how long it takes to list directories.