Hello, we struggle for 5 months now with major errors on our production system: especially in our Java app that uses Apache FOP to create XML Reports that include images (SVG, etc.). We got “Image not found” erros every now and then!
This does not occur with the very same application on alöl other systems that dont use TrueNas. But right after we changed the production system to TrueNas these strange errors occured.
We ruled out Virus scanners (deactivated completely) even though they already scanned before TrueNas. We tried so many application workarounds to somehow tell FOP to use another ResourceResolver but it somehow always falls back to the standard “file system path” (e.g. "relative path “images/Logo1.svg”).
We are out of ideas to solve or workaround this purely random error - when we restart the same report job, it usually works then in 99% of case - if not, it works after the next retry.
I found similar posts here but without any response or hint what else to try.
How can we solve this random “image not found” which is basically “file not found” problem that is a lie because the file exists when we first test it in the application itself? (And it really exsist e.g. in “C:\temp\images\…” but we can’t trust anything anymore since the administration changed to TrueNas.
You need to provide more details on your TrueNAS server and your whole setup and how it interacts with your software. We are not there to observe what is happening and can not guess.
If this is a production system, and your I.T. Department does not know how to handle this situation, stop paying them. They did a bad thing for your company if they cannot solve this.
You can solve a lot on your own however ZFS failures, which it sounds like what you are describing, are nothing to play around with. If your business data is very important, then I’d advise contacting the TrueNAS Company and ask them for support. It will cost money but it is a dedicated team and from what I understand, good support.
Make sure you have a backup of all you data, you may be needing it.
This is a forum to share experience and find other fellows with similar probles and perhaps workarounds/solutions on the application side. “Call support” is not an answer. Of course they already checked and tried a lot of things but I am not involved in the IT - I am on the development side and it seems that “only our application” seems to have this problem.
We really can’t troubleshoot your described problem as there isn’t enough information to go on. Nothing is pointing to a TrueNAS or ZFS issue. It really needs to be worked on by your IT and development team together. If the IT team needs additional support for TrueNAS and ZFS, they need to contact TrueNAS Company and inquire about getting support.
@JavaAndTrueNas
How is Apache FOP being run? This is not a base app of the Truenas system and Truenas itself would have no support for it.
Typically since Apache FOP is an formatting objects process or and requires Java to work, it would be run in a custom docker container or in a virtual machine. The fact that, "And it really exsist e.g. in “C:\temp\images\…” leads me to guess it may be in a full blown VM that runs a webserver.
If you give the forum more information on how it was setup, how the current setup differs from the old (different server? different storage server? Updated? setup maybe someone can help put things on the correct path.
Well we have a pretty standard Java Application, running within its Java VM on a Windows OS (Windows Server2025). For several years these “Windows Servers” where VirtualMachines (I guess on VMWare). Last year out IT changed the production server and now we have new hardware and they changed everything to TrueNas. We (as application developers) don’t see anything of this (except new IP Adresses/share names) and therefore our applications ruin “as always”: e.g. writing to a Windows “C:\Temp” folder.
After switching to TrueNas on the new server Hardware, the problems with sometimes “missing” files began: we inserted a lot of logs in our applications and the files “missing” weren’t really missing - when we log their existence they “were there”, but the 3rd party API we use (Apache FOP XSL Processor) didn’t “find” such files sometimes.
I write these words in quotes because with Virtualization and TrueNas we can’t be sure of anything “existing”/”being there” because misc caching effects especially from TrueNas (and perhaps new, very fast NVMe memory ..) seem to trigger this problem.
So, we (app devs) are in a dark room confronted with these occasional “missing” files that are not really missing …
