System crash when attempting to start a VM with Hailo PCIe device

Ok, now I’ll finish my attempt to compile, after that I’ll decide. If I reinstall i may also use the module without enabling the mode :slight_smile:

I don’t think you’ll be able to use the module without developer mode. Doesn’t that need a writable root filesystem?

BTW, I also solved the problem of the strange error I got here. I had the same error while compiling and I found this. Changed to my pool directory and everything worked like a charm. :man_facepalming:

1 Like

Apparently I do not have the authorization to file issues with my account, the request to open a new one is refused.

Thats irritating but I can’t help there.
You should probably ask a member of iX staff about this.
Did you roughly follow this?

Something odd happens: when I login the system accept my credentials but the interface shows that I’m not logged in, and does not allow to login again. If I open a new issue it says that I’m not authorized because I am not logged in. I opened a second account but the result is the same. I guess this has something to do with the “great wall” of China where I’m leaving.
I opened a ticket in the customer support system to open a ticket. :rofl:
UPDATE - I did not purchased from them, the customer support cannot help. Anyone from the staff may reply here?

1 Like

Hi, I recently purchased Hailo device, too. For now I’m using it in Proxmox, not in TrueNAS, but while setting it up I found people reporting similar problems to yours.

Check this guide on using Hailo in a VM: Enabling Virtual Machine to access Hailo-8 devices on community.hailo.ai (sadly this forum does not allow me to paste links)
It starts with generic steps to enable passthrough on host, which you probably don’t need on TrueNAS, but then it talks about adjusting reset methods for Hailo device (Search for “Setting Hailo-8 reset method to null”).

When host start a VM, it wants to reset VM’s PCI devices, but Hailo device doesn’t support this. This may cause crashes/reboots/freezes on the host.

Thanks you so much @machekku
Unfortunately when I arrived at the last instruction I got an error:

# sudo update-initramfs -u -k all
ln: failed to create hard link '/boot/initrd.img-6.12.15-production+truenas.dpkg-bak': Read-only file system
cp: cannot create regular file '/boot/initrd.img-6.12.15-production+truenas.dpkg-bak': Read-only file system

TN does not allow to access this part of the fs, even in developer mode, so finally cannot use this solution.

I think you followed the guide too far. Let’s focus on restart method part.

You can check this manually by following these steps:

  1. Find ID of your hailo device in lspci -D output. On my machine it is:
0000:05:00.0 Co-processor: Hailo Technologies Ltd. Hailo-8 AI Processor (rev 01)
  1. Check your current reset method by executing
cat /sys/bus/pci/devices/0000\:05\:00.0/reset_method

(just rememeber to use your device ID).
This will most probably print flr bus.

  1. You may optionally try to reset hailo device now. But note: This will probably crash your system, just like when starting the VM! This is the reset command:
echo 1 > /sys/bus/pci/devices/0000\:05\:00.0/reset
  1. Disable reset methods for hailo device (since - according to the guide on hailo forums, it does not support resetting).
echo " " >  /sys/bus/pci/devices/0000\:05\:00.0/reset_method 
  1. Try resetting hailo device now. Hopefully nothing bad happens (but be careful: if this theory is wrong, your system may crash)
echo 1 > /sys/bus/pci/devices/0000\:05\:00.0/reset

You will probably see a response like this:

write error: Inappropriate ioctl for device

which means “sorry, I don’t know how to reset this device”.
6. If you got here without a crash, try starting the VM.

Note: Command from point 4 works only until next TrueNAS reboot. You need to find a way to run it on system startup.

1 Like

It works! thanks @machekku
I’m in developer mode now I have to switch back to standard installation and check if it works in that conditions as well (it should) and how to set it up at boot time.

Theoretically may work, but in practice it doesn’t because TN check the reset methods of the devices and if it is not set hide the device.
I found it when I changed the reset_method to null and then created the VM: when I wanted to add the device it was not anymore in the list of avaible devices, even though it appeared at the command lspci -D. Even if was not in the list at a certain point I got the message:

# Warning

PCI device does not have a reset mechanism defined and you may experience inconsistent/degraded behavior when starting/stopping the VM.

I tried to create first the VM, attach the device and after that change the reset method. It worked, I installed Debian but at the reboot the device was not avaiable anymore and the VM gave error.
The solution exists but does not work with TN.

You could try setting reset method to bus. While playing with hailo, I noticed that resetting with bus setting does nothing on my system. It seems that bus works in some hardware configurations but may be invalid in other (and then nothing happens). If this is true also for you, then you could have a non-empty reset method that doesn’t really do any reset.
You can watch dmesg output while attempting to reset hailo to see what really happens. (and as usual, be careful, in case this unfortunately crashes your system)

Thanks for sharing that. However I gave up already and transferred the Hailo device on a another minipc on which I’m installing a vanilla debian to run Frigate. On the mininas I’ll leave only Truenas with Plex.
I’m still playing aruond with these toys, therefore one of these days I may decide to give a try to your idea just for my knowledge.

Hi,

any new solution frof TrueNAS and Hailo?

Thank you

You may try the solution suggested by @machekku:

  1. as root set the reset method to bus
    echo "bus" > /sys/bus/pci/devices/[your_device_no]/reset_method
  2. try reset the device:
    echo 1 > /sys/bus/pci/devices/[your_device_no]/reset
    NB if the method does not work your system may crash and you may loose the data.
  3. If the method works you may try creating a VM and passing the Hailo device to it.
  4. If everything goes good build in the VM the Hailo drivers and install Frigate with Docker