NOTE: I’m not allowed to post all of the information since I’m a new user, so I’m hoping that it will let me post it as follow up comments.
I’m having trouble processing with Intel hardware.
First of all, I’ve been working on a branch that includes automation to create/install a Debian package. One of the primary reasons, and challenges, has been supporting Intel OpenCL. It detects available GPU hardware and customizes the installation for that. In order to make that work, I’ve modified the nodeodm docker image to install Intel’s OpenCL Debian release packages. It was also necessary to ensure that the devices had matching permissions on the host and in the container, so I’ve also made the process run as an “odm” user, changing ownership of everything under /var/www to that user.
(BTW, the intention is to offer this as PRs when it’s all working.)
I’ve verified this on various hardware that is available to me and my coworkers and have dozens of successful runs. However, while also trying to optimize speed and quality, we’ve started to run into issues where some tasks (with identical source data and options) works with NVIDA hardware, but fails with Intel hardware.
The error that we see is “Uh oh! Processing stopped because of strange values in the reconstruction. This is often a sign that the input data has some issues or the software cannot deal with it.” I’ve read that others have run into this with memory limits, so I’ve explicitly told docker-compose to give the container most of the available memory. This hasn’t seemed to have any effect, and I can watch “docker stats” and see that the container never uses more than 1GB of RAM, even though a few are available.
I’ve also jumped into the container and run the failing command and approximately 1/5 attempts succeed:
Console output with Intel is in the drive folder, named console_intel.txt
Console output with NVIDIA is in the drive folder, named console_nvidia.txt
I’m sorry if this is all too much information, but I want to be clear about what I’ve done and what’s happening. Thanks so much for any help! It would be very much appreciated.
Thanks for tackling this awesome PR, and thanks for looking to help upstream it to us!
This is all well beyond me, so I apologize, but do you know if perhaps the Intel iGPU is exhausting its shared/hardware memory pool during processing? That might be one difference between the nVidia and Intel OpenCL processing.
Thanks! I hope that it can be useful for others. It’s definitely not perfect, but it’s helping us to install it on different devices.
Ah, I hadn’t looked into shared memory and was focusing on docker container memory. I’m searching/reading now to figure out how to monitor that. Thanks for the tip!