I’m using a WebODM docker image to generate orthophotos; Although the orthophoto data was successfully generated, the processing process did not use GPU acceleration. I see a message in the log of the log.json: Cannot initialize CUDA, nvidia-smi detected
I checked the source code of webodm, it seems to be built based on the CUDA-11.2, and my CUDA version is 12.9. So, is it because the CUDA version is inconsistent?
Looking forward to your reply.
The attachment is the complete log.json log.json (75.2 KB)
Following the instructions there, I removed all environment configurations related to cuda-11-2, rebuilt a new Docker image, and the issue was successfully resolved.
It’s infrequently used in the code. I tried to build the docker image to enable opencv to use GPU, but the image uses an old version of ubuntu, and it errored out. The build step is convoluted since it tries to build everything in the “SuperBuild” and does so in parallel, so errors are very confusing.
Anyway, most of the code is not using the GPU, so it only accelerates a few areas. I use nvtop to watch usage of the GPU. It does do a lot of multi-threaded calcs, so that is an area you can speed things up, but there’s definitely a few bottlenecks that are single threaded.
Also, watchout for long running containers, they can lose the GPU silently and degrade to non-gpu use.
As you mentioned, I ran watch -n 1 nvidia-smi on my device to monitor the GPU-Util value. I noticed that the GPU was only active for less than 10 seconds while DensifyPointCloud was running, and the total time taken was approximately 13 minutes.
