Opendronemap/nodeodm:gpu - nvidia-smi detected - cannot initialize CUDA

jamesgriffiths · July 19, 2024, 12:29am

Hi Guys,

I have downloaded the latest version of wedODM (2.5.4) from git, and have this version running on a new Ubuntu 24.04 install with the latest docker, nvidia-container-toolkit, cuda-toolkit, etc installed.

I can successfully run webODM 2.5.4 with GPU on this install (with a bit of fiddling), and can run NodeODM remotely through through port 3000 with cpu only. But, have had no luck running nodeodm remotely through port 3000 with gpu.

I get the following log…
[INFO] nvidia-smi detected
[INFO] Using CPU for feature extraction: Cannot initialize CUDA

The nodeodm container was set up as follows…
“docker run -p 3000:3000 --gpus all opendronemap/nodeodm:gpu -q 1 --token secret”

Any ideas on how to resolve this conundrum are very welcome.

Cheers

J

Saijin_Naib · July 19, 2024, 5:13am

Are you positive you have the latest Docker images?

We just pushed a fix to ODM/NodeODM for GPU:

github.com/OpenDroneMap/NodeODM

libtbb2 is missing in nodeodm:gpu docker container

opened 04:55AM - 15 Jul 24 UTC

fri-K

### What is the problem? NodeODM started as docker container is failing becau…se of lack of library > [INFO] Writing MVS Textured file in: /var/www/data/0fac5ca1-719a-4e71-b68f-06ac94092e3e/odm_texturing_25d/odm_textured_model_geo.obj [INFO] running "/code/SuperBuild/install/bin/texrecon" "/var/www/data/0fac5ca1-719a-4e71-b68f-06ac94092e3e/opensfm/undistorted/reconstruction.nvm" "/var/www/data/0fac5ca1-719a-4e71-b68f-06ac94092e3e/odm_meshing/odm_25dmesh.ply" "/var/www/data/0fac5ca1-719a-4e71-b68f-06ac94092e3e/odm_texturing_25d/odm_textured_model_geo" -d gmi -o gauss_clamping -t none --no_intermediate_results --nadir_mode --max_texture_size=8192 /code/SuperBuild/install/bin/texrecon: error while loading shared libraries: **libtbb.so.2**: cannot open shared object file: **No such file or directory** ### What should be the expected behaviour? NodeODM should proceed without issues ### How can we reproduce this? (What steps did you do to trigger the problem? Be detailed) Start new task from WebODM and it fails every time Workaround is to install it manually: `docker exec -it --user root NODE_ODM_HASH /bin/bash -c 'apt update && apt install libtbb2 -y'` It's a new issue, previous images were running fine, new image id is: 8055b7f678a5

jamesgriffiths · July 19, 2024, 5:32am

Hi Saijin, thanks for your response,

I think so. I ran a fresh install of docker, nvidia-container-toolkit, cuda-toolkit yesterday and downloaded a fresh version of WebODM a couple of hours ago.

When I run apt install libtbb2 inside the web-odm-node-1 container I get…

root@#############:/var/www# apt install libtbb2
Reading package lists… Done
Building dependency tree
Reading state information… Done
libtbb2 is already the newest version (2020.1-2).

However, when I run nvidia-smi I get…

±----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| No running processes found |
±----------------------------------------------------------------------------------------+
root@########:/var/www#

It seems the GPU is found but is running with the wrong CUDA version.

J

Saijin_Naib · July 19, 2024, 11:44pm

Hmm, we might have to have other folks weigh in for what is working for them or not.

jamesgriffiths · July 20, 2024, 10:45pm

Thanks Saijin,

And, thanks to the rest of the team for an awesome imagery processing platform.

I can confirm that running ODM directly with GPU enabled (e.g. docker run -ti --rm -v /home/xxxxxx/datasets:/datasets --gpus all opendronemap/odm:gpu --project-path /datasets project) works perfectly on both my Ubuntu 24.04 machines (machine 1: CPU = Tripper 5995wx and GPU = GeForce RTX 3060. machine 2:CPU = Xeon E5/Core i7 and GPU = T400 4GB) with the latest docker, cuda-toolkit, and nvidia-ctk installed on both.

docker version
Client: Docker Engine - Community
Version: 27.0.3
API version: 1.46
Go version: go1.21.11
Git commit: 7d4bcd8
Built: Sat Jun 29 00:02:23 2024
OS/Arch: linux/amd64
Context: default

cuda-toolkit-12-5 version (12.5.1-1)

NVIDIA Container Toolkit CLI version 1.16.0

When I run the latest opendronemap/nodeodm:gpu image, it runs on cpu only and I get
[INFO] nvidia-smi detected
[INFO] Using CPU for feature extraction: Cannot initialize CUDA

Cheers

j

fri.K · August 17, 2024, 9:34pm

Hi,
I’m experiencing similar issue. If host is running cuda v12.2, nodeodm docker container prints cuda version 11.2 in nvidia-smi and refuses to use GPU with:

Blockquote
[INFO] nvidia-smi detected
[INFO] Using CPU for feature extraction: Cannot initialize CUDA

dpkg in container shows that only 11.2 libraries are installed. Just to check if it is possible to use GPU with CUDA I started nanominer-linux-3.7.4-cuda11 and it works without an issue.
As a very temporary (and brutal) workaround I removed all cuda 11.2 related packages from running container and installed cuda 12.2. With this simple trick nvidia-smi reports correct version and nodeODM no longer complains about CUDA.
Now I’m observing that bin/OpenMVS/DensifyPointCloud is executed on GPU.

I’m going to check if there is some minimalistic set of packages that would allow me to unblock CUDA as reinstalling everything is bit of pain

TIP: To manage packages on nodeODM you need to be a root, so you need to execute following command:

docker exec -it --user root nodeODMhashORname /bin/bash

dhuyp · August 19, 2024, 8:04am

Hello,

Similar issue here. But 12.6 on the host and 12.6 in the container. Everything is up-to-date on a Fedora system. Maybe, the code doesn’t manage 12.6 CUDA?

fri.K · August 20, 2024, 7:28am

Are you sure that container has only 12.6? To check it (if you’re using official build) run

docker exec -it nodeODMhashORname /bin/bash -c '/usr/bin/dpkg -l | grep cuda'

dhuyp · August 20, 2024, 12:09pm

I have check directly with the command ‘nvidia-smi’ with docker container exec on the node.

dhuyp · August 26, 2024, 10:37pm

Any solution? I see some people have the same issue.

dhuyp · September 2, 2024, 7:21pm

I add some information:

odm@9c55e3d9b36f:/var/www$ nvidia-smi
Mon Sep 2 19:12:32 2024
±----------------------------------------------------------------------------------------+
| NVIDIA-SMI 560.35.03 Driver Version: 560.35.03 CUDA Version: 12.6 |
|-----------------------------------------±-----------------------±---------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA GeForce RTX 3080 Off | 00000000:01:00.0 On | N/A |
| 0% 40C P8 28W / 320W | 553MiB / 10240MiB | 0% Default |
| | | N/A |
±----------------------------------------±-----------------------±---------------------+
±----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
±----------------------------------------------------------------------------------------+

odm@9c55e3d9b36f:/var/www$ nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2021 NVIDIA Corporation
Built on Sun_Feb_14_21:12:58_PST_2021
Cuda compilation tools, release 11.2, V11.2.152
Build cuda_11.2.r11.2/compiler.29618528_0

For information the system is a Fedora up-to-date. I used proprietary driver of Fedora comming from nvidia-container-tookit and cuda-fedora39 (the second is for the host). WebODM is the only container where I have problem with Cuda and it works ok otherwise on the system.

EDIT: the error is different now. Before I had:

[INFO] nvidia-smi detected
[INFO] Using CPU for feature extraction: Cannot initialize CUDA

Now I have only nvidia-smi detected and the node just don’t use GPU (according to nvidia-smi and regarding the extraction methods (Extracting ROOT_DSPSIFT features for image DJI_0510.JPG)

cjgars · September 16, 2024, 9:56pm

Pushing on fri.K’s work a bit more, we were able to get a docker image that mitigated this problem and got CUDA working.

Add the following to Dockerfile.gpu after the chown commands and rebuild the image. Even with these packages removed, CUDA works fine. This allows us to use the same fixed image with ClusterODM, as well.

Installing CUDA 12 into the container is not required.

RUN dpkg -r cuda-libraries-11-2
RUN dpkg -r cuda-cudart-11-2
RUN dpkg -r cuda-compat-11-2
RUN dpkg -r cuda-nvrtc-11-2
RUN dpkg -r cuda-nvtx-11-2

Saijin_Naib · September 17, 2024, 5:38am

Excellent work, and welcome!

Do you think you have the time to polish this up as a PR for the image so others can benefit as well?

smathermather · September 18, 2024, 3:23pm

Do you have a sense, is there any way to do this at a higher level with apt?

cjgars · September 19, 2024, 1:07am

Yes. I tested the below and it seems to work equally as well.

RUN apt-get remove -y cuda-libraries-11-2 cuda-cudart-11-2 cuda-compat-11-2 cuda-nvrtc-11-2 cuda-nvtx-11-2

github.com/OpenDroneMap/NodeODM

Fix for CUDA init failure

OpenDroneMap:master ← cjgarson:master

opened 01:06AM - 19 Sep 24 UTC

cjgarson

+3 -1

There appear to be conflicts between the container-based CUDA libraries and most… host CUDA libraries/Nvidia drivers, such that NodeODM output will complain that CUDA cannot be initialized even though `nvidia-smi` is detected. Removing these packages appears to resolve the conflict without causing any other issues. Installing newer versions of the same does not appear to be necessary. See this thread for more info: https://community.opendronemap.org/t/opendronemap-nodeodm-gpu-nvidia-smi-detected-cannot-initialize-cuda Also changed maintainer style to resolve deprecation warnings. Submitting PR on request from Saijin.

dhuyp · October 7, 2024, 3:27pm

Still not working for me unfortunately or maybe did I miss a step?
I change the Dockerfile and shoot “webodm.sh --gpu start”

EDIT: I access directly to the running container to uninstall. It works and removes the “Cannot initialize CUDA”. Unfortunately still no GPU for features extraction.

Saijin_Naib · October 7, 2024, 4:07pm

If the image dimensions do not fit into a CUDA-accerable texture map, it will fall back to CPU.

There will be a log entry for this. Can you check your log?

dhuyp · October 7, 2024, 4:49pm

Nothing in console about that

INFO]    ==============
[INFO]    Running dataset stage
[INFO]    Loading dataset from: /var/www/data/56893a75-f6bf-4eaa-ace5-7202c406eac6/images
[INFO]    Loading 1471 images
[INFO]    Found 1471 usable images
[WARNING] GPS position not available for DJI_0918.JPG.102.JPG
[WARNING] GPS position not available for DJI_0919.JPG.102.JPG
[WARNING] GPS position not available for DJI_0917.JPG.102.JPG
[WARNING] GPS position not available for DJI_0916.JPG.102.JPG
[WARNING] GPS position not available for DJI_0915.JPG.102.JPG
[WARNING] GPS position not available for DJI_0914.JPG.102.JPG
[WARNING] GPS position not available for DJI_0913.JPG.102.JPG
[WARNING] GPS position not available for DJI_0912.JPG.102.JPG
[WARNING] GPS position not available for DJI_0911.JPG.102.JPG
[WARNING] GPS position not available for DJI_0910.JPG.102.JPG
[INFO]    Finished dataset stage
[INFO]    Running split stage
[INFO]    Normal dataset, will process all at once.
[INFO]    Finished split stage
[INFO]    Running merge stage
[INFO]    Normal dataset, nothing to merge.
[INFO]    Finished merge stage
[INFO]    Running opensfm stage
[INFO]    Found 26 image masks
[INFO]    Maximum photo dimensions: 2048px
[INFO]    Photo dimensions for feature extraction: 1024px
[INFO]    nvidia-smi detected
[INFO]    ['use_exif_size: no', 'flann_algorithm: KDTREE', 'feature_process_size: 1024', 'feature_min_frames: 10000', 'processes: 8', 'matching_gps_neighbors: 0', 'matching_gps_distance: 0', 'matching_graph_rounds: 50', 'optimize_camera_parameters: yes', 'reconstruction_algorithm: incremental', 'undistorted_image_format: tif', 'bundle_outlier_filtering_type: AUTO', 'sift_peak_threshold: 0.066', 'align_orientation_prior: vertical', 'triangulation_type: ROBUST', 'retriangulation_ratio: 2', 'matcher_type: FLANN', 'feature_type: DSPSIFT', 'align_method: orientation_prior', 'local_bundle_radius: 0']
[INFO]    Wrote reference_lla.json
[INFO]    running "/code/SuperBuild/install/bin/opensfm/bin/opensfm" detect_features "/var/www/data/56893a75-f6bf-4eaa-ace5-7202c406eac6/opensfm"
2024-10-07 16:44:44,905 INFO: Planning to use 70009.611328125 MB of RAM for both processing queue and parallel processing.
2024-10-07 16:44:44,905 INFO: Scale-space expected size of a single image : 23.8125 MB
2024-10-07 16:44:44,905 INFO: Expecting to queue at most 200 images while parallel processing of 8 images.
2024-10-07 16:44:44,907 INFO: Reading data for image DJI_0003.JPG (queue-size=0)
2024-10-07 16:44:44,907 INFO: Reading data for image DJI_0197.JPG (queue-size=0)

(Extract)

system · November 6, 2024, 4:49pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.