"nvidia-smi not found in PATH, using CPU" error even though the path is in there

WebODM
,
#1

So I have the path to the nvidia-smi in my PATH env variable and have restarted ODM but still receive this error when processing:

[INFO] nvidia-smi not found in PATH, using CPU

Here are results running it:

C:\Program Files\NVIDIA Corporation\NVSMI>nvidia-smi.exe
Tue Jan 4 12:30:25 2022
±----------------------------------------------------------------------------+
| NVIDIA-SMI 461.40 Driver Version: 461.40 CUDA Version: 11.2 |
|-------------------------------±---------------------±---------------------+
| GPU Name TCC/WDDM | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 GeForce GTX 970M WDDM | 00000000:01:00.0 On | N/A |
| N/A 52C P8 5W / N/A | 539MiB / 3072MiB | 2% Default |
| | | N/A |
±------------------------------±---------------------±---------------------+

1 Like
#2

Are you using WebODM for Windows native, or Docker?

#3

sorry should have mentioned… it is docker

1 Like
#4

What docker image are you using? Did you pull the ODM GPU one?
https://hub.docker.com/layers/opendronemap/odm/gpu/images/sha256-99a5b4fc35cffb0e97a7fd00d64098b166122eb3a5e3214827e71e5c2c707334?context=explore

#5

not too sure…cloned repo, pulled latest then ran docker for windows and then ./webodm start in terminal on ubuntu

#6
opendronemap/nodeodm IN USE latest 8d058e352fe3 about 23 hours ago 1.72 GB
opendronemap/webodm_webapp IN USE latest 5c272bac345e 13 days ago 1.59 GB
redis IN USE latest 7614ae9453d1 14 days ago 112.69 MB
opendronemap/webodm_db IN USE latest e40c0f274bba 4 months ago 694.72 MB
1 Like
#7

Mmm, not sure we have GPU passthrough working with Docker on Windows.

AFAIK, that’s still pretty experimental on NVIDIA and Microsoft’s end, and requires specific drivers and versions of Windows/WSL2/WSL2-Kernel:

I was never successful myself in getting this working.

#8

ok…thanks for looking.

the native windows has support for it?

1 Like
#9

Indeed it does! Provided your GPU meets the required CUDA version support.

It looks like, with the latest Windows drivers, your 970m should be sufficient for our CUDA runtime of v11.4 in the WebODM for Windows native install.

#10

Hi, I have installed webodm windows native version. I have an nvidia geforce g210 graphics card and it tells me it is unsupported. Does it depend on the drivers?

1 Like
#11

Welcome!

Unfortunately, the GeForce 210 only supports CUDA v6.5, and we’re using v11.4.

You’d need a newer card and the latest drivers, both.

#12

Could nvidia quadro P400 be okay?

1 Like
#13

Looks like Compute v6.5, and latest driver for it supports CUDA 11.5, so yes!
CUDA GPUs | NVIDIA Developer
BkNVR510Quadro_Win10.book (nvidia.com)

1 Like
#14

please try this way from github

This led me to finding another solution by looking into /etc/nvidia-container-runtime/config.toml file where the ldconfig is by default set to “@/sbin/ldconfig”. This for some reason seems to not be working and also produces the error above:

root@banshee:/var/log# docker run --rm --gpus=all nvidia/cuda:11.4-base nvidia-smi
NVIDIA-SMI couldn't find libnvidia-ml.so library in your system. Please make sure that the NVIDIA Display Driver is properly installed and present in your system.
Please also try adding directory that contains libnvidia-ml.so to your system PATH.

Changing the ldconfig path to “/sbin/ldconfig” (instead of “@/sbin/ldconfig”) does indeed fix the problem:

root@banshee:/var/log# docker run --rm --gpus=all nvidia/cuda:11.4-base nvidia-smi
Sun Jan  5 20:39:45 2020       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 430.64       Driver Version: 430.64       CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 970     On   | 00000000:01:00.0  On |                  N/A |
| 32%   39C    P8    16W / 170W |    422MiB /  4038MiB |      3%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
+-----------------------------------------------------------------------------+
1 Like
#15

I have pulled latest image :
opendronemap/nodeodm:gpu
Digest:sha256:19084574a08db473f6b50a9ee9ae8a655780a4feb7bffb9b9704a06ad122ca0d
Then:

docker exec -it opendronemap/nodeodm:gpu /bin/bash

then:

odm@cdc30658474a:/var/www$ nvidia-smi
bash: nvidia-smi: command not found
odm@cdc30658474a:/var/www$

Same behavior when

cd WebODM/
./webodm.sh --port 80 --gpu start

then:

docker exec -it webodm_node-odm_1 /bin/bash

odm@cdc30658474a:/var/www$ nvidia-smi
bash: nvidia-smi: command not found
odm@cdc30658474a:/var/www$

My point is , there are missing nvidia drivers inside the opendronemap/nodeodm:gpu container.
Because nvidia-smi is part from nvidia-driver-***

OR , there is no need nvidia-driver-*** to be installed inside …
not sure …
CUDA is there , btw

1 Like
#16

Yeah, I don’t think the nvidia drivers/smi need to be inside the container as they’re going to be communicating with the hardware outside of it, which does need it.

#17

have you tried my way in my previous chat? let me know if it works (or not)

1 Like
#18

Additional tips from me

Please change the ./webodm.sh

From:

lspci | grep 'VGA.*NVIDIA'
		if [ "${?}" -eq 0 ]; then
			export GPU_NVIDIA=true
			set -e
			return
		fi

To:

lspci | grep 'NVIDIA'
		if [ "${?}" -eq 0 ]; then
			export GPU_NVIDIA=true
			set -e
			return
		fi

IT SEEMS SUCCESS :smiley:

>>> ./webodm.sh start --gpu
01:00.0 3D controller: NVIDIA Corporation GP108M [GeForce MX150] (rev a1)
Checking for docker...   OK
Checking for docker-compose...   OK
Starting WebODM...

Using the following environment:
================================
Host: localhost
Port: 8000
Media directory: appmedia
SSL: NO
SSL key: 
SSL certificate: 
SSL insecure port redirect: 80
Celery Broker: redis://broker
Default Nodes: 1
================================
Make sure to issue a ./webodm.sh down if you decide to change the environment.

docker-compose -f docker-compose.yml -f docker-compose.nodeodm.gpu.nvidia.yml up --scale node-odm=1
Pulling node-odm (opendronemap/nodeodm:gpu)...
1 Like