Windows Docker GPU

[thadwald]

When I run WebODM with the —gpu option, I get the following message in the task log:

nvidia-smi not found in PATH, using cpu

I did nothing special to enable gpu when I added the docker image. Please advise.

[Saijin_Naib]

Sorry for the trouble! Could you please start with giving me a bit more information about your system?

For instance:

1. Operating System and Version
   eg: Windows 11, MacOS 12.4, Ubuntu Linux 22.04LTS, etc…
2. Hardware Specifications
   eg: 32GB RAM, i7-6700k, NVIDIA GTX 1050TI OC, 1TB SSD, etc…
3. WebODM Install Method
   eg: Native installer, Docker, Snap, GitHub download, compiled from source
4. WebODM/ODM Version
   eg: WebODM v1.9.14 Build 64, ODM v2.8.7, etc…
5. Procesing Node
   eg: Automatic, Lightning, local (node-odm-1), etc…
6. Screenshots demonstrating the issue/behavior/error messages

[thadwald]

Thank you for your response

1. Operating System and Version
   Windows 11, latest version, Latest Nvidia drivers from the Nvidia website
2. Hardware Specifications
   128GB Ram, Intel i9-7980ex processor, Quadro P5000 graphics, 1TB Samsung 990 PRO
3. WebODM Install Method
   Docker
4. WebODM/ODM Version
   WebODM v2.0.0, ODM v (not sure, fresh install, latest updates)
5. Procesing Node
   local (node-odm-1)
6. Screenshots demonstrating the issue/behavior/error messages
   Hopefully this explanation will suffice:
   This is a fresh install on a system that was gifted to me. During some tests on small datasets I graphed the load on the graphics card. There was none, so I looked at the log file and found the line I noted in my original post:

        nvidia-smi not found in PATH, using cpu

The nvidia-smi command works in a WSL terminal, so it would appear that the correct driver is installed on my system.

[Saijin_Naib]

Can you try following this post from about #4 on?

Installation of WebODM GPU doesn't detect NVIDIA MX150

Before starting the process make sure your system has virtualization extensions enable in bios (VT-x o AMD-v), Windows Subsystem for Linux depends on those extensions to run. Also double check that your Windows 10 instance in updated to 21H2, if not do a manual update using the update asistand in the media creation tool page “Download Windows 10” Procedure: Install the WLS Cuda driver for windows, use this URL to download the latest driver from nvidia: GPU in Windows Subsystem for Linux (…

[thadwald]

Ahh, so you install docker from inside the WSL subsystem.
I’ll try it and report back.

As I understand, it’s also possible to compile and install WebODM/
ODM directly on Windows. has anyone done a performance comparison between those.

Also, It appears that with this approach, I would be running 2 nested virtual machines, kind of. Has anyone tested and documented any performance penalties of doing that?

EDIT: I see now that you suggested to try from #4 on… I’ll try that

[Cronix]

This is the post I used to get it working properly. It’s a lot simpler than most other things I’ve read through/tried.

NodeODM:GPU exited with strange error code

You dont need all the above. In my case it works just fine with docker: in bash/cmd run nvidia-smi run benchmark in bash/cmd (to confirm docker sees cuda drivers) docker run --gpus all nvcr.io/nvidia/k8s/cuda-sample:nbody nbody -gpu -benchmark changed odm files (start.sh and docker-compose.nodeodm.gpu.nvidia.yml- they are located in webodm directory) check topic above what lines to change run in bash webodm.sh --gpu Works just fine… at least in my case

It automatically downloaded everything needed. I just had to quickly edit the 2 files mentioned after that and start with the --gpu option.

Steps:

1. Install latest nvidia windows drivers

From webodm directory you installed to:

2. run “docker run --gpus all nvcr.io/nvidia/k8s/cuda-sample:nbody nbody -gpu -benchmark”
   This will download the docker images needed, and run a test using the gpu

3. Edit webodm.sh file. Search for “export GPU_NVIDIA=false” and change it to true:

export GPU_NVIDIA=true

4. Edit docker-compose.nodeodm.gpu.nvidia.yml. Ensure the “devices” section at the very bottom looks like this (with proper indentation, which the below is not using, but the main text should already be there and you should only have to update the values):

devices:
  - driver: nvidia
    count: 1
    capabilities: [gpu]

5. start webodm with working GPU, ./webodm.sh start --gpu

An easy way to see if it worked is in docker, check which image node-odm-1 is using. It should say opendronemap/nodeodm:gpu with the important part being :gpu

[thadwald]

The GPU test line:

“docker run --gpus all nvcr.io/nvidia/k8s/cuda-sample:nbody nbody -gpu -benchmark

… downloads the image and hangs.

I can get it to run by removing docker desktop and installing the lInux Docker on WSL. But WebODM isn’t configured to work with Docker running in WSL.

[Cronix]

That’s odd. I’m using docker desktop for windows via WSL2. Are you using WSL1 or 2? All commands were entered via a Git Bash terminal within the directory I installed webodm into, which was my regular windows user profile (C:/Users/[username]/webodm). I have not done anything within WSL shell itself. I installed WebODM via manual installation.

I’m not manually doing anything from within the WSL instance cli.

It would be nice if webodm could just detect gpu and set it up for you. It took me weeks of trial and error, following different (very long) forum posts, multiple reinstalls, etc. Just by passing the --gpu flag, I’d think it would be able to use the appropriate settings without editing things and it seems there should be an easy way to automate setting this up (including downloading appropriate images if not present) if the --gpu flag is passed.

[Saijin_Naib]

We’d definitely love testing and contribution of such a fix to our docker setup!

[thadwald]

On top of my own mistakes, it appears that Docker Desktop v4.17.1 is broken:
Docker Desktop for 4.17.1 breaks any --gpu usage for existing installations · Issue #13324 · docker/for-win · GitHub

The only two options are rolling back the upgrade or waiting for the fix. I don’t have option 1 because this is a fresh install so I guess I wait.

[Cronix]

That sucks. It looks like you can grab the previous version 4.17.0 from here: Docker Desktop release notes

There are other versions along the right hand side as well.

[thadwald]

hmm…
Running the 4.17.0 setup triggered Windows Defender, which stopped the process.
Now running 4.17.1 installer–downloaded and successfully run a few days ago-- triggers the AV as well. There’s some chatter in the Docker github discussion about the same.

[thadwald]

The Windows Defender Trojan alarm on Docker Desktop was a false positive.
It’s fixed now with the latest update.

[Svallone]

I’m updating the post, very similar issue but don’t think is antivirus related.

Successfullly installed webodm on laptop with windows 11 and nvidia 3060 dgpu following this or this post.

Trying the same with a desktop PC with a new nvidia 4070ti but the GPU test below downloads the image and hangs.

@thadwald

“docker run --gpus all nvcr.io/nvidia/k8s/cuda-sample:nbody nbody -gpu -benchmark

I tried several times to reinstall docker desktop and webodm but nothing seems to fix this issue. Checked that docker desktop runs on wsl2 with docker info:

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc., v0.10.3)
  compose: Docker Compose (Docker Inc., v2.15.1)
  dev: Docker Dev Environments (Docker Inc., v0.1.0)
  extension: Manages Docker extensions (Docker Inc., v0.2.18)
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc., 0.6.0)
  scan: Docker Scan (Docker Inc., v0.25.0)
  scout: Command line tool for Docker Scout (Docker Inc., v0.6.0)

Server:
 Containers: 5
  Running: 5
  Paused: 0
  Stopped: 0
 Images: 6
 Server Version: 20.10.23
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 2456e983eb9e37e47538f59ea18f2043c9a73640
 runc version: v1.1.4-0-g5fd4c4d
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
  cgroupns
 Kernel Version: 5.15.90.1-microsoft-standard-WSL2
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 24
 Total Memory: 60.83GiB
 Name: docker-desktop
 ID: 3RHJ:JPPX:L2XA:OX6A:TIPQ:WFGE:YJVZ:YT7Q:OYDC:P3BU:EDUV:2EB2
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

[Svallone]

Oooh! Now I see the problem:

Laptop, docker version is 4.17.0
Desktop, docker version is 4.17.1

I will try to downgrade to version 4.17 to see if this fix the GPU issue also in my case!

I will keep you posted

[Svallone]

Ok, that was the main issue.

Docker now runs GPU tests flawlessly. But:

.\webodm.sh start --gpu

Does not run yet even after changing webodm.sh and and docker-compose.nodeodm.gpu.nvidia.yml as suggested above.

If you want to run .\webodm.sh start --gpu from git bash correctly, you MUST run this command before, always in bash (not tested in cmd):

docker run -p 3000:3000 -v /mnt/external_hd:/var/www/data opendronemap/nodeodm

It should be added to the points above!