Can anyone suggest what may be causing a “strange error” to make nodeodm:gpu processing fail?
I’m trying nodeodm:gpu with a small dataset of 50 images and it’s failing with the following message in the console output:
File “/code/stages/openmvs.py”, line 83, in run_densify
system.run(’"%s" “%s” %s’ % (context.omvs_densify_path,
File “/code/opendm/system.py”, line 106, in run
raise SubprocessException(“Child returned {}”.format(retcode), retcode)
opendm.system.SubprocessException: Child returned 127
===== Done, human-readable information to follow… =====
Can you try re-running it without GPU acceleration in the normal Docker image to see if this is a problem with GPU passthrough via WSL2 with your AMD GPU?
I can run nodeodm:gpu containers on another W10 machine, a laptop with on onboard Intel GPU and a Nvidia GeForce GTX 1050 4GB, and it will complete the processing successfully but when I monitor the CPU & GPU activity via the Task Manager it looks to do the vast majority of processing with the CPU, and a little bit with the onboard Intel GPU, and nothing with the Nvidia GPU.
I ran into a brick wall trying to follow the Nvidia CUDA on WSL 2 installation instructions. I’m using Docker Desktop for Windows and the instructions for installing the NVidia Container Toolkit don’t seem to work with Docker Desktop.
Any suggestions?
Thanks Saijin, on the laptop with the Nvidia GPU this is the result:
PS C:\Users\johnny5> docker run --rm -it --gpus all nvcr.io/nvidia/k8s/cuda-sample:nbody nbody -gpu -benchmark
Run “nbody -benchmark [-numbodies=]” to measure performance.
-fullscreen (run n-body simulation in fullscreen mode)
-fp64 (use double precision floating point values for simulation)
-hostmem (stores simulation data in host memory)
-benchmark (run benchmark to measure performance)
-numbodies= (number of bodies (>= 1) to run in simulation)
-device= (where d=0,1,2… for the CUDA device to use)
-numdevices= (where i=(number of CUDA devices > 0) to use for simulation)
-compare (compares simulation results running once on the default GPU and once on the CPU)
-cpu (run n-body simulation on the CPU)
-tipsy=<file.bin> (load a tipsy model file for simulation)
NOTE: The CUDA Samples are not meant for performance measurements. Results may vary when GPU Boost is enabled.
Windowed mode
Simulation data stored in video memory
Single precision floating point simulation
1 Devices used for simulation
GPU Device 0: “Pascal” with compute capability 6.1
Compute 6.1 CUDA device: [NVIDIA GeForce GTX 1050]
5120 bodies, total time for 10 iterations: 4.656 ms
= 56.301 billion interactions per second
= 1126.017 single-precision GFLOP/s at 20 flops per interaction
Before starting the process make sure your system has virtualization extensions enable in bios (VT-x o AMD-v), Windows Subsystem for Linux depends on those extensions to run.
Also double check that your Windows 10 instance in updated to 21H2, if not do a manual update using the update asistand in the media creation tool page “Download Windows 10”
Install the WLS component for Windows 10, double check that your windows 10 is updated to 21H2. Open powershell or console in Windows and run:
wsl.exe --install
This procedure install the Virtual Machine Platform, Windows Subsystem for Linux, WLS Kernel, and Ubuntu for WLS. After the process finish, a new window opens with the linux console requesting to create a new user and password.
Enable Nvidia repository to install the needed libraries inside WLS. DO NOT INSTALL DRIVERS INSIDE WLS, follow the instructions in:
7.*Repository configuration | nvidia-docker**
If the procedure does not work try the following in WLS console:
If everything work till this point, the GPU can be seen from inside WLS running the command: nvidia-smi
Confirm that your docker instance can access the GPU running a cuda benchmark docker run --gpus all nvcr.io/nvidia/k8s/cuda-sample:nbody nbody -gpu -benchmark
Follow the WebODM install for linux; before launching webodm.sh modify the shell and mark the GPU_NVIDIA as true and save the file:
modify the docker-compose.nodeodm.gpu.nvidia.yml file and adjust it to export all GPUs available inside WLS to the container, this is a limitation of running docker inside WLS
# Chaining this file to the main docker-compose file adds
# a default processing node instance. This is best for users
# who are just getting started with WebODM.
version: '2.1'
services:
webapp:
depends_on:
- node-odm
environment:
- WO_DEFAULT_NODES
node-odm:
image: opendronemap/nodeodm:gpu
ports:
- "3000"
restart: unless-stopped
oom_score_adj: 500
deploy:
resources:
reservations:
devices:
- capabilities: [gpu]
There was a long-unmaintained project that transpiled CUDA to OpenCL so it would run across all the GPU architectures, but we’re not using that, so your Radeon will not be able to process CUDA code, nor will anything but an appropriate CUDA-compatible NVIDIA GPU.
So the take-aways for running WebODM on a W10 machine with GPU are:
Only Nvidia GPUs are supported; Radeon GPUs are not supported (because they have no CUDA support)
Don’t use Docker Desktop because the GPU pass-through functionality does not work. Instead you must…
a) install the Nvidia WLS CUDA driver for Windows,
b) install Docker & Docker Compose in your WSL Linux environment,
c) install the Nvidia libraries for Docker in the WSL Linux environment,
d) download and run the Nvidia CUDA WLS Installer in the WSL Linux environment,
e) install WebODM in the WSL Linux Docker environment, but before launching WebODM:
i) modify webodm.sh and docker-compose.nodeodm.gpu.nvidia.yml
iii) then run “./webodm.sh start --gpu all”
I’m not sure we can’t get Docker GPU passthrough to work, since it should be, in effect, more or less the same process as getting WSL2 GPU passthrough to work.
