Solution to all my problems
After a lot of reading, testing and help from you guys, here are the solutions to the problem I encountered.
Before you read, know that the tests were made on:
-
OS: Windows 11, Ubuntu 22.04 LTS, Pop!_OS 22.04 LTS
-
Laptop: MSI GE76 Raider
-
CPU: Intel 12th gen i7-12700H
-
RAM: 32 Gb
-
GPU: RTX 3070 Ti
Problem 1
Description
At first, on Windows 11, I used Docker and Ubuntu in Windows Subsystem for Linux (WSL) to run WebODM, but my NVIDIA GPU wasn’t recognized nor used to process the data. In the log, we could read the following:
[INFO] nvidia-smi **not** found **in** PATH, **using** CPU
In Linux, I got the same error.
Solution
- Installing the right OS and Kernel version
- Installing NVIDIA drivers
- Installing docker and docker-compose
- Installing NVIDIA-container-toolkit
- Check if everything is ready
- Running WebODM with GPU
Installing the right OS and Kernel version
Windows
If you choose to work on Windows, it’s a pretty straight forward process to install it. Just go on the Microsoft website and follow the instructions.
Ubuntu
- If you choose Ubuntu, go to the download page of the Ubuntu website and download the .iso file.
- Create a bootable USB (e.g., using Etcher)
- Boot from your USB drive
- Follow the instructions on the screen for the installation. If you check the box “Download update during installation”, it will also install your NVIDIA drivers.
BUT! I realized that Ubuntu 22.04 comes with the Linux kernel version 5.15. After reading on the internet (and that was also my case), there seem to be a bug with this version that leads to not recognizing my Wi-Fi card… Upgrading the kernel version solved the issue (connected with an ethernet cable…). To upgrade it, I used the “mainline” utility.
To install mainline:
sudo add-apt-repository ppa:cappelikan/ppa
sudo apt update
sudo apt install mainline
To list the kernel versions available, use
mainline --list
Then, install the one you want to use
mainline --install <version number>
Or to install the latest version,
mainline --install-latest
After upgrading the kernel version, my Wi-Fi adapter appeared again in "Settings"… but now the NVIDIA drivers were not recognized anymore using the nvidia-smi command…
I had to choose whether I wanted internet or my NVIDIA driver to work for WebODM…
I finally decided to install Pop!_OS instead.
Pop!_OS
To install Pop!_OS:
- Go to the download page and choose the version with NVIDIA drivers preloaded
- Create a bootable USB (e.g., using Etcher)
- Boot from your USB drive
- Follow the instructions on the screen for the installation
That is a pretty fluent installation. Pop!_OS recognized my Wi-Fi card (since it uses a more recent kernel version) and the NVIDIA drivers were already installed.
I still recommend updating your system using
sudo apt update
sudo apt full-upgrade
Installing NVIDIA drivers
Windows
To install the latest NVIDIA driver, use the installer provided by NVIDIA
Ubuntu
If you chose to download the updates during installation, the drivers are probably already installed.
If not, you have 2 choices
- Install the drivers following NVIDIA documentation
- Install the drivers from the NVIDIA driver page
Pop!_OS
If you chose the .iso file with the NVIDIA drivers preloaded, you have nothing to do. The drivers are already installed.
In either case, you can check if they’re correctly installed by running nvidia-smi in a terminal. You should see something like that:
[email protected]:~/Drone/WebODM$ nvidia-smi
Fri Jul 29 13:23:54 2022
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 515.57 Driver Version: 516.59 CUDA Version: 11.7 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA GeForce ... On | 00000000:01:00.0 Off | N/A |
| N/A 47C P8 10W / N/A | 176MiB / 8192MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| No running processes found |
+-----------------------------------------------------------------------------+
Installing docker and docker-compose
Windows
- Install Docker Desktop by running the installer. Get it on the official website.
- Check that the installation work correctly by running the following command in a terminal:
sudo docker run hello-world
Pop!_OS
Install the docker engine following the documentation. It’s a copy-paste process of a couple of commands. Just follow the steps.
Install also docker-compose by running
apt install docker-compose
Check that the installation work correctly by running the following command in a terminal:
sudo docker run hello-world
Installing NVIDIA-container-toolkit
To be able to use CUDA through Docker, you’ll need to install the nvidia-container-toolkit. You don’t have to do it on Windows, though.
Ubuntu
Install it from the NVIDIA documentation
- Set up Docker
curl https://get.docker.com | sh \
&& sudo systemctl --now enable docker
- Set up NVIDIA container toolkit
distribution=$(. /etc/os-release;echo $ID$VERSION_ID) \
&& curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
&& curl -s -L https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | \
sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update
sudo apt-get install -y nvidia-docker2
sudo systemctl restart docker
- Test it
sudo docker run --rm --gpus all nvidia/cuda:11.0.3-base-ubuntu20.04 nvidia-smi
The output should be:
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 450.51.06 Driver Version: 450.51.06 CUDA Version: 11.0 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 Tesla T4 On | 00000000:00:1E.0 Off | 0 |
| N/A 34C P8 9W / 70W | 0MiB / 15109MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| No running processes found |
+-----------------------------------------------------------------------------+
Pop!_OS
Because the nvidia-container-toolkit is only supported by a couple of distribution (see this page), you have some manipulations to do to be able to install it on Pop!_OS. You can also find them here:
distribution="ubuntu20.04" \
&& curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add - \
&& curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list
vi /etc/apt/preferences.d/nvidia-docker-pin-1002
with content;
Package: *
Pin: origin nvidia.github.io
Pin-Priority: 1002
sudo apt-get install -y nvidia-docker2
sudo systemctl restart docker
sudo docker run --rm --gpus all nvidia/cuda:11.0-base nvidia-smi
Check if everything is ready
- Test NVIDIA drivers detection
nvidia-smi
- Test Docker
sudo docker run hello-word
- Test CUDA
sudo docker run --rm --gpus all nvidia/cuda:11.0.3-base-ubuntu20.04 nvidia-smi
Running WebODM with GPU
Windows
- Open Docker
- Open a terminal (e.g. with Git Bash, not Ubuntu in WSL)
- Go to the WebODM folder
- Run
webodm.sh start --gpu
Pop!_OS
- Open a terminal (e.g. with Git Bash, not a Ubuntu in WSL)
- Go to the WebODM folder
- Run
webodm.sh start --gpu
Then, when you process a project, you should see in the log at the beginning something like nvidia-smi was detected.
Problem 2
Description
The processing time was faster with CPU only than with the GPU acceleration… which was not logical.
Solution
In my first tests, I use the fast-ortho option to process 120 images. Then the CPU-only option was faster (about 1 min 54 sec vs. 4 min for the GPU).
Then, by selecting the Default option, with the 3D rendering and DSM, that was a lot faster with GPU acceleration (about 9 min vs 16 min with CPU only).
I suppose that the GPU is more efficient for more complex map processing.
I hope this helps. 🚁