Invalid Docker-Compose.nodeodm.gpu.nvidia.yml

[korn704]

Good evening everyone,

I wanted to try the new GPU capabilities by launching ./webodm.sh start --gpu

Unfortunately, I don’t get far:

Any ideas?

Thank you.

[Saijin_Naib]

Hmm…

What OS? What Docker version? What version of docker-compose?

Have you pulled down the latest changes to WebODM prior to testing?

[korn704]

Saijin,

I’m running Ubuntu 20.04.3 LTS, Docker Engine - Community 20.10.12
This is a brand new install of WebODM just to test the new GPU capabilities. Latest changes of WebODM as of last night.

Thank you for looking into this.

[Saijin_Naib]

And tools like topgrade or sudo apt update && sudo apt upgrade && sudo apt dist-upgrade && sudo apt full-upgrade yield no updates needing to be done?

[korn704]

Nothing new Saijin. This is a machine I just boot up for this last night, everything should be latest.

Any other ideas?

This is where it points to an error: docker-compose.nodeodm.gpu.nvidia.yml

[Saijin_Naib]

Is this Ubuntu instance running on hardware, or virtualized/hypervized?

[pierotofy]

Try to update your docker-compose binary to the latest version (perhaps you’re running an older version?)

[korn704]

Saijin, the Ubuntu instance is running virtualized. ESXI - GPU Passthrough.

[korn704]

Piero thank you for the tip. That fixed the issue. Somehow the new installation installed docker-compose 1.25. Pulling the latest docker-compose installed 1.29 and allowed me to the next step.

However now, is advising that no drivers for Nvidia capabilities could be selected.

I wonder if its because of the GPU Passthrough virtualization? strange as nvidia-smi shows the GPU listed. Maybe an issue with a specific driver? Quadro K1200.

Running ubuntu-drivers devices shows driver 470 - as recommended:

[Saijin_Naib]

It needs to support CUDA 11.3+

topgrade did not show that package as outdated? Quite odd.

[korn704]

Saijin,

nvidia-smi shows CUDA 11.4

[plaksivaya_tryapka]

@korn704

I wonder if its because of the GPU Passthrough virtualization?

I used GPU Passthrough virtualization with Nvidia Tesla T4 on ESXi 6.5 - worked fine, ODM logs printed that GPU is detected. But it wasn’t significantly faster.

[korn704]

Thank you for the information, I figured ESXI Pass through should work fine as it does on different platforms, including on Linux VMs (Ubuntu in this case).I maybe missing something…

Interesting that your Tesla T4 which is way more capable - the difference wasn’t significantly faster than without a GPU. I’m guessing the other pipelines that are not multi threaded or optimized continue to be a bottleneck. I wanted to run some comparisons with the Quadro K1200 (ESXI Server) against a faster GPU like that one in my primary machine, two GTX 1080 TI. (3584 NVIDIA CUDA Cores each).

I mainly use DJI Terra, and Reality Capture on the primary machine. I continue to be amazed as to how fast the processing is and how good the quality of data is when using multi GPUs, high memory, and high amounts of CPU Cores.

[korn704]

Revisited the issue over the weekend. I got it working, I was missing the Nvidia Docker, and Nvidia-container-toolkit packages in Ubuntu.

Additional information here:

docker: Error response from daemon: could not select device driver "" with capabilities: [[gpu]].

https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html#docker

Hope it helps someone, thank you.