WebODM + AMD GPU - Program exited with strange error Code

Hi,

I installed the native Windows WebODM version yesterday, and I was not able to finish a job yet, as the process of creating an ortho image never finishes. Below are details from the last job which exited with “a strange error code”.

I’m pretty sure it has to do with the fact that I’m using an AMD GPU. I get a system error - DensifyPointCloud.exe (nvcuda.dll was not found).
I was able to finish the job on a Windows PC with a Nvidia GPU.

Is it possible to run the latest WebODM build with an AMD GPU?

Regards

TASK OUTPUT:

===== Dumping Info for Geeks (developers need this to fix bugs) =====
Child returned 3221225781
Traceback (most recent call last):
File “D:\WebODM\resources\app\apps\ODM\stages\odm_app.py”, line 94, in execute
self.first_stage.run()
File “D:\WebODM\resources\app\apps\ODM\opendm\types.py”, line 346, in run
self.next_stage.run(outputs)
File “D:\WebODM\resources\app\apps\ODM\opendm\types.py”, line 346, in run
self.next_stage.run(outputs)
File “D:\WebODM\resources\app\apps\ODM\opendm\types.py”, line 346, in run
self.next_stage.run(outputs)
[Previous line repeated 1 more time]
File “D:\WebODM\resources\app\apps\ODM\opendm\types.py”, line 327, in run
self.process(self.args, outputs)
File “D:\WebODM\resources\app\apps\ODM\stages\openmvs.py”, line 83, in process
system.run(’%s “%s” %s’ % (context.omvs_densify_path,
File “D:\WebODM\resources\app\apps\ODM\opendm\system.py”, line 106, in run
raise SubprocessException(“Child returned {}”.format(retcode), retcode)
opendm.system.SubprocessException: Child returned 3221225781

===== Done, human-readable information to follow… =====

[ERROR] The program exited with a strange error code. Please report it at https://community.opendronemap.org

OPTIONS:

camera-lens: brown, dem-resolution: 10, dsm: true, feature-type: orb, max-concurrency: 5, mesh-octree-depth: 8, orthophoto-compression: JPEG, smrf-threshold: 3, smrf-window: 10

SYSTEM:

WebODM: Windows installer, version 1.9.11 build 43,
OS: Windows 10, v 21H1, build 19043.1348
CPU: AMD Ryzen 7 3700x
GPU: AMD RX 580
RAM: 32GB

1 Like

Whoops! Looks like we’re not gracefully falling back to CPU-only processing when we don’t find a suitable NVIDIA GPU.

I wonder if we could use a CUDA wrapper or translator in cases like this…

I’ll file a bug on GitHub for you.

1 Like

Much appreciated! I am able to work on the other PC with the NVIDIA GPU for now, but would welcome a AMD solution if possible :slight_smile:

1 Like

v.1.9.11 Build 44 (out now) should address this issue :slight_smile:

2 Likes

Now, back at my desk, I updated WebODM to build 45 (you guys are fast!) and tested it with a little benchmark job.

It now runs successfully past the DensifyPointCloud stage, detecting that an NVIDIA card is not available:

[INFO] Running openmvs stage
[INFO] running D:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\bin\opensfm export_openmvs “D:\WebODM\resources\app\apps\NodeODM\data\6e8461ab-8e42-47d9-8079-6cae56b71c07\opensfm”
[INFO] Running dense reconstruction. This might take a while.
[INFO] Estimating depthmaps
[INFO] No CUDA drivers detected, using CPU
[INFO] running D:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\OpenMVS\DensifyPointCloud “D:\WebODM\resources\app\apps\NodeODM\data\6e8461ab-8e42-47d9-8079-6cae56b71c07\opensfm\undistorted\openmvs\scene.mvs” --resolution-level 2 --min-resolution 684 --max-resolution 2697 --max-threads 5 --number-views-fuse 2 -w “D:\WebODM\resources\app\apps\NodeODM\data\6e8461ab-8e42-47d9-8079-6cae56b71c07\opensfm\undistorted\openmvs\depthmaps” -v 0 --geometric-iters 0 --cuda-device -1

However, I now get an error message at a later stage (meshing?), see info below. The same job with same settings again finishes at the NVIDIA machine. The error message mentions issues with the input data, but as the job finishes at the other machine, I doubt that this is the real reason for the error.

[INFO] Finished odm_filterpoints stage
[INFO] Running odm_meshing stage
[INFO] Writing ODM Mesh file in: D:\WebODM\resources\app\apps\NodeODM\data\6e8461ab-8e42-47d9-8079-6cae56b71c07\odm_meshing\odm_mesh.ply
[INFO] running “D:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\PoissonRecon” --in “D:\WebODM\resources\app\apps\NodeODM\data\6e8461ab-8e42-47d9-8079-6cae56b71c07\odm_filterpoints\point_cloud.ply” --out “D:\WebODM\resources\app\apps\NodeODM\data\6e8461ab-8e42-47d9-8079-6cae56b71c07\odm_meshing\odm_mesh.dirty.ply” --depth 8 --pointWeight 4.0 --samplesPerNode 1.0 --threads 4 --maxMemory 15 --bType 2 --linearFit
[INFO] running “D:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\OpenMVS\ReconstructMesh” -i “D:\WebODM\resources\app\apps\NodeODM\data\6e8461ab-8e42-47d9-8079-6cae56b71c07\odm_meshing\odm_mesh.dirty.ply” -o “D:\WebODM\resources\app\apps\NodeODM\data\6e8461ab-8e42-47d9-8079-6cae56b71c07\odm_meshing\odm_mesh.ply” --remove-spikes 0 --remove-spurious 20 --smooth 0 --target-face-num 400000
===== Dumping Info for Geeks (developers need this to fix bugs) =====
Child returned 3221225477
Traceback (most recent call last):
File “D:\WebODM\resources\app\apps\ODM\stages\odm_app.py”, line 94, in execute
self.first_stage.run()
File “D:\WebODM\resources\app\apps\ODM\opendm\types.py”, line 346, in run
self.next_stage.run(outputs)
File “D:\WebODM\resources\app\apps\ODM\opendm\types.py”, line 346, in run
self.next_stage.run(outputs)
File “D:\WebODM\resources\app\apps\ODM\opendm\types.py”, line 346, in run
self.next_stage.run(outputs)
[Previous line repeated 3 more times]
File “D:\WebODM\resources\app\apps\ODM\opendm\types.py”, line 327, in run
self.process(self.args, outputs)
File “D:\WebODM\resources\app\apps\ODM\stages\odm_meshing.py”, line 24, in process
mesh.screened_poisson_reconstruction(tree.filtered_point_cloud,
File “D:\WebODM\resources\app\apps\ODM\opendm\mesh.py”, line 207, in screened_poisson_reconstruction
system.run(’"{reconstructmesh}" -i “{infile}” ’
File “D:\WebODM\resources\app\apps\ODM\opendm\system.py”, line 106, in run
raise SubprocessException(“Child returned {}”.format(retcode), retcode)
opendm.system.SubprocessException: Child returned 3221225477

===== Done, human-readable information to follow… =====

[ERROR] Uh oh! Processing stopped because of strange values in the reconstruction. This is often a sign that the input data has some issues or the software cannot deal with it. Have you followed best practices for data acquisition? See Flying Tips — OpenDroneMap 2.6.7 documentation

1 Like

Oof… Yeah, it looks like something is crashing with an allocation error in that part of the pipeline.

Sorry you’re hitting these bugs, but thanks for giving great logs so we can surface what is going on here faster.

You’re welcome for the logs. Hope you can identify the problem.

1 Like

Hello!

I have encountered the same issue running ODM (native) on a PC with a Radeon RX 560 GPU, and posted a report on:

Thanks in advance for trying to sort this out.

1 Like

Welcome!

I know you are running Windows 7 on that machine and it isn’t likely that WebODM will work under 7, but do you think you might upgrade to Windows 10 on that hardware and try Build 45?

Hello Saijin:

Thanks for your quick reply and the suggestion to upgrade to Win 10. This is something I will have to do eventually but I was hoping to postpone a little longer.

I am not trying to run WebODM, just Windows Native ODM version 2.6.7. Does this run under Win 7 SP1? If it does, then would it be possible implement the same fix on Windows Native ODM as you made to WebODM build 45? Or perhaps I have misunderstood something?

Thanks for helping a newbie!

1 Like

I think we can, but I’m not sure if that fix will be backported to regular ODM yet, though the fix is live within WebODM. It may be something that is held for 2.6.8 or something.

The upgrade should be pretty painless and I performed it for one of our users this past week to get them running more smoothly with WebODM (Win 7- Win 11).

Thanks Saijin, I understand. Appreciate your help and advice.

1 Like

Mm, strange. @david86 could you share with us the contents of this folder?

2 Likes

Sure, but don’t be disappointed, all there is is an old dirty mesh :sweat_smile:

2 Likes