Processing failed after 16h37m

Imaging at -85 deg camera angle with an M2P, an area of scattered clumps of trees and grassy areas which I have stitched together previously without problems- suggestions as to why it may have failed this time would be appreciated.

[Montaray Road - 01/10/2021]
553 images 16:37:57
Processing failed (4294967295)

Created on: 01/10/2021, 09:36:41

Processing Node: node-odm-1 (manual)

Options: dem-resolution: 2.0, dsm: true, dtm: true, feature-quality: ultra, mesh-size: 200000, min-num-features: 9000, optimize-disk-space: true, orthophoto-resolution: 2.0, pc-geometric: true, pc-quality: ultra, pc-rectify: true

Here is the end of the log where it failed:

2021-10-01 18:30:55,715 DEBUG: Undistorting image DJI_0896.JPG
2021-10-01 18:30:55,736 DEBUG: Undistorting image DJI_0897.JPG
2021-10-01 18:30:57,269 DEBUG: Undistorting image DJI_0356.JPG
[INFO] running E:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\bin\opensfm export_visualsfm --points “E:\WebODM\resources\app\apps\NodeODM\data\4dcb5b98-d3e8-42cb-a362-7ae2327d6f48\opensfm”
[INFO] Finished opensfm stage
[INFO] Running openmvs stage
[INFO] running E:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\bin\opensfm export_openmvs “E:\WebODM\resources\app\apps\NodeODM\data\4dcb5b98-d3e8-42cb-a362-7ae2327d6f48\opensfm”
[INFO] Running dense reconstruction. This might take a while.
[INFO] Estimating depthmaps
[INFO] running E:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\OpenMVS\DensifyPointCloud “E:\WebODM\resources\app\apps\NodeODM\data\4dcb5b98-d3e8-42cb-a362-7ae2327d6f48\opensfm\undistorted\openmvs\scene.mvs” --resolution-level 1 --min-resolution 2736 --max-resolution 5472 --max-threads 16 --number-views-fuse 2 -w “E:\WebODM\resources\app\apps\NodeODM\data\4dcb5b98-d3e8-42cb-a362-7ae2327d6f48\opensfm\undistorted\openmvs\depthmaps” -v 0
===== Dumping Info for Geeks (developers need this to fix bugs) =====
Child returned 3221226505
Traceback (most recent call last):
File “E:\WebODM\resources\app\apps\ODM\stages\odm_app.py”, line 94, in execute
self.first_stage.run()
File “E:\WebODM\resources\app\apps\ODM\opendm\types.py”, line 341, in run
self.next_stage.run(outputs)
File “E:\WebODM\resources\app\apps\ODM\opendm\types.py”, line 341, in run
self.next_stage.run(outputs)
File “E:\WebODM\resources\app\apps\ODM\opendm\types.py”, line 341, in run
self.next_stage.run(outputs)
[Previous line repeated 1 more time]
File “E:\WebODM\resources\app\apps\ODM\opendm\types.py”, line 322, in run
self.process(self.args, outputs)
File “E:\WebODM\resources\app\apps\ODM\stages\openmvs.py”, line 79, in process
system.run(’%s “%s” %s’ % (context.omvs_densify_path,
File “E:\WebODM\resources\app\apps\ODM\opendm\system.py”, line 106, in run
raise SubprocessException(“Child returned {}”.format(retcode), retcode)
opendm.system.SubprocessException: Child returned 3221226505

===== Done, human-readable information to follow… =====

[ERROR] The program exited with a strange error code. Please report it at https://community.opendronemap.org

1 Like

Can you re-try with --feature-quality high and everything else the same?

1 Like

OK thanks, I will start it now.

1 Like

That worked, but I’m not sure why ultra feature quality would be a problem.

1 Like

I’m guessing here, but super big reconstruction boundary due to sky and background not being masked and thus really big memory requirement, and lowering feature-quality side-steps this issue a bit.

1 Like

No sky in any of the images, most taken at -85deg, with a few at around -60 of a house

1 Like

I encountered the same error at the same point in processing in a job I just ran. I used about triple the images, but memory did not seem to be an issue. I was monitoring resource usage at the time of failure. CPU had been maxed for awhile, but memory was at 50% utilization or less. All jobs at high quality or lower have been successful, but I haven’t managed an ultra quality job yet. I’m planning to queue up some experiments to see what the magical flag combination is, if there is one.

Imaging at -70 deg camera angle with a Parrot Anafi, altitude 125ft AGL, river flood plain with minimal vegetation, for the purpose of creating a DTM of restoration earthworks.

Processed on a platform with an AMD Ryzen 7 5800x 8 core, 128gb ram.

Options: build-overviews: true, dem-resolution: 1.0, dtm: true, feature-quality: ultra, orthophoto-resolution: 1.0, pc-geometric: true, pc-las: true, pc-quality: ultra, pc-rectify: true, verbose: true, rerun-from: opensfm

This was a rerun because it failed before, and I had the ignore-gsd flag set. A different thread said that can lead to unpredictable failures, so I removed it and set manual resolutions instead.

Here’s the end of my log:

2021-10-08 07:14:03,943 DEBUG: Undistorting image P4663414.JPG
2021-10-08 07:14:03,946 DEBUG: Undistorting image P4663415.JPG
2021-10-08 07:14:05,723 DEBUG: Undistorting image P4663416.JPG
[INFO] running C:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\bin\opensfm export_visualsfm --points “C:\WebODM\resources\app\apps\NodeODM\data\04cfef4b-09d8-43ba-9356-c8ce83862d14\opensfm”
[INFO] Finished opensfm stage
[INFO] Running openmvs stage
[INFO] running C:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\bin\opensfm export_openmvs “C:\WebODM\resources\app\apps\NodeODM\data\04cfef4b-09d8-43ba-9356-c8ce83862d14\opensfm”
[INFO] Running dense reconstruction. This might take a while.
[INFO] Estimating depthmaps
[INFO] running C:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\OpenMVS\DensifyPointCloud “C:\WebODM\resources\app\apps\NodeODM\data\04cfef4b-09d8-43ba-9356-c8ce83862d14\opensfm\undistorted\openmvs\scene.mvs” --resolution-level 1 --min-resolution 2304 --max-resolution 4608 --max-threads 16 --number-views-fuse 2 -w “C:\WebODM\resources\app\apps\NodeODM\data\04cfef4b-09d8-43ba-9356-c8ce83862d14\opensfm\undistorted\openmvs\depthmaps” -v 0
===== Dumping Info for Geeks (developers need this to fix bugs) =====
Child returned 3221226505
Traceback (most recent call last):
File “C:\WebODM\resources\app\apps\ODM\stages\odm_app.py”, line 89, in execute
self.first_stage.run()
File “C:\WebODM\resources\app\apps\ODM\opendm\types.py”, line 340, in run
self.next_stage.run(outputs)
File “C:\WebODM\resources\app\apps\ODM\opendm\types.py”, line 340, in run
self.next_stage.run(outputs)
File “C:\WebODM\resources\app\apps\ODM\opendm\types.py”, line 340, in run
self.next_stage.run(outputs)
[Previous line repeated 1 more time]
File “C:\WebODM\resources\app\apps\ODM\opendm\types.py”, line 321, in run
self.process(self.args, outputs)
File “C:\WebODM\resources\app\apps\ODM\stages\openmvs.py”, line 79, in process
system.run(’%s “%s” %s’ % (context.omvs_densify_path,
File “C:\WebODM\resources\app\apps\ODM\opendm\system.py”, line 106, in run
raise SubprocessException(“Child returned {}”.format(retcode), retcode)
opendm.system.SubprocessException: Child returned 3221226505

===== Done, human-readable information to follow… =====

[ERROR] The program exited with a strange error code. Please report it at https://community.opendronemap.org

1 Like

Welcome!

Sorry you had trouble, but thanks for posting good debug info.

So, same job on same machine with high instead of ultra passes reliably? Interesting…

Yes, same image set - high worked, ultra failed.

1 Like