Processing failed after 16h37m

Gordon · 1 October 2021 19:42

Imaging at -85 deg camera angle with an M2P, an area of scattered clumps of trees and grassy areas which I have stitched together previously without problems- suggestions as to why it may have failed this time would be appreciated.

[Montaray Road - 01/10/2021]
553 images 16:37:57
Processing failed (4294967295)

Created on: 01/10/2021, 09:36:41

Processing Node: node-odm-1 (manual)

Options: dem-resolution: 2.0, dsm: true, dtm: true, feature-quality: ultra, mesh-size: 200000, min-num-features: 9000, optimize-disk-space: true, orthophoto-resolution: 2.0, pc-geometric: true, pc-quality: ultra, pc-rectify: true

Here is the end of the log where it failed:

2021-10-01 18:30:55,715 DEBUG: Undistorting image DJI_0896.JPG
2021-10-01 18:30:55,736 DEBUG: Undistorting image DJI_0897.JPG
2021-10-01 18:30:57,269 DEBUG: Undistorting image DJI_0356.JPG
[INFO] running E:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\bin\opensfm export_visualsfm --points “E:\WebODM\resources\app\apps\NodeODM\data\4dcb5b98-d3e8-42cb-a362-7ae2327d6f48\opensfm”
[INFO] Finished opensfm stage
[INFO] Running openmvs stage
[INFO] running E:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\bin\opensfm export_openmvs “E:\WebODM\resources\app\apps\NodeODM\data\4dcb5b98-d3e8-42cb-a362-7ae2327d6f48\opensfm”
[INFO] Running dense reconstruction. This might take a while.
[INFO] Estimating depthmaps
[INFO] running E:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\OpenMVS\DensifyPointCloud “E:\WebODM\resources\app\apps\NodeODM\data\4dcb5b98-d3e8-42cb-a362-7ae2327d6f48\opensfm\undistorted\openmvs\scene.mvs” --resolution-level 1 --min-resolution 2736 --max-resolution 5472 --max-threads 16 --number-views-fuse 2 -w “E:\WebODM\resources\app\apps\NodeODM\data\4dcb5b98-d3e8-42cb-a362-7ae2327d6f48\opensfm\undistorted\openmvs\depthmaps” -v 0
===== Dumping Info for Geeks (developers need this to fix bugs) =====
Child returned 3221226505
Traceback (most recent call last):
File “E:\WebODM\resources\app\apps\ODM\stages\odm_app.py”, line 94, in execute
self.first_stage.run()
File “E:\WebODM\resources\app\apps\ODM\opendm\types.py”, line 341, in run
self.next_stage.run(outputs)
File “E:\WebODM\resources\app\apps\ODM\opendm\types.py”, line 341, in run
self.next_stage.run(outputs)
File “E:\WebODM\resources\app\apps\ODM\opendm\types.py”, line 341, in run
self.next_stage.run(outputs)
[Previous line repeated 1 more time]
File “E:\WebODM\resources\app\apps\ODM\opendm\types.py”, line 322, in run
self.process(self.args, outputs)
File “E:\WebODM\resources\app\apps\ODM\stages\openmvs.py”, line 79, in process
system.run(‘%s “%s” %s’ % (context.omvs_densify_path,
File “E:\WebODM\resources\app\apps\ODM\opendm\system.py”, line 106, in run
raise SubprocessException(“Child returned {}”.format(retcode), retcode)
opendm.system.SubprocessException: Child returned 3221226505

===== Done, human-readable information to follow… =====

[ERROR] The program exited with a strange error code. Please report it at https://community.opendronemap.org

Saijin_Naib · 1 October 2021 19:50

Can you re-try with --feature-quality high and everything else the same?

Gordon · 1 October 2021 19:51

OK thanks, I will start it now.

Gordon · 2 October 2021 21:15

That worked, but I’m not sure why ultra feature quality would be a problem.

Saijin_Naib · 2 October 2021 22:40

I’m guessing here, but super big reconstruction boundary due to sky and background not being masked and thus really big memory requirement, and lowering feature-quality side-steps this issue a bit.

Gordon · 3 October 2021 00:56

No sky in any of the images, most taken at -85deg, with a few at around -60 of a house

WillN · 9 October 2021 16:17

I encountered the same error at the same point in processing in a job I just ran. I used about triple the images, but memory did not seem to be an issue. I was monitoring resource usage at the time of failure. CPU had been maxed for awhile, but memory was at 50% utilization or less. All jobs at high quality or lower have been successful, but I haven’t managed an ultra quality job yet. I’m planning to queue up some experiments to see what the magical flag combination is, if there is one.

Imaging at -70 deg camera angle with a Parrot Anafi, altitude 125ft AGL, river flood plain with minimal vegetation, for the purpose of creating a DTM of restoration earthworks.

Processed on a platform with an AMD Ryzen 7 5800x 8 core, 128gb ram.

Options: build-overviews: true, dem-resolution: 1.0, dtm: true, feature-quality: ultra, orthophoto-resolution: 1.0, pc-geometric: true, pc-las: true, pc-quality: ultra, pc-rectify: true, verbose: true, rerun-from: opensfm

This was a rerun because it failed before, and I had the ignore-gsd flag set. A different thread said that can lead to unpredictable failures, so I removed it and set manual resolutions instead.

Here’s the end of my log:

2021-10-08 07:14:03,943 DEBUG: Undistorting image P4663414.JPG
2021-10-08 07:14:03,946 DEBUG: Undistorting image P4663415.JPG
2021-10-08 07:14:05,723 DEBUG: Undistorting image P4663416.JPG
[INFO] running C:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\bin\opensfm export_visualsfm --points “C:\WebODM\resources\app\apps\NodeODM\data\04cfef4b-09d8-43ba-9356-c8ce83862d14\opensfm”
[INFO] Finished opensfm stage
[INFO] Running openmvs stage
[INFO] running C:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\bin\opensfm export_openmvs “C:\WebODM\resources\app\apps\NodeODM\data\04cfef4b-09d8-43ba-9356-c8ce83862d14\opensfm”
[INFO] Running dense reconstruction. This might take a while.
[INFO] Estimating depthmaps
[INFO] running C:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\OpenMVS\DensifyPointCloud “C:\WebODM\resources\app\apps\NodeODM\data\04cfef4b-09d8-43ba-9356-c8ce83862d14\opensfm\undistorted\openmvs\scene.mvs” --resolution-level 1 --min-resolution 2304 --max-resolution 4608 --max-threads 16 --number-views-fuse 2 -w “C:\WebODM\resources\app\apps\NodeODM\data\04cfef4b-09d8-43ba-9356-c8ce83862d14\opensfm\undistorted\openmvs\depthmaps” -v 0
===== Dumping Info for Geeks (developers need this to fix bugs) =====
Child returned 3221226505
Traceback (most recent call last):
File “C:\WebODM\resources\app\apps\ODM\stages\odm_app.py”, line 89, in execute
self.first_stage.run()
File “C:\WebODM\resources\app\apps\ODM\opendm\types.py”, line 340, in run
self.next_stage.run(outputs)
File “C:\WebODM\resources\app\apps\ODM\opendm\types.py”, line 340, in run
self.next_stage.run(outputs)
File “C:\WebODM\resources\app\apps\ODM\opendm\types.py”, line 340, in run
self.next_stage.run(outputs)
[Previous line repeated 1 more time]
File “C:\WebODM\resources\app\apps\ODM\opendm\types.py”, line 321, in run
self.process(self.args, outputs)
File “C:\WebODM\resources\app\apps\ODM\stages\openmvs.py”, line 79, in process
system.run(‘%s “%s” %s’ % (context.omvs_densify_path,
File “C:\WebODM\resources\app\apps\ODM\opendm\system.py”, line 106, in run
raise SubprocessException(“Child returned {}”.format(retcode), retcode)
opendm.system.SubprocessException: Child returned 3221226505

===== Done, human-readable information to follow… =====

[ERROR] The program exited with a strange error code. Please report it at https://community.opendronemap.org

Saijin_Naib · 9 October 2021 17:10

Welcome!

Sorry you had trouble, but thanks for posting good debug info.

So, same job on same machine with high instead of ultra passes reliably? Interesting…

Gordon · 9 October 2021 21:04

Yes, same image set - high worked, ultra failed.

system · 8 November 2021 21:05

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.