Additional processing time with GCPs

I’m trying to process a 112 JPEG image using the DSM + DTM mode. The Ortho/DSM/DTM processes in about 50 minutes and look great. When I try to add GCPs and process, the process seems to hang up at processing the naive 3D. I have waited 16 hours before I cancelled it. It seems to get through the pair matching just fine and says the images/GCPs are well defined. I think I have entered the GCPs properly (5 Images per GCP / 5 GCPs / All the coordinates are in WGS 84). Any thoughts?

Konrad in TX

Where does it fail? I wonder if it is related to this bug: https://github.com/OpenDroneMap/ODM/issues/1065

I believe I have a different problem. The EXIF data on the JPEGs is good. When I process to an Ortho or DTM the match to the GCPs has an average error of 6’. I don’t have the current failure in front of me, but I believe the process made it through the pair matching and triangulation before it was lost in 3D model generation. Maybe I didn’t wait long enough. But a standard image which processed in under an hour shouldn’t take 16+ hours when I add GCPs.

I found and copied the last few lines of the processing status
2020-01-25 22:50:27,938 INFO: Removed outliers: 19632
2020-01-25 22:50:27,945 INFO: -------------------------------------------------------
2020-01-25 22:50:27,982 INFO: DJI_0098.JPG resection inliers: 1122 / 1134
2020-01-25 22:50:28,045 DEBUG: Ceres Solver Report: Iterations: 2, Initial cost: 9.603738e+03, Final cost: 9.394952e+03, Termination: CONVERGENCE
2020-01-25 22:50:28,056 INFO: Adding DJI_0098.JPG to the reconstruction
2020-01-25 22:50:28,074 INFO: -------------------------------------------------------
2020-01-25 22:50:28,075 INFO: Shots and/or GCPs are well-conditionned. Using naive 3D-3D alignment.
2020-01-25 22:50:31,657 INFO: Shots and/or GCPs are well-conditionned. Using naive 3D-3D alignment.

Hmm, so it’s not even getting past OpenSfM before taking a long time, odd. @pierotofy – any theories?

Not really sure, haven’t seen this one before.

@konradkern - this is WebODM, correct? Is everything on the same computer, or is the processing node on a different machine? (Also are you using Lightning cloud processing?)

The saga continues to unfold. First problem I found and fixed were the GCP locations. The GCP tool was taking my values with 8 significant figures in Lat/long truncating to two significant figures. Every one of my GCPs would have had a huge error. So I fixed that.

It ran again. In the MVE Depth Map Reconstruction there are a string of “Global View Selection failed”.

“MVE Scene to Pointset” seemed to run without errors

"MVE FSSR Mesh Cleaning " returns an error “mesh is empty” and threw up on itself with error 1.

Remember this is an image set which has run successful without the GCPs.

Ideas?

Tried to run the work again by increasing the resolution and reducing the number of cores from 4-3. Last calls
Error: Mesh is empty!
Traceback (most recent call last):
File “/code/run.py”, line 57, in
app.execute()
File “/code/stages/odm_app.py”, line 92, in execute
self.first_stage.run()
File “/code/opendm/types.py”, line 370, in run
self.next_stage.run(outputs)
File “/code/opendm/types.py”, line 370, in run
self.next_stage.run(outputs)
File “/code/opendm/types.py”, line 370, in run
self.next_stage.run(outputs)
File “/code/opendm/types.py”, line 370, in run
self.next_stage.run(outputs)
File “/code/opendm/types.py”, line 351, in run
self.process(self.args, outputs)
File “/code/stages/mve.py”, line 124, in process
system.run(’%s -t%s --no-clean --component-size=0 “%s” “%s”’ % (context.meshclean_path, min(1.0, args.mve_confidence), tree.mve_model, mve_filtered_model), env_vars={‘OMP_NUM_THREADS’: args.max_concurrency})
File “/code/opendm/system.py”, line 76, in run
raise Exception(“Child returned {}”.format(retcode))
Exception: Child returned 1

Ideas?

@konradkern - I ran into something that looks similar in nature with Error: mesh is empty!

I am trying to process a multi spectral set (attached) and can’t understand the error. The set is small with only 10imgs/band just to verify it runs - but it keep hitting the same issue both in webODM and ODM.

link to data

Log ends with:

[INFO] running /code/SuperBuild/src/elibs/mve/apps/meshclean/meshclean -t0.6 --no-clean --component-size=0 “/datasets/code/mve/mve_dense_point_cloud.ply” “/datasets/code/mve/mve_dense_point_cloud.filtered.ply”
MVE FSSR Mesh Cleaning (built on Mar 19 2020, 15:23:46)
Loading mesh: /datasets/code/mve/mve_dense_point_cloud.ply
PLY Loader: comment Export generated by libmve
Reading PLY: 0 verts… done.
Error: Mesh is empty!
Traceback (most recent call last):
File “/code/run.py”, line 61, in
app.execute()
File “/code/stages/odm_app.py”, line 92, in execute
self.first_stage.run()
File “/code/opendm/types.py”, line 344, in run
self.next_stage.run(outputs)
File “/code/opendm/types.py”, line 344, in run
self.next_stage.run(outputs)
File “/code/opendm/types.py”, line 344, in run
self.next_stage.run(outputs)
File “/code/opendm/types.py”, line 344, in run
self.next_stage.run(outputs)
File “/code/opendm/types.py”, line 325, in run
self.process(self.args, outputs)
File “/code/stages/mve.py”, line 89, in process
system.run(’%s -t%s --no-clean --component-size=0 “%s” “%s”’ % (context.meshclean_path, min(1.0, args.mve_confidence), tree.mve_model, mve_filtered_model), env_vars={‘OMP_NUM_THREADS’: args.max_concurrency})
File “/code/opendm/system.py”, line 76, in run
raise Exception(“Child returned {}”.format(retcode))
Exception: Child returned 1

Looking for ideas as well…
For a while I had problem with the multi-camera mode where ODM did not read the data properly and didn’t enter the proper mode. Once that cleared with @pierotofy’s help (tnx again ! https://github.com/OpenDroneMap/ODM/pull/1097) , I ran into the same error you mentioned here

Any ideas?
Tnx