Split-Merge Fail with Docker

First Time Reaching Out for Help, have googled, and looked alot of post, but cant seem to figure it out, i am no computer expert, enough to poke around.

I am trying to process, 900+ images so i used the split function, which converged all the images, then aligned. I am now getting an error to finish the processing.

Any Help would be greatly apprecitated, as i expect to need to do a few of these larger data sets, as i fly for ag and anything over 30 acres gets to be alot of images.

Thanks

docker run -ti --rm -v C:\droneprojects\K807_31\7_31:/datasets/code opendronemap/odm --project-path /datasets --split 300 --split-overlap 100 --merge all
[INFO] Initializing OpenDroneMap app - Fri Aug 02 18:14:33 2019
[INFO] ==============
[INFO] build_overviews: False
[INFO] cameras: {}
[INFO] crop: 3
[INFO] debug: False
[INFO] dem_decimation: 1
[INFO] dem_euclidean_map: False
[INFO] dem_gapfill_steps: 3
[INFO] dem_resolution: 5
[INFO] depthmap_resolution: 640
[INFO] dsm: False
[INFO] dtm: False
[INFO] end_with: odm_orthophoto
[INFO] fast_orthophoto: False
[INFO] gcp: None
[INFO] ignore_gsd: False
[INFO] images: None
[INFO] matcher_distance: 0
[INFO] matcher_neighbors: 8
[INFO] max_concurrency: 6
[INFO] merge: all
[INFO] mesh_octree_depth: 9
[INFO] mesh_point_weight: 4
[INFO] mesh_samples: 1.0
[INFO] mesh_size: 100000
[INFO] min_num_features: 8000
[INFO] mve_confidence: 0.6
[INFO] name: code
[INFO] opensfm_depthmap_method: PATCH_MATCH
[INFO] opensfm_depthmap_min_consistent_views: 3
[INFO] opensfm_depthmap_min_patch_sd: 1
[INFO] orthophoto_bigtiff: IF_SAFER
[INFO] orthophoto_compression: DEFLATE
[INFO] orthophoto_cutline: False
[INFO] orthophoto_no_tiled: False
[INFO] orthophoto_resolution: 5
[INFO] pc_classify: False
[INFO] pc_csv: False
[INFO] pc_ept: False
[INFO] pc_filter: 2.5
[INFO] pc_las: False
[INFO] project_path: /datasets
[INFO] rerun: None
[INFO] rerun_all: False
[INFO] rerun_from: None
[INFO] resize_to: 2048
[INFO] skip_3dmodel: False
[INFO] sm_cluster: None
[INFO] smrf_scalar: 1.25
[INFO] smrf_slope: 0.15
[INFO] smrf_threshold: 0.5
[INFO] smrf_window: 18.0
[INFO] split: 300
[INFO] split_overlap: 100.0
[INFO] texturing_data_term: gmi
[INFO] texturing_keep_unseen_faces: False
[INFO] texturing_nadir_weight: 16
[INFO] texturing_outlier_removal_type: gauss_clamping
[INFO] texturing_skip_global_seam_leveling: False
[INFO] texturing_skip_hole_filling: False
[INFO] texturing_skip_local_seam_leveling: False
[INFO] texturing_skip_visibility_test: False
[INFO] texturing_tone_mapping: none
[INFO] time: False
[INFO] use_3dmesh: False
[INFO] use_exif: False
[INFO] use_fixed_camera_params: False
[INFO] use_hybrid_bundle_adjustment: False
[INFO] use_opensfm_dense: False
[INFO] verbose: False
[INFO] ==============
[INFO] Running dataset stage
[INFO] Loading dataset from: /datasets/code/images
[INFO] Loading images database: /datasets/code/images.json
[INFO] Found 983 usable images
[INFO] Coordinates file already exist: /datasets/code/odm_georeferencing/coords.txt
[INFO] Parsing SRS header: WGS84 UTM 16N
[INFO] Finished dataset stage
[INFO] Running split stage
[INFO] Large dataset detected (983 photos) and split set at 300. Preparing split merge.
[WARNING] /datasets/code/opensfm/image_list.txt already exists, not rerunning OpenSfM setup
[WARNING] Detect features already done: /datasets/code/opensfm/features exists
[WARNING] Match features already done: /datasets/code/opensfm/matches exists
[WARNING] Submodels directory already exist at: /datasets/code/submodels
[INFO] Dataset has been split into 4 submodels. Reconstructing each submodel…
[INFO] Reconstructing /datasets/code/submodels/submodel_0000/opensfm
[WARNING] Found a valid OpenSfM tracks file in: /datasets/code/submodels/submodel_0000/opensfm/tracks.csv
[WARNING] Found a valid OpenSfM reconstruction file in: /datasets/code/submodels/submodel_0000/opensfm/reconstruction.json
[INFO] Reconstructing /datasets/code/submodels/submodel_0001/opensfm
[WARNING] Found a valid OpenSfM tracks file in: /datasets/code/submodels/submodel_0001/opensfm/tracks.csv
[WARNING] Found a valid OpenSfM reconstruction file in: /datasets/code/submodels/submodel_0001/opensfm/reconstruction.json
[INFO] Reconstructing /datasets/code/submodels/submodel_0002/opensfm
[WARNING] Found a valid OpenSfM tracks file in: /datasets/code/submodels/submodel_0002/opensfm/tracks.csv
[WARNING] Found a valid OpenSfM reconstruction file in: /datasets/code/submodels/submodel_0002/opensfm/reconstruction.json
[INFO] Reconstructing /datasets/code/submodels/submodel_0003/opensfm
[WARNING] Found a valid OpenSfM tracks file in: /datasets/code/submodels/submodel_0003/opensfm/tracks.csv
[WARNING] Found a valid OpenSfM reconstruction file in: /datasets/code/submodels/submodel_0003/opensfm/reconstruction.json
[WARNING] Found a alignment done progress file in: /datasets/code/opensfm/alignment_done.txt
[INFO] Submodel submodel_0000 has already been aligned.
[INFO] Submodel submodel_0001 has already been aligned.
[INFO] Submodel submodel_0002 has already been aligned.
[INFO] Submodel submodel_0003 has already been aligned.
[INFO] ========================
[INFO] Processing submodel_0000
[INFO] ========================
[INFO] running /code/run.py --project-path /datasets/code/submodels --merge --orthophoto-cutline --dem-euclidean-map --skip-3dmodel submodel_0000
usage: run.py [options]
run.py: error: argument --merge: expected one argument
Traceback (most recent call last):
File “/code/run.py”, line 56, in
app.execute()
File “/code/stages/odm_app.py”, line 93, in execute
self.first_stage.run()
File “/code/opendm/types.py”, line 376, in run
self.next_stage.run(outputs)
File “/code/opendm/types.py”, line 357, in run
self.process(self.args, outputs)
File “/code/stages/splitmerge.py”, line 150, in process
system.run(" ".join(map(quote, argv)), env_vars=os.environ.copy())
File “/code/opendm/system.py”, line 76, in run
raise Exception(“Child returned {}”.format(retcode))
Exception: Child returned 2

Hey @bulf90 :hand: thanks for reporting this, it’s a bug. We have a fix on the way in https://github.com/OpenDroneMap/ODM/pull/1014/files. For the time being you can simply add “code” to the command to work around it.

docker run -ti --rm -v C:\droneprojects\K807_31\7_31:/datasets/code opendronemap/odm --project-path /datasets --split 300 --split-overlap 100 --merge all code
2 Likes

Thanks a bunch @pierotofy. This worked. Also now know where to look to find more issues.

Side question: when splitting a data set, i can normally run 400-500 photos with out any issues, if i split down to that or less and run large data sets, will there be any other points that will cause memory issues? Seems like it fails in the reconstruct process. Is there any other process when merging them back together will cause me problems.

The othro produced looks great for this project. Would like to start processing larger acreage fields knowing i can get the split merge to work.

Andrew

Where does it fail in the reconstruct process? Can you share the log?

I have run into a similar issue with split-merge: apparently there is some issue with the GRASS and polygons when cutting the submodel outputs, which results in the same output after all the submodels have been processed:

[INFO] Executing grass script from /datasets/code/submodels/submodel_0000/odm_orthophoto/grass_cutline_tmpdir: /usr/bin/grass76 --tmp-location /datasets/code/submodels/submodel_0000/odm_orthophoto/grass_cutline_tmpdir/odm_orthophoto.scaled.tif --exec bash script.she[0m
Starting GRASS GIS…
Creating new GRASS GIS location …
Executing …
Reading band 2 of 4…
Link to raster map created.
ERROR: Unable to open data source
/datasets/code/submodels/submodel_0000/odm_georeferencing/odm_georeferenced_model.bounds.gpkg
ERROR: Vector map <crop_area> not found
Using tiles processing for edge detection
Creating edge map
Finding cutlines in both directions
Creating vector polygons
WARNING: Categories will be unique sequence, raster values will be lost.
Erasing temporary files…
ERROR: Vector map <crop_area> not found
ERROR: Vector map not found
FAILURE:
Unable to open datasource `result.gpkg’ with the following drivers.

And after the process is done, it gives the following error:

[INFO] Merging all crop bounds: [’/datasets/code/submodels/submodel_0000/odm_georeferencing/odm_georeferenced_model.bounds.gpkg’, ‘/datasets/code/submodels/submodel_0001/odm_georeferencing/odm_georeferenced_model.bounds.gpkg’, ‘/datasets/code/submodels/submodel_0002/odm_georeferencing/odm_georeferenced_model.bounds.gpkg’, ‘/datasets/code/submodels/submodel_0003/odm_georeferencing/odm_georeferenced_model.bounds.gpkg’, ‘/datasets/code/submodels/submodel_0004/odm_georeferencing/odm_georeferenced_model.bounds.gpkg’, ‘/datasets/code/submodels/submodel_0005/odm_georeferencing/odm_georeferenced_model.bounds.gpkg’, ‘/datasets/code/submodels/submodel_0006/odm_georeferencing/odm_georeferenced_model.bounds.gpkg’, ‘/datasets/code/submodels/submodel_0007/odm_georeferencing/odm_georeferenced_model.bounds.gpkg’, ‘/datasets/code/submodels/submodel_0008/odm_georeferencing/odm_georeferenced_model.bounds.gpkg’, ‘/datasets/code/submodels/submodel_0009/odm_georeferencing/odm_georeferenced_model.bounds.gpkg’, ‘/datasets/code/submodels/submodel_0010/odm_georeferencing/odm_georeferenced_model.bounds.gpkg’, ‘/datasets/code/submodels/submodel_0011/odm_georeferencing/odm_georeferenced_model.bounds.gpkg’, ‘/datasets/code/submodels/submodel_0012/odm_georeferencing/odm_georeferenced_model.bounds.gpkg’, ‘/datasets/code/submodels/submodel_0013/odm_georeferencing/odm_georeferenced_model.bounds.gpkg’, ‘/datasets/code/submodels/submodel_0014/odm_georeferencing/odm_georeferenced_model.bounds.gpkg’, ‘/datasets/code/submodels/submodel_0015/odm_georeferencing/odm_georeferenced_model.bounds.gpkg’, ‘/datasets/code/submodels/submodel_0016/odm_georeferencing/odm_georeferenced_model.bounds.gpkg’, ‘/datasets/code/submodels/submodel_0017/odm_georeferencing/odm_georeferenced_model.bounds.gpkg’, ‘/datasets/code/submodels/submodel_0018/odm_georeferencing/odm_georeferenced_model.bounds.gpkg’, ‘/datasets/code/submodels/submodel_0019/odm_georeferencing/odm_georeferenced_model.bounds.gpkg’, ‘/datasets/code/submodels/submodel_0020/odm_georeferencing/odm_georeferenced_model.bounds.gpkg’, ‘/datasets/code/submodels/submodel_0021/odm_georeferencing/odm_georeferenced_model.bounds.gpkg’, ‘/datasets/code/submodels/submodel_0022/odm_georeferencing/odm_georeferenced_model.bounds.gpkg’, ‘/datasets/code/submodels/submodel_0023/odm_georeferencing/odm_georeferenced_model.bounds.gpkg’]e[0m
Traceback (most recent call last):
File “/code/run.py”, line 56, in
app.execute()
File “/code/stages/odm_app.py”, line 93, in execute
self.first_stage.run()
File “/code/opendm/types.py”, line 376, in run
self.next_stage.run(outputs)
File “/code/opendm/types.py”, line 376, in run
self.next_stage.run(outputs)
File “/code/opendm/types.py”, line 357, in run
self.process(self.args, outputs)
File “/code/stages/splitmerge.py”, line 206, in process
Cropper.merge_bounds(all_bounds, merged_bounds_file, 0)
File “/code/opendm/cropper.py”, line 82, in merge_bounds
layer = ds.GetLayer()
AttributeError: ‘NoneType’ object has no attribute ‘GetLayer’

My workaround to this is to download each submodel one by one and cut them manually, which is far from perfect, but it does work. I will try adding the “code” and see if that helps

@smathermather-cm this is the end when if fails when i dont run split-merge, 970 images.

Termination: CONVERGENCE
2019-08-06 09:40:13,296 INFO: Adding DJI_0160.JPG to the reconstruction
2019-08-06 09:40:14,042 DEBUG: Local bundle sets: interior 30 boundary 87 other 695
2019-08-06 09:40:20,111 DEBUG: Ceres Solver Report: Iterations: 5, Initial cost: 8.419775e+03, Final cost: 8.180024e+03, Termination: CONVERGENCE
2019-08-06 09:40:55,941 INFO: Removed outliers: 24
2019-08-06 09:40:56,282 INFO: -------------------------------------------------------
2019-08-06 09:40:56,347 INFO: DJI_0159.JPG resection inliers: 3555 / 3565
2019-08-06 09:40:56,549 DEBUG: Ceres Solver Report: Iterations: 5, Initial cost: 1.939920e+03, Final cost: 4.812671e+02, Termination: CONVERGENCE
2019-08-06 09:40:56,554 INFO: Adding DJI_0159.JPG to the reconstruction
2019-08-06 09:40:57,237 DEBUG: Local bundle sets: interior 30 boundary 87 other 696
2019-08-06 09:41:02,955 DEBUG: Ceres Solver Report: Iterations: 4, Initial cost: 8.395599e+03, Final cost: 8.188013e+03, Termination: CONVERGENCE
2019-08-06 09:41:38,138 INFO: Removed outliers: 23
2019-08-06 09:41:38,469 INFO: -------------------------------------------------------
2019-08-06 09:41:38,537 INFO: DJI_0158.JPG resection inliers: 3617 / 3650
2019-08-06 09:41:38,675 DEBUG: Ceres Solver Report: Iterations: 5, Initial cost: 2.120102e+03, Final cost: 5.102256e+02, Termination: CONVERGENCE
2019-08-06 09:41:38,682 INFO: Adding DJI_0158.JPG to the reconstruction
2019-08-06 09:41:39,346 DEBUG: Local bundle sets: interior 30 boundary 87 other 697
2019-08-06 09:41:45,266 DEBUG: Ceres Solver Report: Iterations: 5, Initial cost: 8.398295e+03, Final cost: 8.169157e+03, Termination: CONVERGENCE
2019-08-06 09:42:20,833 INFO: Removed outliers: 18
2019-08-06 09:42:21,169 INFO: -------------------------------------------------------
2019-08-06 09:42:21,267 INFO: DJI_0157.JPG resection inliers: 4117 / 4118
2019-08-06 09:42:21,410 DEBUG: Ceres Solver Report: Iterations: 5, Initial cost: 1.118338e+03, Final cost: 4.971320e+02, Termination: CONVERGENCE
2019-08-06 09:42:21,414 INFO: Adding DJI_0157.JPG to the reconstruction
2019-08-06 09:42:21,684 INFO: Re-triangulating
Killed
Traceback (most recent call last):
File “/code/run.py”, line 56, in
app.execute()
File “/code/stages/odm_app.py”, line 93, in execute
self.first_stage.run()
File “/code/opendm/types.py”, line 376, in run
self.next_stage.run(outputs)
File “/code/opendm/types.py”, line 376, in run
self.next_stage.run(outputs)
File “/code/opendm/types.py”, line 376, in run
self.next_stage.run(outputs)
File “/code/opendm/types.py”, line 357, in run
self.process(self.args, outputs)
File “/code/stages/run_opensfm.py”, line 29, in process
octx.reconstruct(self.rerun())
File “/code/opendm/osfm.py”, line 47, in reconstruct
self.run(‘reconstruct’)
File “/code/opendm/osfm.py”, line 21, in run
(context.opensfm_path, command, self.opensfm_project_path))
File “/code/opendm/system.py”, line 76, in run
raise Exception(“Child returned {}”.format(retcode))
Exception: Child returned 137

This looks like something might have built wrong, but @pierotofy would be the expert on reading this.

You ran out of memory. Either add more RAM, resize your images, or use a service such as webodm.net.

Yep, thats what i was wondering when using split-merge, is this the process that is the biggest memory hog? When it goes through the submodels, if i normally can do 400, and split to that, is there any process after the split that consumes as much memory as this?

I would prefer to use ODM, over Web.

Thanks for the help