Issue with image_groups to split

Hello,

I’m having trouble splitting work between nodes with no-GPS info images. I’m using ClusterODM, with some nodes that appears correctly connected. Docker desktop on Win10. The dataset consists of 283 images, and one image_groups.txt file in the project folder.

I used --split-overlap 0 as specified by doc for local split merge (Splitting Large Datasets — OpenDroneMap 2.4.7 documentation).

This command worked, but I assume there is no split since --split is not used (I didn’t see any queue on any of the connected nodes too ? I didn’t watched during the whole process through):

docker run -ti --rm -v /c/datasets:/datasets opendronemap/odm --project-path /datasets test --split-overlap 0 --sm-cluster http://192.168.56.1:3000/ --split-image-groups /c/datasets/test/image_groups.txt

But when I’m trying to provide --split argument with :

docker run -ti --rm -v /c/datasets:/datasets opendronemap/odm --project-path /datasets test_split_bis --split 100 --split-overlap 0 --sm-cluster http://192.168.56.1:3000/ --split-image-groups /c/datasets/test_split_bis/image_groups.txt

(all image_groups have less than 100 images each)

It outputs the following :

[INFO] Initializing ODM - Thu Apr 22 02:06:38 2021
[INFO] ==============
[INFO] build_overviews: False
[INFO] camera_lens: auto
[INFO] cameras: {}
[INFO] crop: 3
[INFO] debug: False
[INFO] dem_decimation: 1
[INFO] dem_euclidean_map: False
[INFO] dem_gapfill_steps: 3
[INFO] dem_resolution: 5
[INFO] depthmap_resolution: 640
[INFO] dsm: False
[INFO] dtm: False
[INFO] end_with: odm_report
[INFO] fast_orthophoto: False
[INFO] feature_quality: high
[INFO] feature_type: sift
[INFO] force_gps: False
[INFO] gcp: None
[INFO] geo: None
[INFO] gps_accuracy: 10
[INFO] ignore_gsd: False
[INFO] matcher_distance: 0
[INFO] matcher_neighbors: 8
[INFO] matcher_type: flann
[INFO] max_concurrency: 18
[INFO] merge: all
[INFO] mesh_octree_depth: 11
[INFO] mesh_size: 200000
[INFO] min_num_features: 8000
[INFO] name: test_split_bis
[INFO] optimize_disk_space: False
[INFO] orthophoto_compression: DEFLATE
[INFO] orthophoto_cutline: False
[INFO] orthophoto_kmz: False
[INFO] orthophoto_no_tiled: False
[INFO] orthophoto_png: False
[INFO] orthophoto_resolution: 5
[INFO] pc_classify: False
[INFO] pc_csv: False
[INFO] pc_ept: False
[INFO] pc_filter: 2.5
[INFO] pc_las: False
[INFO] pc_quality: medium
[INFO] pc_rectify: False
[INFO] pc_sample: 0
[INFO] pc_tile: False
[INFO] primary_band: auto
[INFO] project_path: /datasets
[INFO] radiometric_calibration: none
[INFO] rerun: None
[INFO] rerun_all: False
[INFO] rerun_from: None
[INFO] resize_to: 2048
[INFO] skip_3dmodel: False
[INFO] skip_band_alignment: False
[INFO] skip_report: False
[INFO] sm_cluster: True
[INFO] smrf_scalar: 1.25
[INFO] smrf_slope: 0.15
[INFO] smrf_threshold: 0.5
[INFO] smrf_window: 18.0
[INFO] split: 100
[INFO] split_image_groups: /c/datasets/test_split_bis/image_groups.txt
[INFO] split_overlap: 0.0
[INFO] texturing_data_term: gmi
[INFO] texturing_keep_unseen_faces: False
[INFO] texturing_outlier_removal_type: gauss_clamping
[INFO] texturing_skip_global_seam_leveling: False
[INFO] texturing_skip_local_seam_leveling: False
[INFO] texturing_tone_mapping: none
[INFO] tiles: False
[INFO] time: False
[INFO] use_3dmesh: False
[INFO] use_exif: False
[INFO] use_fixed_camera_params: False
[INFO] use_hybrid_bundle_adjustment: False
[INFO] verbose: False
[INFO] ==============
[INFO] Running dataset stage
[INFO] Loading dataset from: /datasets/test_split_bis/images
[INFO] Loading 283 images
[INFO] Wrote images database: /datasets/test_split_bis/images.json
[INFO] Found 283 usable images
[WARNING] GPS position not available for a-10.png
[WARNING] GPS position not available for a-11.png
//continue for each image
[WARNING] GPS position not available for d-8.png
[WARNING] GPS position not available for d-9.png
[WARNING] Could not generate coordinates file. The orthophoto will not be georeferenced.
[INFO] Finished dataset stage
[INFO] Running split stage
[INFO] Setting max-concurrency to 17 to better handle remote splits
[INFO] Large dataset detected (283 photos) and split set at 100. Preparing split merge.
[INFO] Writing exif overrides
[INFO] Maximum photo dimensions: 1920px
[INFO] No GPS information, using BOW matching
[WARNING] Using BOW matching, will use HAHOG feature type, not SIFT
[INFO] Not using the GPU
[INFO] [‘use_exif_size: no’, ‘flann_algorithm: KDTREE’, ‘feature_process_size: 960’, ‘feature_min_frames: 8000’, ‘processes: 17’, ‘matching_gps_neighbors: 8’, ‘matching_gps_distance: 0’, ‘optimize_camera_parameters: yes’, ‘undistorted_image_format: tif’, ‘bundle_outlier_filtering_type: AUTO’, ‘align_orientation_prior: vertical’, ‘triangulation_type: ROBUST’, ‘retriangulation_ratio: 2’, ‘matcher_type: WORDS’, ‘feature_type: HAHOG’, ‘align_method: orientation_prior’, ‘local_bundle_radius: 0’, ‘submodels_relpath: …/submodels/opensfm’, ‘submodel_relpath_template: …/submodels/submodel_%04d/opensfm’, ‘submodel_images_relpath_template: …/submodels/submodel_%04d/images’, ‘submodel_size: 100’, ‘submodel_overlap: 0.0’]
[INFO] running /code/SuperBuild/src/opensfm/bin/opensfm extract_metadata “/datasets/test_split_bis/opensfm”
[INFO] running /code/SuperBuild/src/opensfm/bin/opensfm create_submodels “/datasets/test_split_bis/opensfm”
WARNING:opensfm.actions.create_submodels:Skipping a-10.png because of missing GPS
WARNING:opensfm.actions.create_submodels:Skipping a-11.png because of missing GPS
//continue for each image
WARNING:opensfm.actions.create_submodels:Skipping d-8.png because of missing GPS
WARNING:opensfm.actions.create_submodels:Skipping d-9.png because of missing GPS
Traceback (most recent call last):
File “/code/SuperBuild/src/opensfm/bin/opensfm_main.py”, line 15, in
commands.command_runner(commands.opensfm_commands, default_dataset_type)
File “/code/SuperBuild/src/opensfm/opensfm/commands/command_runner.py”, line 31, in command_runner
command.run(data, args)
File “/code/SuperBuild/src/opensfm/opensfm/commands/command.py”, line 12, in run
self.run_impl(data, args)
File “/code/SuperBuild/src/opensfm/opensfm/commands/create_submodels.py”, line 11, in run_impl
create_submodels.run_dataset(dataset)
File “/code/SuperBuild/src/opensfm/opensfm/actions/create_submodels.py”, line 24, in run_dataset
_cluster_images(meta_data, data.config[“submodel_size”])
File “/code/SuperBuild/src/opensfm/opensfm/actions/create_submodels.py”, line 97, in _cluster_images
labels, centers = tools.kmeans(positions, K)[1:]
File “/code/SuperBuild/src/opensfm/opensfm/large/tools.py”, line 35, in kmeans
return cv2.kmeans(samples, nclusters, None, criteria, attempts, flags)
cv2.error: OpenCV(4.5.0) /code/SuperBuild/src/opencv/modules/core/src/kmeans.cpp:242: error: (-215:Assertion failed) data0.dims <= 2 && type == CV_32F && K > 0 in function ‘kmeans’

===== Dumping Info for Geeks (developers need this to fix bugs) =====
Child returned 1
Traceback (most recent call last):
File “/code/stages/odm_app.py”, line 83, in execute
self.first_stage.run()
File “/code/opendm/types.py”, line 338, in run
self.next_stage.run(outputs)
File “/code/opendm/types.py”, line 319, in run
self.process(self.args, outputs)
File “/code/stages/splitmerge.py”, line 69, in process
octx.run(“create_submodels”)
File “/code/opendm/osfm.py”, line 27, in run
system.run(’%s/bin/opensfm %s “%s”’ %
File “/code/opendm/system.py”, line 82, in run
raise SubprocessException(“Child returned {}”.format(retcode), retcode)
opendm.system.SubprocessException: Child returned 1
===== Done, human-readable information to follow… =====

[ERROR] Uh oh! Processing stopped because of strange values in the reconstruction. This is often a sign that the input data has some issues or the software cannot deal with it. Have you followed best practices for data acquisition? See Flying Tips — OpenDroneMap 2.4.7 documentation

I assumed it was not an issue with the data since it worked without split with the same data, and image_groups was provided as specified for no-GPS data.

1 Like

This is interesting as I am seeing something somewhat similar with a dataset with split merge that contains GPS:

[INFO] Initializing ODM - Wed Apr 21 16:41:16 2021
[INFO] ==============
[INFO] build_overviews: False
[INFO] camera_lens: brown
[INFO] cameras: {}
[INFO] crop: 3
[INFO] debug: False
[INFO] dem_decimation: 1
[INFO] dem_euclidean_map: True
[INFO] dem_gapfill_steps: 3
[INFO] dem_resolution: 5
[INFO] depthmap_resolution: 640
[INFO] dsm: False
[INFO] dtm: False
[INFO] end_with: odm_report
[INFO] fast_orthophoto: False
[INFO] feature_quality: high
[INFO] feature_type: sift
[INFO] force_gps: False
[INFO] gcp: None
[INFO] geo: None
[INFO] gps_accuracy: 10
[INFO] ignore_gsd: False
[INFO] matcher_distance: 0
[INFO] matcher_neighbors: 12
[INFO] matcher_type: flann
[INFO] max_concurrency: 19
[INFO] merge: all
[INFO] mesh_octree_depth: 11
[INFO] mesh_size: 200000
[INFO] min_num_features: 8000
[INFO] name: 7b5e9678-dc95-485d-aea2-a3c1a6a1e2c8
[INFO] optimize_disk_space: False
[INFO] orthophoto_compression: DEFLATE
[INFO] orthophoto_cutline: True
[INFO] orthophoto_no_tiled: False
[INFO] orthophoto_png: False
[INFO] orthophoto_resolution: 5
[INFO] pc_classify: False
[INFO] pc_csv: False
[INFO] pc_ept: False
[INFO] pc_filter: 2.5
[INFO] pc_las: False
[INFO] pc_quality: high
[INFO] pc_rectify: False
[INFO] pc_sample: 0
[INFO] pc_tile: False
[INFO] primary_band: auto
[INFO] project_path: /var/www/data
[INFO] radiometric_calibration: none
[INFO] rerun: None
[INFO] rerun_all: False
[INFO] rerun_from: None
[INFO] resize_to: 2048
[INFO] skip_3dmodel: True
[INFO] skip_band_alignment: False
[INFO] skip_report: True
[INFO] sm_cluster: None
[INFO] smrf_scalar: 1.25
[INFO] smrf_slope: 0.15
[INFO] smrf_threshold: 0.5
[INFO] smrf_window: 18.0
[INFO] split: 999999
[INFO] split_image_groups: None
[INFO] split_overlap: 150
[INFO] texturing_data_term: gmi
[INFO] texturing_keep_unseen_faces: False
[INFO] texturing_outlier_removal_type: gauss_clamping
[INFO] texturing_skip_global_seam_leveling: False
[INFO] texturing_skip_local_seam_leveling: False
[INFO] texturing_tone_mapping: none
[INFO] tiles: False
[INFO] time: False
[INFO] use_3dmesh: False
[INFO] use_exif: False
[INFO] use_fixed_camera_params: False
[INFO] use_hybrid_bundle_adjustment: True
[INFO] verbose: True
[INFO] ==============
[INFO] Running dataset stage
[INFO] Loading dataset from: /var/www/data/7b5e9678-dc95-485d-aea2-a3c1a6a1e2c8/images
[INFO] Loading 1977 images
[INFO] Wrote images database: /var/www/data/7b5e9678-dc95-485d-aea2-a3c1a6a1e2c8/images.json
[INFO] Found 1977 usable images
[INFO] Parsing SRS header: WGS84 UTM 37S
[INFO] Finished dataset stage
[INFO] Running split stage
[INFO] Normal dataset, will process all at once.
[INFO] Finished split stage
[INFO] Running merge stage
[INFO] Normal dataset, nothing to merge.
[INFO] Finished merge stage
[INFO] Running opensfm stage
[INFO] Writing exif overrides
[INFO] Maximum photo dimensions: 5472px
[INFO] PyOpenCL is missing (not a GPU build)
[INFO] Altitude data detected, enabling it for GPS alignment
[INFO] Enabling hybrid bundle adjustment
[INFO] ['use_exif_size: no', 'flann_algorithm: KDTREE', 'feature_process_size: 2736', 'feature_min_frames: 8000', 'processes: 19', 'matching_gps_neighbors: 12', 'matching_gps_distance: 0', 'optimize_camera_parameters: yes', 'undistorted_image_format: tif', 'bundle_outlier_filtering_type: AUTO', 'align_orientation_prior: vertical', 'triangulation_type: ROBUST', 'retriangulation_ratio: 2', 'camera_projection_type: BROWN', 'feature_type: SIFT', 'use_altitude_tag: yes', 'align_method: auto', 'bundle_interval: 100', 'bundle_new_points_ratio: 1.2', 'local_bundle_radius: 1']
[WARNING] Detect features already done: /var/www/data/7b5e9678-dc95-485d-aea2-a3c1a6a1e2c8/opensfm/features exists
[WARNING] Match features already done: /var/www/data/7b5e9678-dc95-485d-aea2-a3c1a6a1e2c8/opensfm/matches exists
[WARNING] Found a valid OpenSfM tracks file in: /var/www/data/7b5e9678-dc95-485d-aea2-a3c1a6a1e2c8/opensfm/tracks.csv
[WARNING] Found a valid OpenSfM reconstruction file in: /var/www/data/7b5e9678-dc95-485d-aea2-a3c1a6a1e2c8/opensfm/reconstruction.json
[WARNING] Cannot export cameras to /var/www/data/7b5e9678-dc95-485d-aea2-a3c1a6a1e2c8/cameras.json. Expecting ',' delimiter: line 689099 column 40 (char 19611971).
[INFO] running /code/SuperBuild/src/opensfm/bin/opensfm export_geocoords --reconstruction --proj '+proj=utm +zone=37 +south +datum=WGS84 +units=m +no_defs +type=crs' --offset-x 524915.0 --offset-y 9314675.0 "/var/www/data/7b5e9678-dc95-485d-aea2-a3c1a6a1e2c8/opensfm"
Traceback (most recent call last):
File "/code/SuperBuild/src/opensfm/bin/opensfm_main.py", line 15, in
commands.command_runner(commands.opensfm_commands, default_dataset_type)
File "/code/SuperBuild/src/opensfm/opensfm/commands/command_runner.py", line 31, in command_runner
command.run(data, args)
File "/code/SuperBuild/src/opensfm/opensfm/commands/command.py", line 12, in run
self.run_impl(data, args)
File "/code/SuperBuild/src/opensfm/opensfm/commands/export_geocoords.py", line 11, in run_impl
export_geocoords.run_dataset(
File "/code/SuperBuild/src/opensfm/opensfm/actions/export_geocoords.py", line 50, in run_dataset
reconstructions = data.load_reconstruction()
File "/code/SuperBuild/src/opensfm/opensfm/dataset.py", line 434, in load_reconstruction
reconstructions = io.reconstructions_from_json(io.json_load(fin))
File "/code/SuperBuild/src/opensfm/opensfm/io.py", line 737, in json_load
return json.load(fp)
File "/usr/lib/python3.8/json/__init__.py", line 293, in load
return loads(fp.read(),
File "/usr/lib/python3.8/json/__init__.py", line 357, in loads
return _default_decoder.decode(s)
File "/usr/lib/python3.8/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python3.8/json/decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 689099 column 40 (char 19611971)
===== Dumping Info for Geeks (developers need this to fix bugs) =====
Child returned 1
===== Done, human-readable information to follow... =====

[ERROR] Uh oh! Processing stopped because of strange values in the reconstruction. This is often a sign that the input data has some issues or the software cannot deal with it. Have you followed best practices for data acquisition? See https://docs.opendronemap.org/flying.html
Traceback (most recent call last):
File "/code/stages/odm_app.py", line 83, in execute
self.first_stage.run()
File "/code/opendm/types.py", line 338, in run
self.next_stage.run(outputs)
File "/code/opendm/types.py", line 338, in run
self.next_stage.run(outputs)
File "/code/opendm/types.py", line 338, in run
self.next_stage.run(outputs)
File "/code/opendm/types.py", line 319, in run
self.process(self.args, outputs)
File "/code/stages/run_opensfm.py", line 75, in process
octx.run('export_geocoords --reconstruction --proj \'%s\' --offset-x %s --offset-y %s' %
File "/code/opendm/osfm.py", line 27, in run
system.run('%s/bin/opensfm %s "%s"' %
File "/code/opendm/system.py", line 82, in run
raise SubprocessException("Child returned {}".format(retcode), retcode)
opendm.system.SubprocessException: Child returned 1
1 Like

I did some more testing, I noted that when --split-image-groups is used, image_groups isn’t copied to opensfm (even with image_groups being in the project folder).

The error is different when not using --split-image-groups (so image_groups.txt is correctly copied to opensfm). With the following command :

docker run -ti --rm -v /c/datasets:/datasets opendronemap/odm --project-path /datasets test --split 20 --split-overlap 0 --sm-cluster http://172.17.0.3:3000/ --matcher-neighbors 0 --matcher-type bow --feature-type hahog

// all the WARNING:opensfm.actions.create_submodels:Skipping *.png because of missing GPS
WARNING:opensfm.actions.create_submodels:Skipping d9.png because of missing GPS
/code/SuperBuild/src/opensfm/opensfm/actions/create_submodels.py:79: RuntimeWarning: invalid value encountered in true_divide
centers /= centers_count
/usr/local/lib/python3.8/dist-packages/numpy/core/fromnumeric.py:3372: RuntimeWarning: Mean of empty slice.
return _methods._mean(a, axis=axis, dtype=dtype,
/usr/local/lib/python3.8/dist-packages/numpy/core/_methods.py:170: RuntimeWarning: invalid value encountered in true_divide
ret = ret.dtype.type(ret / rcount)
Traceback (most recent call last):
File “/code/SuperBuild/src/opensfm/bin/opensfm_main.py”, line 15, in
commands.command_runner(commands.opensfm_commands, default_dataset_type)
File “/code/SuperBuild/src/opensfm/opensfm/commands/command_runner.py”, line 31, in command_runner
command.run(data, args)
File “/code/SuperBuild/src/opensfm/opensfm/commands/command.py”, line 12, in run
self.run_impl(data, args)
File “/code/SuperBuild/src/opensfm/opensfm/commands/create_submodels.py”, line 11, in run_impl
create_submodels.run_dataset(dataset)
File “/code/SuperBuild/src/opensfm/opensfm/actions/create_submodels.py”, line 26, in run_dataset
_add_cluster_neighbors(meta_data, data.config[“submodel_overlap”])
File “/code/SuperBuild/src/opensfm/opensfm/actions/create_submodels.py”, line 107, in _add_cluster_neighbors
clusters = tools.add_cluster_neighbors(positions, labels, centers, max_distance)
File “/code/SuperBuild/src/opensfm/opensfm/large/tools.py”, line 42, in add_cluster_neighbors
reference = geo.TopocentricConverter(reflla[0], reflla[1], 0)
IndexError: invalid index to scalar variable.

===== Dumping Info for Geeks (developers need this to fix bugs) =====
Child returned 1
Traceback (most recent call last):
File “/code/stages/odm_app.py”, line 83, in execute
self.first_stage.run()
File “/code/opendm/types.py”, line 338, in run
self.next_stage.run(outputs)
File “/code/opendm/types.py”, line 319, in run
self.process(self.args, outputs)
File “/code/stages/splitmerge.py”, line 69, in process
octx.run(“create_submodels”)
File “/code/opendm/osfm.py”, line 27, in run
system.run(’%s/bin/opensfm %s “%s”’ %
File “/code/opendm/system.py”, line 82, in run
raise SubprocessException(“Child returned {}”.format(retcode), retcode)
opendm.system.SubprocessException: Child returned 1
===== Done, human-readable information to follow… =====

[ERROR] Uh oh! Processing stopped because of strange values in the reconstruction. This is often a sign that the input data has some issues or the software cannot deal with it. Have you followed best practices for data acquisition? See Flying Tips — OpenDroneMap 2.4.7 documentation

1 Like