Project fails when combining nadir and oblique datasets

I’ve successfully run the nadir and oblique sets independently using the fast-ortho, but when I combine the two and customize the settings it errors out. I’ve included text from the console report below.

settings:
[INFO] Initializing ODM 2.9.2 - Wed Nov 09 02:52:16 2022
[INFO] ==============
[INFO] 3d_tiles: False
[INFO] auto_boundary: True
[INFO] auto_boundary_distance: 0
[INFO] bg_removal: False
[INFO] boundary: {}
[INFO] build_overviews: False
[INFO] camera_lens: auto
[INFO] cameras: {}
[INFO] cog: True
[INFO] copy_to: None
[INFO] crop: 3
[INFO] debug: False
[INFO] dem_decimation: 1
[INFO] dem_euclidean_map: False
[INFO] dem_gapfill_steps: 3
[INFO] dem_resolution: 5
[INFO] depthmap_resolution: 640
[INFO] dsm: True
[INFO] dtm: False
[INFO] end_with: odm_postprocess
[INFO] fast_orthophoto: False
[INFO] feature_quality: high
[INFO] feature_type: sift
[INFO] force_gps: False
[INFO] gcp: None
[INFO] geo: None
[INFO] gps_accuracy: 10
[INFO] ignore_gsd: True
[INFO] matcher_neighbors: 0
[INFO] matcher_type: flann
[INFO] max_concurrency: 128
[INFO] merge: all
[INFO] mesh_octree_depth: 11
[INFO] mesh_size: 300000
[INFO] min_num_features: 10000
[INFO] name: cbef36be-b4be-4ec7-9d7e-f5388bcb491d
[INFO] no_gpu: False
[INFO] optimize_disk_space: False
[INFO] orthophoto_compression: DEFLATE
[INFO] orthophoto_cutline: False
[INFO] orthophoto_kmz: False
[INFO] orthophoto_no_tiled: False
[INFO] orthophoto_png: False
[INFO] orthophoto_resolution: 5
[INFO] pc_classify: False
[INFO] pc_copc: False
[INFO] pc_csv: False
[INFO] pc_ept: True
[INFO] pc_filter: 1.0
[INFO] pc_geometric: False
[INFO] pc_las: False
[INFO] pc_quality: medium
[INFO] pc_rectify: False
[INFO] pc_sample: 0
[INFO] pc_tile: False
[INFO] primary_band: auto
[INFO] project_path: /var/www/data
[INFO] radiometric_calibration: none
[INFO] rerun: None
[INFO] rerun_all: False
[INFO] rerun_from: None
[INFO] resize_to: 2048
[INFO] rolling_shutter: False
[INFO] rolling_shutter_readout: 0
[INFO] sfm_algorithm: incremental
[INFO] skip_3dmodel: False
[INFO] skip_band_alignment: False
[INFO] skip_orthophoto: False
[INFO] skip_report: False
[INFO] sky_removal: False
[INFO] sm_cluster: None
[INFO] sm_no_align: False
[INFO] smrf_scalar: 1.25
[INFO] smrf_slope: 0.15
[INFO] smrf_threshold: 0.5
[INFO] smrf_window: 18.0
[INFO] split: 999999
[INFO] split_image_groups: None
[INFO] split_overlap: 150
[INFO] texturing_data_term: gmi
[INFO] texturing_keep_unseen_faces: False
[INFO] texturing_outlier_removal_type: gauss_clamping
[INFO] texturing_skip_global_seam_leveling: False
[INFO] texturing_skip_local_seam_leveling: False
[INFO] texturing_tone_mapping: none
[INFO] tiles: False
[INFO] time: False
[INFO] use_3dmesh: False
[INFO] use_exif: False
[INFO] use_fixed_camera_params: False
[INFO] use_hybrid_bundle_adjustment: False
[INFO] verbose: False

Process (top):
[INFO] Running dataset stage
[INFO] Loading dataset from: /var/www/data/cbef36be-b4be-4ec7-9d7e-f5388bcb491d/images
[INFO] Loading 891 images
[INFO] Found 891 usable images
[INFO] Parsing SRS header: WGS84 UTM 10N
[INFO] Finished dataset stage
[INFO] Running split stage
[INFO] Normal dataset, will process all at once.
[INFO] Finished split stage
[INFO] Running merge stage
[INFO] Normal dataset, nothing to merge.
[INFO] Finished merge stage
[INFO] Running opensfm stage
[INFO] Maximum photo dimensions: 5472px
[INFO] Photo dimensions for feature extraction: 2736px
[INFO] nvidia-smi not found in PATH, using CPU
[INFO] Altitude data detected, enabling it for GPS alignment
[INFO] [‘use_exif_size: no’, ‘flann_algorithm: KDTREE’, ‘feature_process_size: 2736’, ‘feature_min_frames: 10000’, ‘processes: 128’, ‘matching_gps_neighbors: 0’, ‘matching_gps_distance: 0’, ‘matching_graph_rounds: 50’, ‘optimize_camera_parameters: yes’, ‘reconstruction_algorithm: incremental’, ‘undistorted_image_format: tif’, ‘bundle_outlier_filtering_type: AUTO’, ‘sift_peak_threshold: 0.066’, ‘align_orientation_prior: vertical’, ‘triangulation_type: ROBUST’, ‘retriangulation_ratio: 2’, ‘matcher_type: FLANN’, ‘feature_type: SIFT’, ‘use_altitude_tag: yes’, ‘align_method: auto’, ‘local_bundle_radius: 0’]
[INFO] Wrote reference_lla.json
[INFO] running “/code/SuperBuild/install/bin/opensfm/bin/opensfm” detect_features “/var/www/data/cbef36be-b4be-4ec7-9d7e-f5388bcb491d/opensfm”
2022-11-09 02:52:24,655 INFO: Planning to use 221628.50507812502 MB of RAM for both processing queue and parallel processing.
2022-11-09 02:52:24,655 INFO: Scale-space expected size of a single image : 170.66471099853516 MB
2022-11-09 02:52:24,655 INFO: Expecting to queue at most 200 images while parallel processing of 128 images.

Process (btm):
2022-11-09 09:30:25,426 DEBUG: Undistorting image DJI_0155.JPG
[INFO] running “/code/SuperBuild/install/bin/opensfm/bin/opensfm” export_visualsfm --points “/var/www/data/cbef36be-b4be-4ec7-9d7e-f5388bcb491d/opensfm”
[INFO] Finished opensfm stage
[INFO] Running openmvs stage
[INFO] running “/code/SuperBuild/install/bin/opensfm/bin/opensfm” export_openmvs “/var/www/data/cbef36be-b4be-4ec7-9d7e-f5388bcb491d/opensfm”
[INFO] Depthmap resolution set to: 684px
[INFO] Running dense reconstruction. This might take a while.
[INFO] Estimating depthmaps
[INFO] nvidia-smi not found in PATH, using CPU
[INFO] running “/code/SuperBuild/install/bin/OpenMVS/DensifyPointCloud” “/var/www/data/cbef36be-b4be-4ec7-9d7e-f5388bcb491d/opensfm/undistorted/openmvs/scene.mvs” --resolution-level 3 --dense-config-file “/var/www/data/cbef36be-b4be-4ec7-9d7e-f5388bcb491d/opensfm/undistorted/openmvs/Densify.ini” --min-resolution 684 --max-resolution 5472 --max-threads 128 --number-views-fuse 2 --sub-resolution-levels 2 -w “/var/www/data/cbef36be-b4be-4ec7-9d7e-f5388bcb491d/opensfm/undistorted/openmvs/depthmaps” -v 0 --cuda-device -2 --geometric-iters 0
09:34:32 [App ] Build date: Oct 23 2022, 00:34:16
09:34:32 [App ] CPU: AMD EPYC 7601 32-Core Processor (128 cores)
09:34:32 [App ] RAM: 251.75GB Physical Memory 2.00GB Virtual Memory
09:34:32 [App ] OS: Linux 5.15.0-52-generic (x86_64)
09:34:32 [App ] Disk:
09:34:32 [App ] SSE & AVX compatible CPU & OS detected
09:34:32 [App ] Command line: DensifyPointCloud /var/www/data/cbef36be-b4be-4ec7-9d7e-f5388bcb491d/opensfm/undistorted/openmvs/scene.mvs --resolution-level 3 --dense-config-file /var/www/data/cbef36be-b4be-4ec7-9d7e-f5388bcb491d/opensfm/undistorted/openmvs/Densify.ini --min-resolution 684 --max-resolution 5472 --max-threads 128 --number-views-fuse 2 --sub-resolution-levels 2 -w /var/www/data/cbef36be-b4be-4ec7-9d7e-f5388bcb491d/opensfm/undistorted/openmvs/depthmaps -v 0 --cuda-device -2 --geometric-iters 0
09:34:33 [App ] Find a camera not pointing towards the scene center; the scene will be considered unbounded (no ROI)
09:35:06 [App ] Preparing images for dense reconstruction completed: 891 images (33s443ms)
09:35:19 [App ] Selecting images for dense reconstruction completed: 0 images (12s936ms)

Fused depth-maps 0 (100%, 0ms)
09:35:19 [App ] Densifying point-cloud completed: 0 points (46s597ms)
09:35:19 [App ] MEMORYINFO: {
09:35:19 [App ] VmPeak: 36390128 kB
09:35:19 [App ] VmSize: 19041848 kB
09:35:19 [App ] } ENDINFO
[INFO] running “/code/SuperBuild/install/bin/OpenMVS/DensifyPointCloud” --filter-point-cloud -1 -i “/var/www/data/cbef36be-b4be-4ec7-9d7e-f5388bcb491d/opensfm/undistorted/openmvs/scene_dense.mvs” -v 0 --cuda-device -2 --geometric-iters 0
09:35:20 [App ] Build date: Oct 23 2022, 00:34:16
09:35:20 [App ] CPU: AMD EPYC 7601 32-Core Processor (128 cores)
09:35:20 [App ] RAM: 251.75GB Physical Memory 2.00GB Virtual Memory
09:35:20 [App ] OS: Linux 5.15.0-52-generic (x86_64)
09:35:20 [App ] Disk:
09:35:20 [App ] SSE & AVX compatible CPU & OS detected
09:35:20 [App ] Command line: DensifyPointCloud --filter-point-cloud -1 -i /var/www/data/cbef36be-b4be-4ec7-9d7e-f5388bcb491d/opensfm/undistorted/openmvs/scene_dense.mvs -v 0 --cuda-device -2 --geometric-iters 0
09:35:20 [App ] error: no valid point-cloud for the ROI estimation

Info for Geeks:

===== Dumping Info for Geeks (developers need this to fix bugs) =====
Child returned 1
Traceback (most recent call last):
File “/code/stages/odm_app.py”, line 88, in execute
self.first_stage.run()
File “/code/opendm/types.py”, line 378, in run
self.next_stage.run(outputs)
File “/code/opendm/types.py”, line 378, in run
self.next_stage.run(outputs)
File “/code/opendm/types.py”, line 378, in run
self.next_stage.run(outputs)
[Previous line repeated 1 more time]
File “/code/opendm/types.py”, line 359, in run
self.process(self.args, outputs)
File “/code/stages/openmvs.py”, line 209, in process
system.run(’"%s" %s’ % (context.omvs_densify_path, ’ '.join(config + gpu_config + extra_config)))
File “/code/opendm/system.py”, line 109, in run
raise SubprocessException(“Child returned {}”.format(retcode), retcode)
opendm.system.SubprocessException: Child returned 1

===== Done, human-readable information to follow… =====

[ERROR] Uh oh! Processing stopped because of strange values in the reconstruction. This is often a sign that the input data has some issues or the software cannot deal with it. Have you followed best practices for data acquisition? See Flying Tips — OpenDroneMap 2.9.1 documentation

You definitely can’t combine angles with fast orthophoto. Nadir only, really.

Try defaults and then if that works, turn up the feature-quality and pc-quality .

1 Like

I only ran fast ortho on the individual sets. Seemed to work ok with the oblique but I haven’t looked it over too closely. For the combined set, it’s turned off. This section really has me scratching my chin-

09:35:06 [App ] Preparing images for dense reconstruction completed: 891 images (33s443ms)
09:35:19 [App ] Selecting images for dense reconstruction completed: 0 images (12s936ms)

1 Like

Yeah, I’m confused about that too. But have you run either of the individual sets with defaults? Fast-orthophoto is quite a bit different as a toolchain from the usual approach.

1 Like

I haven’t. I’ll run one tonight.

1 Like

Just for sanity’s sake, unless you have just an absurd amount of RAM, please drop --ignore-gsd unless you’re 1000% sure you need it.

Missed that. He does have an insane amount of ram, but maybe not that much.

1 Like

I ran the combined set with default settings and got the same error. The only difference was it errored out after 4 hours this time compared to 6.5 hours last time. One thing to note - I restarted the same project that failed from “load data set”.

[INFO] running “/code/SuperBuild/install/bin/opensfm/bin/opensfm” export_visualsfm --points “/var/www/data/53c552a6-d2e5-4202-b5df-3818d66dbbc5/opensfm”
[INFO] Finished opensfm stage
[INFO] Running openmvs stage
[INFO] running “/code/SuperBuild/install/bin/opensfm/bin/opensfm” export_openmvs “/var/www/data/53c552a6-d2e5-4202-b5df-3818d66dbbc5/opensfm”
[INFO] Depthmap resolution set to: 684px
[INFO] Running dense reconstruction. This might take a while.
[INFO] Estimating depthmaps
[INFO] nvidia-smi detected
[INFO] running “/code/SuperBuild/install/bin/OpenMVS/DensifyPointCloud” “/var/www/data/53c552a6-d2e5-4202-b5df-3818d66dbbc5/opensfm/undistorted/openmvs/scene.mvs” --resolution-level 0 --dense-config-file “/var/www/data/53c552a6-d2e5-4202-b5df-3818d66dbbc5/opensfm/undistorted/openmvs/Densify.ini” --min-resolution 684 --max-resolution 684 --max-threads 128 --number-views-fuse 2 --sub-resolution-levels 2 -w “/var/www/data/53c552a6-d2e5-4202-b5df-3818d66dbbc5/opensfm/undistorted/openmvs/depthmaps” -v 0 --cuda-device -1 --geometric-iters 0
09:30:24 [App ] Build date: Nov 7 2022, 22:48:40
09:30:24 [App ] CPU: AMD EPYC 7601 32-Core Processor (128 cores)
09:30:24 [App ] RAM: 251.75GB Physical Memory 2.00GB Virtual Memory
09:30:24 [App ] OS: Linux 5.15.0-52-generic (x86_64)
09:30:24 [App ] Disk:
09:30:24 [App ] SSE & AVX compatible CPU & OS detected
09:30:24 [App ] Command line: DensifyPointCloud /var/www/data/53c552a6-d2e5-4202-b5df-3818d66dbbc5/opensfm/undistorted/openmvs/scene.mvs --resolution-level 0 --dense-config-file /var/www/data/53c552a6-d2e5-4202-b5df-3818d66dbbc5/opensfm/undistorted/openmvs/Densify.ini --min-resolution 684 --max-resolution 684 --max-threads 128 --number-views-fuse 2 --sub-resolution-levels 2 -w /var/www/data/53c552a6-d2e5-4202-b5df-3818d66dbbc5/opensfm/undistorted/openmvs/depthmaps -v 0 --cuda-device -1 --geometric-iters 0
09:30:24 [App ] MapSMtoCores for SM 8.6 is undefined; default to use 64 cores/SM
09:30:25 [App ] Find a camera not pointing towards the scene center; the scene will be considered unbounded (no ROI)
09:30:25 [App ] Preparing images for dense reconstruction completed: 891 images (400ms)
09:30:31 [App ] Selecting images for dense reconstruction completed: 0 images (6s187ms)

Fused depth-maps 0 (100%, 0ms)
09:30:31 [App ] Densifying point-cloud completed: 0 points (6s718ms)
09:30:31 [App ] MEMORYINFO: {
09:30:31 [App ] VmPeak: 14816832 kB
09:30:31 [App ] VmSize: 14751296 kB
09:30:31 [App ] } ENDINFO
[INFO] running “/code/SuperBuild/install/bin/OpenMVS/DensifyPointCloud” --filter-point-cloud -1 -i “/var/www/data/53c552a6-d2e5-4202-b5df-3818d66dbbc5/opensfm/undistorted/openmvs/scene_dense.mvs” -v 0 --cuda-device -1 --geometric-iters 0
09:30:32 [App ] Build date: Nov 7 2022, 22:48:40
09:30:32 [App ] CPU: AMD EPYC 7601 32-Core Processor (128 cores)
09:30:32 [App ] RAM: 251.75GB Physical Memory 2.00GB Virtual Memory
09:30:32 [App ] OS: Linux 5.15.0-52-generic (x86_64)
09:30:32 [App ] Disk:
09:30:32 [App ] SSE & AVX compatible CPU & OS detected
09:30:32 [App ] Command line: DensifyPointCloud --filter-point-cloud -1 -i /var/www/data/53c552a6-d2e5-4202-b5df-3818d66dbbc5/opensfm/undistorted/openmvs/scene_dense.mvs -v 0 --cuda-device -1 --geometric-iters 0
09:30:32 [App ] MapSMtoCores for SM 8.6 is undefined; default to use 64 cores/SM
09:30:32 [App ] error: no valid point-cloud for the ROI estimation

===== Dumping Info for Geeks (developers need this to fix bugs) =====
Child returned 1
Traceback (most recent call last):
File “/code/stages/odm_app.py”, line 88, in execute
self.first_stage.run()
File “/code/opendm/types.py”, line 380, in run
self.next_stage.run(outputs)
File “/code/opendm/types.py”, line 380, in run
self.next_stage.run(outputs)
File “/code/opendm/types.py”, line 380, in run
self.next_stage.run(outputs)
[Previous line repeated 1 more time]
File “/code/opendm/types.py”, line 361, in run
self.process(self.args, outputs)
File “/code/stages/openmvs.py”, line 226, in process
raise e
File “/code/stages/openmvs.py”, line 220, in process
system.run(’"%s" %s’ % (context.omvs_densify_path, ’ '.join(config + gpu_config + extra_config)))
File “/code/opendm/system.py”, line 109, in run
raise SubprocessException(“Child returned {}”.format(retcode), retcode)
opendm.system.SubprocessException: Child returned 1

===== Done, human-readable information to follow… =====

[ERROR] Uh oh! Processing stopped because of strange values in the reconstruction. This is often a sign that the input data has some issues or the software cannot deal with it. Have you followed best practices for data acquisition? See Flying Tips — OpenDroneMap 2.9.1 documentation

I’m running the nadir set using the default settings now. I previously ran the nadir set using fast ortho & no gpu and it took 1:07 mins. This time I’m using default settings with gpu. If anything, this will be an interesting comparison.

Thanks Saijin and Smather for troubleshooting with me!

Any insight into this line?

09:30:32 [App ] Disk:

Should “Disk” be blank?

Maybe you need to increase virtual memory initial size significantly, and make it equal to max size?

1 Like

Sure, I’ll give it a shot. How do I set the initial size? I’m brand new to Ubuntu. Also, I noticed this warning upon WebODM start-up - “overcommit_memory is set to 0”. I don’t see a sysctl.conf file in the etc directory…

Update on the nadir set:

It ran fine with the default settings.

One difference between this finished project and the failed one (bold):

[INFO] running “/code/SuperBuild/install/bin/opensfm/bin/opensfm” export_visualsfm --points “/var/www/data/d2762d03-2214-42d2-a0ea-a2c3550b164f/opensfm”
[INFO] Finished opensfm stage
[INFO] Running openmvs stage
[INFO] running “/code/SuperBuild/install/bin/opensfm/bin/opensfm” export_openmvs “/var/www/data/d2762d03-2214-42d2-a0ea-a2c3550b164f/opensfm”
[INFO] Depthmap resolution set to: 684px
[INFO] Running dense reconstruction. This might take a while.
[INFO] Estimating depthmaps
[INFO] nvidia-smi detected
[INFO] running “/code/SuperBuild/install/bin/OpenMVS/DensifyPointCloud” “/var/www/data/d2762d03-2214-42d2-a0ea-a2c3550b164f/opensfm/undistorted/openmvs/scene.mvs” --resolution-level 1 --dense-config-file “/var/www/data/d2762d03-2214-42d2-a0ea-a2c3550b164f/opensfm/undistorted/openmvs/Densify.ini” --min-resolution 684 --max-resolution 1741 --max-threads 128 --number-views-fuse 2 --sub-resolution-levels 2 -w “/var/www/data/d2762d03-2214-42d2-a0ea-a2c3550b164f/opensfm/undistorted/openmvs/depthmaps” -v 0 --cuda-device -1 --geometric-iters 0
18:16:20 [App ] Build date: Nov 7 2022, 22:48:40
18:16:20 [App ] CPU: AMD EPYC 7601 32-Core Processor (128 cores)
18:16:20 [App ] RAM: 251.75GB Physical Memory 2.00GB Virtual Memory
18:16:20 [App ] OS: Linux 5.15.0-52-generic (x86_64)
18:16:20 [App ] Disk:
18:16:20 [App ] SSE & AVX compatible CPU & OS detected
18:16:20 [App ] Command line: DensifyPointCloud /var/www/data/d2762d03-2214-42d2-a0ea-a2c3550b164f/opensfm/undistorted/openmvs/scene.mvs --resolution-level 1 --dense-config-file /var/www/data/d2762d03-2214-42d2-a0ea-a2c3550b164f/opensfm/undistorted/openmvs/Densify.ini --min-resolution 684 --max-resolution 1741 --max-threads 128 --number-views-fuse 2 --sub-resolution-levels 2 -w /var/www/data/d2762d03-2214-42d2-a0ea-a2c3550b164f/opensfm/undistorted/openmvs/depthmaps -v 0 --cuda-device -1 --geometric-iters 0
18:16:20 [App ] MapSMtoCores for SM 8.6 is undefined; default to use 64 cores/SM
18:16:21 [App ] The camera directions mean is unbalanced; the scene will be considered unbounded (no ROI)
18:16:22 [App ] Preparing images for dense reconstruction completed: 294 images (1s79ms)
18:16:24 [App ] Selecting images for dense reconstruction completed: 294 images (2s723ms)

1 Like

Just noticed “resolution-level 0” from the failed project. That’s different.

1 Like

Have you run the non-combined duo each with defaults?

1 Like

I ran the nadir this morning, went fine. Running the oblique now. I’ll report back shortly.

2 Likes

I don’t know how to set it in Unbuntu, but for docker I think it needs to be in a .wslconfig file
It needs to be on a fast drive, and perhaps equal to your RAM, plus set min = max size

1 Like

Update - the oblique set failed with default settings.

22:09:30 [App ] Preparing images for dense reconstruction completed: 597 images (284ms)
22:09:34 [App ] Selecting images for dense reconstruction completed: 0 images (3s582ms)

Potential problems:

2022-11-11 19:35:19,443 INFO: Altitude is negative (-256.67766603640365) : viewing directions are probably divergent. Using default altitude of 1.0

[WARNING] Negative GSD estimated, this might indicate a flipped Z-axis.

These flights were flown sequentially using a P4 and Pix4D capture. The only difference was changing the camera angle on the last 2 flights from within Pix4D capture to 55 deg (from the horizon).

1 Like

What are the overlap/sidelap for the obliques? It is possible we’re not getting good reconstruction so it is throwing everything out.

I didn’t change the overlap from the nadir set, so, if I remember correctly, 90 / 75. The altitude was the same as well.

1 Like

Best results thus far for the oblique set. Any thoughts on how I could clean this up?

settings:

1 Like

Can your machine handle --feature-quality ultra, --pc-quality ultra, and upping --matcher-neighbors 32?

Also, do you have an idea of your mean GPS accuracy for this collect? Passing it to --gps-accuracy might help constrain things a bit more.

Finally, trying it with --rolling-shutter might help even more.

1 Like