Memory issue?

beedawg · 9 August 2021 13:53

Hi,
I’m having trouble densifying my point cloud, generated from around 550 photos with no GCP files. I think I’m out of memory on the docker container:

…VmPeak: 2358236 kB
…VmSize: 2268088 kB

I’m struggling to see how to increase memory for the containers. I’ve attempted to use the following command: docker update -m 100g opendronemap/odm:gpu but no docker image can be found. I think this is because I only run the container each time the command is called.

As you can see my kernel does not support the memory limit argument for docker:
–memory=“100g”

I’ve that my system grub config needs to be altered to enable this however I’d prefer not to do this as I’m working on a remote machine via ssh and it’s a pain to access it physically should something go wrong.

So my questions are:

Am I right in thinking it’s a memory issue?
If so, how would I go about fixing it?

docker run --memory=“100g” -ti --rm -v /media/mal/working_files/odm_data:/odm_data --gpus all opendronemap/odm:gpu --project-path /odm_data $ODMPROJ --dsm --skip-3dmodel --feature-quality high --mesh-octree-depth 12 --min-num-features 10000 --pc-quality medium;
WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap.
[INFO] Initializing ODM - Mon Aug 09 13:43:47 2021
[INFO] ==============
[INFO] build_overviews: False
[INFO] camera_lens: auto
[INFO] cameras: {}
[INFO] crop: 3
[INFO] debug: False
[INFO] dem_decimation: 1
[INFO] dem_euclidean_map: False
[INFO] dem_gapfill_steps: 3
[INFO] dem_resolution: 5
[INFO] depthmap_resolution: 640
[INFO] dsm: True
[INFO] dtm: False
[INFO] end_with: odm_report
[INFO] fast_orthophoto: False
[INFO] feature_quality: high
[INFO] feature_type: sift
[INFO] force_gps: False
[INFO] gcp: None
[INFO] geo: None
[INFO] gps_accuracy: 10
[INFO] ignore_gsd: False
[INFO] matcher_distance: 0
[INFO] matcher_neighbors: 8
[INFO] matcher_type: flann
[INFO] max_concurrency: 8
[INFO] merge: all
[INFO] mesh_octree_depth: 12
[INFO] mesh_size: 200000
[INFO] min_num_features: 10000
[INFO] name: Pen-y-banc
[INFO] optimize_disk_space: False
[INFO] orthophoto_compression: DEFLATE
[INFO] orthophoto_cutline: False
[INFO] orthophoto_kmz: False
[INFO] orthophoto_no_tiled: False
[INFO] orthophoto_png: False
[INFO] orthophoto_resolution: 5
[INFO] pc_classify: False
[INFO] pc_csv: False
[INFO] pc_ept: False
[INFO] pc_filter: 2.5
[INFO] pc_las: False
[INFO] pc_quality: medium
[INFO] pc_rectify: False
[INFO] pc_sample: 0
[INFO] pc_tile: False
[INFO] primary_band: auto
[INFO] project_path: /odm_data
[INFO] radiometric_calibration: none
[INFO] rerun: None
[INFO] rerun_all: False
[INFO] rerun_from: None
[INFO] resize_to: 2048
[INFO] skip_3dmodel: True
[INFO] skip_band_alignment: False
[INFO] skip_report: False
[INFO] sm_cluster: None
[INFO] smrf_scalar: 1.25
[INFO] smrf_slope: 0.15
[INFO] smrf_threshold: 0.5
[INFO] smrf_window: 18.0
[INFO] split: 999999
[INFO] split_image_groups: None
[INFO] split_overlap: 150
[INFO] texturing_data_term: gmi
[INFO] texturing_keep_unseen_faces: False
[INFO] texturing_outlier_removal_type: gauss_clamping
[INFO] texturing_skip_global_seam_leveling: False
[INFO] texturing_skip_local_seam_leveling: False
[INFO] texturing_tone_mapping: none
[INFO] tiles: False
[INFO] time: False
[INFO] use_3dmesh: False
[INFO] use_exif: False
[INFO] use_fixed_camera_params: False
[INFO] use_hybrid_bundle_adjustment: False
[INFO] verbose: False
[INFO] ==============
[INFO] Running dataset stage
[INFO] Loading dataset from: /odm_data/Pen-y-banc/images
[INFO] Loading images database: /odm_data/Pen-y-banc/images.json
[INFO] Found 594 usable images
[INFO] Coordinates file already exist: /odm_data/Pen-y-banc/odm_georeferencing/coords.txt
[INFO] Model geo file already exist: /odm_data/Pen-y-banc/odm_georeferencing/odm_georeferencing_model_geo.txt
[INFO] Parsing SRS header: WGS84 UTM 30N
[INFO] Finished dataset stage
[INFO] Running split stage
[INFO] Normal dataset, will process all at once.
[INFO] Finished split stage
[INFO] Running merge stage
[INFO] Normal dataset, nothing to merge.
[INFO] Finished merge stage
[INFO] Running opensfm stage
[WARNING] /odm_data/Pen-y-banc/opensfm/image_list.txt already exists, not rerunning OpenSfM setup
[WARNING] Detect features already done: /odm_data/Pen-y-banc/opensfm/features exists
[WARNING] Match features already done: /odm_data/Pen-y-banc/opensfm/matches exists
[WARNING] Found a valid OpenSfM tracks file in: /odm_data/Pen-y-banc/opensfm/tracks.csv
[WARNING] Found a valid OpenSfM reconstruction file in: /odm_data/Pen-y-banc/opensfm/reconstruction.json
[INFO] Already extracted cameras
[INFO] Export reconstruction stats
[WARNING] Found existing reconstruction stats /odm_data/Pen-y-banc/opensfm/stats/stats.json
[WARNING] Will skip exporting /odm_data/Pen-y-banc/opensfm/reconstruction.geocoords.json
[INFO] Undistorting /odm_data/Pen-y-banc/opensfm …
[WARNING] Already undistorted (nominal)
[WARNING] Found a valid OpenSfM NVM reconstruction file in: /odm_data/Pen-y-banc/opensfm/undistorted/reconstruction.nvm
[INFO] Finished opensfm stage
[INFO] Running openmvs stage
[WARNING] Found existing /odm_data/Pen-y-banc/opensfm/undistorted/openmvs/scene.mvs
[INFO] Running dense reconstruction. This might take a while.
[INFO] Estimating depthmaps
[INFO] running /code/SuperBuild/install/bin/OpenMVS/DensifyPointCloud “/odm_data/Pen-y-banc/opensfm/undistorted/openmvs/scene.mvs” --resolution-level 2 --min-resolution 684 --max-resolution 2093 --max-threads 8 --number-views-fuse 2 -w “/odm_data/Pen-y-banc/opensfm/undistorted/openmvs/depthmaps” -v 0
13:43:52 [App ] Build date: May 25 2021, 16:11:57
13:43:52 [App ] CPU: Intel(R) Core™ i7-7700K CPU @ 4.20GHz (8 cores)
13:43:52 [App ] RAM: 62.70GB Physical Memory 2.00GB Virtual Memory
13:43:52 [App ] OS: Linux 5.4.0-80-generic (x86_64)
13:43:52 [App ] SSE & AVX compatible CPU & OS detected
13:43:52 [App ] Command line: /odm_data/Pen-y-banc/opensfm/undistorted/openmvs/scene.mvs --resolution-level 2 --min-resolution 684 --max-resolution 2093 --max-threads 8 --number-views-fuse 2 -w /odm_data/Pen-y-banc/opensfm/undistorted/openmvs/depthmaps -v 0
13:43:58 [App ] Preparing images for dense reconstruction completed: 594 images (4s969ms)
13:44:21 [App ] Selecting images for dense reconstruction completed: 0 images (23s771ms)
Fused depth-maps 0 (100%, 0ms)
13:44:21 [App ] Densifying point-cloud completed: 0 points (28s809ms)
13:44:21 [App ] MEMORYINFO: {
13:44:21 [App ] VmPeak: 2358236 kB
13:44:21 [App ] VmSize: 2268088 kB
13:44:21 [App ] } ENDINFO
[INFO] running /code/SuperBuild/install/bin/OpenMVS/DensifyPointCloud --filter-point-cloud -1 -i “/odm_data/Pen-y-banc/opensfm/undistorted/openmvs/scene_dense.mvs” -v 0
13:44:21 [App ] Build date: May 25 2021, 16:11:57
13:44:21 [App ] CPU: Intel(R) Core™ i7-7700K CPU @ 4.20GHz (8 cores)
13:44:21 [App ] RAM: 62.70GB Physical Memory 2.00GB Virtual Memory
13:44:21 [App ] OS: Linux 5.4.0-80-generic (x86_64)
13:44:21 [App ] SSE & AVX compatible CPU & OS detected
13:44:21 [App ] Command line: --filter-point-cloud -1 -i /odm_data/Pen-y-banc/opensfm/undistorted/openmvs/scene_dense.mvs -v 0
13:44:21 [App ] error: empty initial point-cloud

===== Dumping Info for Geeks (developers need this to fix bugs) =====
Child returned 1
Traceback (most recent call last):
File “/code/stages/odm_app.py”, line 83, in execute
self.first_stage.run()
File “/code/opendm/types.py”, line 338, in run
self.next_stage.run(outputs)
File “/code/opendm/types.py”, line 338, in run
self.next_stage.run(outputs)
File “/code/opendm/types.py”, line 338, in run
self.next_stage.run(outputs)
[Previous line repeated 1 more time]
File “/code/opendm/types.py”, line 319, in run
self.process(self.args, outputs)
File “/code/stages/openmvs.py”, line 160, in process
system.run(‘%s %s’ % (context.omvs_densify_path, ’ '.join(config)))
File “/code/opendm/system.py”, line 90, in run
raise SubprocessException(“Child returned {}”.format(retcode), retcode)
opendm.system.SubprocessException: Child returned 1
===== Done, human-readable information to follow… =====

[ERROR] Uh oh! Processing stopped because of strange values in the reconstruction. This is often a sign that the input data has some issues or the software cannot deal with it. Have you followed best practices for data acquisition? See https://docs.opendronemap.org/flying.html

==========

Link to my images:
https://liveuclac-my.sharepoint.com/:f:/g/personal/tcrnbgh_ucl_ac_uk/Ev8YOo3rubtHs4VPQHdxXIEBsfJY-bVdFYBMXOEbyw0yEQ?e=b7TMHk

Saijin_Naib · 9 August 2021 15:31

It looks like something failed earlier in the pipeline to yield an empty initial point-cloud, as your log is indicating.

Interesting.

Let me try locally.

beedawg · 10 August 2021 07:38

As a side, I resized all the input images to width 2736px and re-processed using the above commands just fine.

Saijin_Naib · 16 August 2021 17:46

Yeah, I’m getting some weird issues when processing your dataset with the below settings:

Options: cog: true, crop: 0, debug: true, dem-gapfill-steps: 4, dem-resolution: 1, dsm: true, mesh-size: 300000, min-num-features: 12000, orthophoto-resolution: 1, pc-classify: true, pc-ept: true, pc-geometric: true, pc-quality: high, use-3dmesh: true, verbose: true, rerun-from: dataset

As you can see from the debug OpenMVS view, it is reconstructing pretty okay, but it is failing to include an entire corner of your dataset for some odd reason:

Then it goes on to fail to include ANY of the images for reconstruction:

08:26:34 [App     ] Preparing images for dense reconstruction completed: 594 images (28s190ms)
08:27:40 [App     ] Selecting images for dense reconstruction completed: 0 images (1m6s420ms)

Fused depth-maps 0 (100%, 0ms)
08:27:40 [App     ] Densifying point-cloud completed: 0 points (1m34s839ms)

I tried pre-processing, but it didn’t make a difference. I feel it may be because whole swathes of your images have completely different White Balance than other images, potentially confusing the matching algorithm. Unfortunately, XnConvert lacks an Auto White Balance tool, so I’m going to try using another tool to batch them and see if it helps at all.

beedawg · 14 September 2021 15:28

Hey thanks for this. Yeah I did notice the white balance seemed to change half way through the photos. Perhaps the sunlight changed or I accidentally hit a setting on the UAV controller?

Saijin_Naib · 14 September 2021 16:47

Yeah, I’m not certain! Really strange.

Changing the white-balance did not help reconstruction, unfortunately. I’m not sure why that dataset presents difficulties for reconstruction.

system · 14 October 2021 16:48

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.