Hi,
Im trying to process 450 Images and want to get the needed RAM size down. it seems like 32gb, or even 48gb is not enough.
I tried --split 100, but this doesnt change anything, even split 50 doesnt seem to make any changes.
I also tried max-concurrency 1, but this also seems to have no effect…
I can see that the parameter gets recognized, so thats not the problem.
what could I try next? should I be downsizing the images? my images are 5472x3648 ~8Mb
I think downsizing really is not an option though as it will reduce the map quality
1 Like
Sorry for the trouble! Could you please start with giving me a bit more information about your system?
For instance:
-
Operating System and Version
eg: Windows 11, MacOS 15.1, Ubuntu Linux 20.04LTS, etc… -
WebODM Install Method
eg: Native installer, Docker, Snap, GitHub download, compiled from source -
WebODM/ODM Version
eg: WebODM v1.9.12 Build 55, ODM v2.8.0, etc… -
Procesing Node
eg: Automatic, Lightning, local (node-odm-1), etc… -
Screenshots demonstrating the issue/behavior/error messages
Fedora 35
ODM via Docker (CLI)
latest release
local processing
I think I maybe have a hint: my queue-size is constantly increasing up to 82 images for that run.
at the beginning I found that line:
"2022-03-16 16:49:24,868 INFO: Expecting to process 82 images.
2022-03-16 16:49:24,868 INFO: Expecting to process 82 images.
2022-03-16 16:49:24,871 INFO: Reading data for image 100_0035_0001.JPG (queue-size=0
2022-03-16 16:49:24,871 INFO: Reading data for image 100_0035_0093.JPG (queue-size=0
2022-03-16 16:49:24,871 INFO: Reading data for image 100_0035_0213.JPG (queue-size=0
2022-03-16 16:49:24,871 INFO: Reading data for image 100_0035_0333.JPG (queue-size=0
2022-03-16 16:49:25,257 INFO: Reading data for image 100_0035_0094.JPG (queue-size=1
2022-03-16 16:49:25,261 INFO: Extracting ROOT_SIFT features for image 100_0035_0093.JPG
2022-03-16 16:49:25,275 INFO: Reading data for image 100_0035_0002.JPG (queue-size=1
2022-03-16 16:49:25,276 INFO: Extracting ROOT_SIFT features for image 100_0035_0001.JPG
2022-03-16 16:49:25,276 DEBUG: Computing sift with threshold 0.066
2022-03-16 16:49:25,278 INFO: Reading data for image 100_0035_0334.JPG (queue-size=1
2022-03-16 16:49:25,278 INFO: Extracting ROOT_SIFT features for image 100_0035_0333.JPG
2022-03-16 16:49:25,282 INFO: Reading data for image 100_0035_0214.JPG (queue-size=1
2022-03-16 16:49:25,284 INFO: Extracting ROOT_SIFT features for image 100_0035_0213.JPG
2022-03-16 16:49:25,301 DEBUG: Computing sift with threshold 0.066
2022-03-16 16:49:25,313 DEBUG: Computing sift with threshold 0.066
2022-03-16 16:49:25,322 DEBUG: Computing sift with threshold 0.066
2022-03-16 16:49:25,675 INFO: Reading data for image 100_0035_0095.JPG (queue-size=1
2022-03-16 16:49:25,676 INFO: Extracting ROOT_SIFT features for image 100_0035_0094.JPG
2022-03-16 16:49:25,699 DEBUG: Computing sift with threshold 0.066
2022-03-16 16:49:25,732 INFO: Reading data for image 100_0035_0335.JPG (queue-size=1
2022-03-16 16:49:25,733 INFO: Extracting ROOT_SIFT features for image 100_0035_0334.JPG
2022-03-16 16:49:25,756 INFO: Reading data for image 100_0035_0215.JPG (queue-size=1
2022-03-16 16:49:25,757 INFO: Extracting ROOT_SIFT features for image 100_0035_0214.JPG
2022-03-16 16:49:25,758 DEBUG: Computing sift with threshold 0.066
2022-03-16 16:49:25,781 INFO: Reading data for image 100_0035_0003.JPG (queue-size=1
2022-03-16 16:49:25,782 INFO: Extracting ROOT_SIFT features for image 100_0035_0002.JPG
2022-03-16 16:49:25,819 DEBUG: Computing sift with threshold 0.066
2022-03-16 16:49:25,829 DEBUG: Computing sift with threshold 0.066
2022-03-16 16:49:26,213 INFO: Reading data for image 100_0035_0096.JPG (queue-size=1
2022-03-16 16:49:26,213 INFO: Extracting ROOT_SIFT features for image 100_0035_0095.JPG
2022-03-16 16:49:26,247 DEBUG: Computing sift with threshold 0.066
2022-03-16 16:49:26,255 INFO: Reading data for image 100_0035_0336.JPG (queue-size=1
2022-03-16 16:49:26,257 INFO: Extracting ROOT_SIFT features for image 100_0035_0335.JPG
2022-03-16 16:49:26,286 DEBUG: Computing sift with threshold 0.066
2022-03-16 16:49:26,407 INFO: Reading data for image 100_0035_0216.JPG (queue-size=1
2022-03-16 16:49:26,408 INFO: Extracting ROOT_SIFT features for image 100_0035_0215.JPG
2022-03-16 16:49:26,414 INFO: Reading data for image 100_0035_0004.JPG (queue-size=1
2022-03-16 16:49:26,415 INFO: Extracting ROOT_SIFT features for image 100_0035_0003.JPG
2022-03-16 16:49:26,443 DEBUG: Computing sift with threshold 0.066
2022-03-16 16:49:26,475 DEBUG: Computing sift with threshold 0.066
2022-03-16 16:49:26,845 INFO: Reading data for image 100_0035_0337.JPG (queue-size=1
2022-03-16 16:49:26,854 INFO: Reading data for image 100_0035_0097.JPG (queue-size=2
2022-03-16 16:49:27,159 INFO: Reading data for image 100_0035_0217.JPG (queue-size=3
2022-03-16 16:49:27,195 INFO: Reading data for image 100_0035_0005.JPG (queue-size=4
2022-03-16 16:49:27,428 INFO: Reading data for image 100_0035_0338.JPG (queue-size=5
2022-03-16 16:49:27,498 INFO: Reading data for image 100_0035_0098.JPG (queue-size=6
2022-03-16 16:49:30,010 INFO: Reading data for image 100_0035_0218.JPG (queue-size=7
2022-03-16 16:49:30,236 INFO: Reading data for image 100_0035_0006.JPG (queue-size=8
2022-03-16 16:49:30,823 INFO: Reading data for image 100_0035_0339.JPG (queue-size=9
2022-03-16 16:49:31,035 INFO: Reading data for image 100_0035_0099.JPG (queue-size=10
2022-03-16 16:49:32,647 INFO: Reading data for image 100_0035_0007.JPG (queue-size=11
2022-03-16 16:49:32,729 INFO: Reading data for image 100_0035_0219.JPG (queue-size=12
2022-03-16 16:49:32,760 INFO: Reading data for image 100_0035_0341.JPG (queue-size=13
2022-03-16 16:49:32,798 INFO: Reading data for image 100_0035_0100.JPG (queue-size=14
2022-03-16 16:49:34,296 INFO: Reading data for image 100_0035_0008.JPG (queue-size=15
2022-03-16 16:49:34,414 INFO: Reading data for image 100_0035_0342.JPG (queue-size=16
2022-03-16 16:49:34,415 INFO: Reading data for image 100_0035_0220.JPG (queue-size=17
2022-03-16 16:49:34,521 INFO: Reading data for image 100_0035_0101.JPG (queue-size=18
2022-03-16 16:49:35,474 INFO: Reading data for image 100_0035_0222.JPG (queue-size=20
2022-03-16 16:49:35,472 INFO: Reading data for image 100_0035_0009.JPG (queue-size=19
2022-03-16 16:49:35,541 INFO: Reading data for image 100_0035_0343.JPG (queue-size=21
2022-03-16 16:49:35,598 INFO: Reading data for image 100_0035_0103.JPG (queue-size=22
2022-03-16 16:49:36,570 INFO: Reading data for image 100_0035_0344.JPG (queue-size=23
2022-03-16 16:49:36,574 INFO: Reading data for image 100_0035_0223.JPG (queue-size=24
2022-03-16 16:49:36,660 INFO: Reading data for image 100_0035_0010.JPG (queue-size=25
2022-03-16 16:49:36,666 INFO: Reading data for image 100_0035_0104.JPG (queue-size=26
2022-03-16 16:49:38,332 INFO: Reading data for image 100_0035_0345.JPG (queue-size=27
2022-03-16 16:49:38,334 INFO: Reading data for image 100_0035_0224.JPG (queue-size=28
2022-03-16 16:49:38,499 INFO: Reading data for image 100_0035_0011.JPG (queue-size=29
2022-03-16 16:49:38,541 INFO: Reading data for image 100_0035_0105.JPG (queue-size=30
2022-03-16 16:49:40,448 INFO: Reading data for image 100_0035_0346.JPG (queue-size=31
2022-03-16 16:49:40,452 INFO: Reading data for image 100_0035_0106.JPG (queue-size=32
2022-03-16 16:49:40,453 INFO: Reading data for image 100_0035_0225.JPG (queue-size=33
2022-03-16 16:49:40,650 INFO: Reading data for image 100_0035_0012.JPG (queue-size=34
2022-03-16 16:49:42,309 INFO: Reading data for image 100_0035_0107.JPG (queue-size=35
2022-03-16 16:49:42,395 INFO: Reading data for image 100_0035_0226.JPG (queue-size=36
2022-03-16 16:49:42,396 INFO: Reading data for image 100_0035_0347.JPG (queue-size=37
2022-03-16 16:49:42,441 INFO: Reading data for image 100_0035_0013.JPG (queue-size=38
2022-03-16 16:49:44,040 INFO: Reading data for image 100_0035_0108.JPG (queue-size=39
2022-03-16 16:49:44,044 INFO: Reading data for image 100_0035_0348.JPG (queue-size=40
2022-03-16 16:49:44,085 INFO: Reading data for image 100_0035_0227.JPG (queue-size=41
2022-03-16 16:49:44,092 INFO: Reading data for image 100_0035_0014.JPG (queue-size=42
2022-03-16 16:49:45,190 INFO: Reading data for image 100_0035_0109.JPG (queue-size=44
2022-03-16 16:49:45,189 INFO: Reading data for image 100_0035_0015.JPG (queue-size=43
2022-03-16 16:49:45,232 INFO: Reading data for image 100_0035_0349.JPG (queue-size=45
2022-03-16 16:49:45,311 INFO: Reading data for image 100_0035_0228.JPG (queue-size=46
2022-03-16 16:49:46,398 INFO: Reading data for image 100_0035_0350.JPG (queue-size=47
2022-03-16 16:49:46,416 INFO: Reading data for image 100_0035_0016.JPG (queue-size=48
2022-03-16 16:49:46,441 INFO: Reading data for image 100_0035_0110.JPG (queue-size=49
2022-03-16 16:49:46,470 INFO: Reading data for image 100_0035_0229.JPG (queue-size=50
2022-03-16 16:49:47,239 INFO: Reading data for image 100_0035_0351.JPG (queue-size=51
2022-03-16 16:49:47,314 INFO: Reading data for image 100_0035_0230.JPG (queue-size=52
2022-03-16 16:49:47,327 INFO: Reading data for image 100_0035_0111.JPG (queue-size=53
2022-03-16 16:49:47,349 INFO: Reading data for image 100_0035_0018.JPG (queue-size=54
2022-03-16 16:49:48,166 INFO: Reading data for image 100_0035_0352.JPG (queue-size=55
2022-03-16 16:49:48,212 INFO: Reading data for image 100_0035_0112.JPG (queue-size=56
2022-03-16 16:49:48,230 INFO: Reading data for image 100_0035_0231.JPG (queue-size=57
2022-03-16 16:49:48,230 INFO: Reading data for image 100_0035_0019.JPG (queue-size=58
2022-03-16 16:49:48,851 INFO: Reading data for image 100_0035_0353.JPG (queue-size=59
2022-03-16 16:49:49,123 INFO: Reading data for image 100_0035_0113.JPG (queue-size=60
2022-03-16 16:49:49,141 INFO: Reading data for image 100_0035_0232.JPG (queue-size=61
2022-03-16 16:49:49,163 INFO: Reading data for image 100_0035_0020.JPG (queue-size=62
2022-03-16 16:49:49,399 INFO: Reading data for image 100_0035_0354.JPG (queue-size=63
2022-03-16 16:49:49,811 INFO: Reading data for image 100_0035_0114.JPG (queue-size=64
2022-03-16 16:49:49,872 INFO: Reading data for image 100_0035_0233.JPG (queue-size=65
2022-03-16 16:49:49,902 INFO: Reading data for image 100_0035_0021.JPG (queue-size=66
2022-03-16 16:49:50,083 INFO: Reading data for image 100_0035_0355.JPG (queue-size=67
2022-03-16 16:49:50,378 INFO: Reading data for image 100_0035_0115.JPG (queue-size=68
2022-03-16 16:49:50,591 INFO: Reading data for image 100_0035_0234.JPG (queue-size=69
2022-03-16 16:49:50,642 INFO: Reading data for image 100_0035_0022.JPG (queue-size=70
2022-03-16 16:49:50,740 INFO: Reading data for image 100_0035_0356.JPG (queue-size=71
2022-03-16 16:49:50,916 INFO: Reading data for image 100_0035_0116.JPG (queue-size=72
2022-03-16 16:49:51,266 INFO: Reading data for image 100_0035_0235.JPG (queue-size=73
2022-03-16 16:49:51,366 INFO: Reading data for image 100_0035_0023.JPG (queue-size=74
2022-03-16 16:49:51,419 INFO: Reading data for image 100_0035_0358.JPG (queue-size=75
2022-03-16 16:49:51,546 INFO: Reading data for image 100_0035_0117.JPG (queue-size=76
2022-03-16 16:49:51,872 INFO: Reading data for image 100_0035_0236.JPG (queue-size=77
2022-03-16 16:49:52,171 INFO: Reading data for image 100_0035_0359.JPG (queue-size=78
2022-03-16 16:49:52,171 INFO: Reading data for image 100_0035_0024.JPG (queue-size=79
2022-03-16 16:49:52,232 INFO: Reading data for image 100_0035_0118.JPG (queue-size=80
2022-03-16 16:49:52,412 INFO: Reading data for image 100_0035_0237.JPG (queue-size=81
2022-03-16 16:49:52,978 INFO: Reading data for image 100_0035_0025.JPG (queue-size=82
so what does this mean?
1 Like
What parameters are you using? Also, do you have a swap volume or file, and what is your swappiness set to?
Virtual memory is incredibly important to have during processing.
The queue size is more or less defined based upon a guess of how much free memory you have at that moment, divided roughly by two and then chunked up, so in other words, I wouldn’t put too much focus on it.
parameters are just --fast-orthophoto
I have a 64gb swap volume on a gen4 nvme, didnt touch the swappiness settings.
regarding the queue size, I was just curious as in the beginning there was other output between the “reading data…” such as “extracting root_sift…” and then it stopped and printed only the “reading data…” while my ram filled up.
1 Like
I think I solved it. there is no problem. the initial ram usage is just so big that my 16gb machine + 100gb swap file gets so slow I thought it crashed. just tried on an AMD epyc server with 90gb ram und it runs really fast.
2 Likes
Yep… Orders of magnitude slower 
Glad you’re processing successfully now!
I have to ask something again:
why ddoes max-concurrency seem to have no effect? I have a machine with 16 cores and one with 64 cores. the ram usage on the 64 core machine is way higher than on the 16 core machine. I set max-concurrency to 16, but I can still see way more detect_feature processes… why is that?
1 Like
Can you provide full logs?
What machine is this on, your Linux one?
both machines are linux, one is debian and one is fedora.
this is the log of the Debian machine(90gb ram + 80gb swap 32C/64T)
[email protected]:~/photogrammetry# docker run -ti --rm -v ~/photogrammetry:/datasets opendronemap/odm --project-path /datasets nv_dark_small_5deg --fast-orthophoto --auto-boundary --ignore-gsd --orthophoto-resolution 2 --feature-quality ultra --max-concurrency 16
[INFO] Fast orthophoto is turned on, automatically setting --skip-3dmodel
[INFO] Initializing ODM 2.8.1 - Wed Mar 23 15:19:08 2022
[INFO] ==============
[INFO] auto_boundary: True
[INFO] boundary: {}
[INFO] build_overviews: False
[INFO] camera_lens: auto
[INFO] cameras: {}
[INFO] cog: False
[INFO] copy_to: None
[INFO] crop: 3
[INFO] debug: False
[INFO] dem_decimation: 1
[INFO] dem_euclidean_map: False
[INFO] dem_gapfill_steps: 3
[INFO] dem_resolution: 5
[INFO] depthmap_resolution: 640
[INFO] dsm: False
[INFO] dtm: False
[INFO] end_with: odm_postprocess
[INFO] fast_orthophoto: True
[INFO] feature_quality: ultra
[INFO] feature_type: sift
[INFO] force_gps: False
[INFO] gcp: None
[INFO] geo: None
[INFO] gps_accuracy: 10
[INFO] ignore_gsd: True
[INFO] matcher_neighbors: 8
[INFO] matcher_type: flann
[INFO] max_concurrency: 16
[INFO] merge: all
[INFO] mesh_octree_depth: 11
[INFO] mesh_size: 200000
[INFO] min_num_features: 10000
[INFO] name: nv_dark_small_5deg
[INFO] optimize_disk_space: False
[INFO] orthophoto_compression: DEFLATE
[INFO] orthophoto_cutline: False
[INFO] orthophoto_kmz: False
[INFO] orthophoto_no_tiled: False
[INFO] orthophoto_png: False
[INFO] orthophoto_resolution: 2.0
[INFO] pc_classify: False
[INFO] pc_csv: False
[INFO] pc_ept: False
[INFO] pc_filter: 2.5
[INFO] pc_geometric: False
[INFO] pc_las: False
[INFO] pc_quality: medium
[INFO] pc_rectify: False
[INFO] pc_sample: 0
[INFO] pc_tile: False
[INFO] primary_band: auto
[INFO] project_path: /datasets
[INFO] radiometric_calibration: none
[INFO] rerun: None
[INFO] rerun_all: False
[INFO] rerun_from: None
[INFO] resize_to: 2048
[INFO] sfm_algorithm: incremental
[INFO] skip_3dmodel: True
[INFO] skip_band_alignment: False
[INFO] skip_orthophoto: False
[INFO] skip_report: False
[INFO] sm_cluster: None
[INFO] smrf_scalar: 1.25
[INFO] smrf_slope: 0.15
[INFO] smrf_threshold: 0.5
[INFO] smrf_window: 18.0
[INFO] split: 999999
[INFO] split_image_groups: None
[INFO] split_overlap: 150
[INFO] texturing_data_term: gmi
[INFO] texturing_keep_unseen_faces: False
[INFO] texturing_outlier_removal_type: gauss_clamping
[INFO] texturing_skip_global_seam_leveling: False
[INFO] texturing_skip_local_seam_leveling: False
[INFO] texturing_tone_mapping: none
[INFO] tiles: False
[INFO] time: False
[INFO] use_3dmesh: False
[INFO] use_exif: False
[INFO] use_fixed_camera_params: False
[INFO] use_hybrid_bundle_adjustment: False
[INFO] verbose: False
[INFO] ==============
[INFO] Running dataset stage
[INFO] Loading dataset from: /datasets/nv_dark_small_5deg/images
[INFO] Loading images database: /datasets/nv_dark_small_5deg/images.json
[INFO] Found 90 usable images
[INFO] Coordinates file already exist: /datasets/nv_dark_small_5deg/odm_georeferencing/coords.txt
[INFO] Model geo file already exist: /datasets/nv_dark_small_5deg/odm_georeferencing/odm_georeferencing_model_geo.txt
[INFO] Parsing SRS header: WGS84 UTM 32N
[INFO] Finished dataset stage
[INFO] Running split stage
[INFO] Normal dataset, will process all at once.
[INFO] Finished split stage
[INFO] Running merge stage
[INFO] Normal dataset, nothing to merge.
[INFO] Finished merge stage
[INFO] Running opensfm stage
[WARNING] /datasets/nv_dark_small_5deg/opensfm/image_list.txt already exists, not rerunning OpenSfM setup
[WARNING] /datasets/nv_dark_small_5deg/opensfm/exif already exists, not rerunning photo to metadata
[INFO] running "/code/SuperBuild/install/bin/opensfm/bin/opensfm" detect_features "/datasets/nv_dark_small_5deg/opensfm"
2022-03-23 15:19:09,518 INFO: Capping memory usage to ~ 60755.3359375 MB
2022-03-23 15:19:09,518 INFO: Expecting to process 200 images.
2022-03-23 15:19:09,535 INFO: Reading data for image 100_0073_0001.JPG (queue-size=0
2022-03-23 15:19:09,535 INFO: Reading data for image 100_0073_0024.JPG (queue-size=0
2022-03-23 15:19:09,535 INFO: Reading data for image 100_0073_0047.JPG (queue-size=0
2022-03-23 15:19:09,535 INFO: Reading data for image 100_0073_0070.JPG (queue-size=0
2022-03-23 15:19:09,957 INFO: Reading data for image 100_0073_0002.JPG (queue-size=1
2022-03-23 15:19:09,957 INFO: Extracting ROOT_SIFT features for image 100_0073_0001.JPG
2022-03-23 15:19:09,962 INFO: Reading data for image 100_0073_0048.JPG (queue-size=1
2022-03-23 15:19:09,963 INFO: Extracting ROOT_SIFT features for image 100_0073_0047.JPG
2022-03-23 15:19:09,975 DEBUG: Computing sift with threshold 0.066
2022-03-23 15:19:09,977 DEBUG: Computing sift with threshold 0.066
2022-03-23 15:19:09,985 INFO: Reading data for image 100_0073_0071.JPG (queue-size=1
2022-03-23 15:19:09,985 INFO: Extracting ROOT_SIFT features for image 100_0073_0070.JPG
2022-03-23 15:19:09,986 INFO: Extracting ROOT_SIFT features for image 100_0073_0024.JPG
2022-03-23 15:19:09,986 INFO: Reading data for image 100_0073_0025.JPG (queue-size=1
2022-03-23 15:19:10,009 DEBUG: Computing sift with threshold 0.066
2022-03-23 15:19:10,010 DEBUG: Computing sift with threshold 0.066
2022-03-23 15:19:10,401 INFO: Reading data for image 100_0073_0049.JPG (queue-size=1
2022-03-23 15:19:10,402 INFO: Extracting ROOT_SIFT features for image 100_0073_0048.JPG
2022-03-23 15:19:10,419 DEBUG: Computing sift with threshold 0.066
2022-03-23 15:19:10,422 INFO: Reading data for image 100_0073_0003.JPG (queue-size=1
2022-03-23 15:19:10,422 INFO: Extracting ROOT_SIFT features for image 100_0073_0002.JPG
2022-03-23 15:19:10,427 INFO: Reading data for image 100_0073_0026.JPG (queue-size=1
2022-03-23 15:19:10,428 INFO: Extracting ROOT_SIFT features for image 100_0073_0025.JPG
2022-03-23 15:19:10,444 DEBUG: Computing sift with threshold 0.066
2022-03-23 15:19:10,452 DEBUG: Computing sift with threshold 0.066
2022-03-23 15:19:10,464 INFO: Reading data for image 100_0073_0072.JPG (queue-size=1
2022-03-23 15:19:10,465 INFO: Extracting ROOT_SIFT features for image 100_0073_0071.JPG
2022-03-23 15:19:10,483 DEBUG: Computing sift with threshold 0.066
2022-03-23 15:19:10,888 INFO: Reading data for image 100_0073_0050.JPG (queue-size=1
2022-03-23 15:19:10,888 INFO: Extracting ROOT_SIFT features for image 100_0073_0049.JPG
2022-03-23 15:19:10,902 INFO: Reading data for image 100_0073_0027.JPG (queue-size=1
2022-03-23 15:19:10,903 INFO: Extracting ROOT_SIFT features for image 100_0073_0026.JPG
2022-03-23 15:19:10,911 INFO: Reading data for image 100_0073_0004.JPG (queue-size=1
2022-03-23 15:19:10,912 INFO: Extracting ROOT_SIFT features for image 100_0073_0003.JPG
2022-03-23 15:19:10,914 DEBUG: Computing sift with threshold 0.066
2022-03-23 15:19:10,927 DEBUG: Computing sift with threshold 0.066
2022-03-23 15:19:10,934 INFO: Reading data for image 100_0073_0073.JPG (queue-size=1
2022-03-23 15:19:10,934 INFO: Extracting ROOT_SIFT features for image 100_0073_0072.JPG
2022-03-23 15:19:10,944 DEBUG: Computing sift with threshold 0.066
2022-03-23 15:19:10,958 DEBUG: Computing sift with threshold 0.066
2022-03-23 15:19:11,407 INFO: Reading data for image 100_0073_0005.JPG (queue-size=1
2022-03-23 15:19:11,407 INFO: Extracting ROOT_SIFT features for image 100_0073_0004.JPG
2022-03-23 15:19:11,431 DEBUG: Computing sift with threshold 0.066
2022-03-23 15:19:11,463 INFO: Reading data for image 100_0073_0051.JPG (queue-size=1
2022-03-23 15:19:11,464 INFO: Extracting ROOT_SIFT features for image 100_0073_0050.JPG
2022-03-23 15:19:11,479 INFO: Reading data for image 100_0073_0074.JPG (queue-size=1
2022-03-23 15:19:11,480 INFO: Extracting ROOT_SIFT features for image 100_0073_0073.JPG
2022-03-23 15:19:11,502 DEBUG: Computing sift with threshold 0.066
2022-03-23 15:19:11,514 INFO: Reading data for image 100_0073_0028.JPG (queue-size=1
2022-03-23 15:19:11,514 INFO: Extracting ROOT_SIFT features for image 100_0073_0027.JPG
2022-03-23 15:19:11,519 DEBUG: Computing sift with threshold 0.066
2022-03-23 15:19:11,565 DEBUG: Computing sift with threshold 0.066
2022-03-23 15:19:11,926 INFO: Reading data for image 100_0073_0006.JPG (queue-size=1
2022-03-23 15:19:11,927 INFO: Extracting ROOT_SIFT features for image 100_0073_0005.JPG
2022-03-23 15:19:11,972 DEBUG: Computing sift with threshold 0.066
2022-03-23 15:19:12,119 INFO: Reading data for image 100_0073_0052.JPG (queue-size=1
2022-03-23 15:19:12,119 INFO: Extracting ROOT_SIFT features for image 100_0073_0051.JPG
2022-03-23 15:19:12,145 INFO: Reading data for image 100_0073_0075.JPG (queue-size=1
2022-03-23 15:19:12,146 INFO: Extracting ROOT_SIFT features for image 100_0073_0074.JPG
2022-03-23 15:19:12,159 INFO: Reading data for image 100_0073_0029.JPG (queue-size=1
2022-03-23 15:19:12,171 INFO: Extracting ROOT_SIFT features for image 100_0073_0028.JPG
2022-03-23 15:19:12,173 DEBUG: Computing sift with threshold 0.066
2022-03-23 15:19:12,215 DEBUG: Computing sift with threshold 0.066
2022-03-23 15:19:12,237 DEBUG: Computing sift with threshold 0.066
2022-03-23 15:19:12,528 INFO: Reading data for image 100_0073_0007.JPG (queue-size=1
2022-03-23 15:19:12,529 INFO: Extracting ROOT_SIFT features for image 100_0073_0006.JPG
2022-03-23 15:19:12,578 DEBUG: Computing sift with threshold 0.066
2022-03-23 15:19:12,704 INFO: Reading data for image 100_0073_0053.JPG (queue-size=1
2022-03-23 15:19:12,705 INFO: Extracting ROOT_SIFT features for image 100_0073_0052.JPG
2022-03-23 15:19:12,742 DEBUG: Computing sift with threshold 0.066
2022-03-23 15:19:12,779 INFO: Reading data for image 100_0073_0076.JPG (queue-size=1
2022-03-23 15:19:12,780 INFO: Extracting ROOT_SIFT features for image 100_0073_0075.JPG
2022-03-23 15:19:12,809 INFO: Reading data for image 100_0073_0030.JPG (queue-size=1
2022-03-23 15:19:12,810 INFO: Extracting ROOT_SIFT features for image 100_0073_0029.JPG
2022-03-23 15:19:12,830 DEBUG: Computing sift with threshold 0.066
2022-03-23 15:19:12,868 DEBUG: Computing sift with threshold 0.066
2022-03-23 15:19:13,191 INFO: Reading data for image 100_0073_0008.JPG (queue-size=1
2022-03-23 15:19:13,191 INFO: Extracting ROOT_SIFT features for image 100_0073_0007.JPG
2022-03-23 15:19:13,253 DEBUG: Computing sift with threshold 0.066
2022-03-23 15:19:13,485 INFO: Reading data for image 100_0073_0031.JPG (queue-size=1
2022-03-23 15:19:13,486 INFO: Extracting ROOT_SIFT features for image 100_0073_0030.JPG
2022-03-23 15:19:13,521 INFO: Reading data for image 100_0073_0054.JPG (queue-size=1
2022-03-23 15:19:13,521 INFO: Extracting ROOT_SIFT features for image 100_0073_0053.JPG
2022-03-23 15:19:13,523 DEBUG: Computing sift with threshold 0.066
2022-03-23 15:19:13,531 INFO: Reading data for image 100_0073_0077.JPG (queue-size=1
2022-03-23 15:19:13,532 INFO: Extracting ROOT_SIFT features for image 100_0073_0076.JPG
2022-03-23 15:19:13,570 DEBUG: Computing sift with threshold 0.066
2022-03-23 15:19:13,586 DEBUG: Computing sift with threshold 0.066
2022-03-23 15:19:13,879 INFO: Reading data for image 100_0073_0009.JPG (queue-size=1
2022-03-23 15:19:13,880 INFO: Extracting ROOT_SIFT features for image 100_0073_0008.JPG
2022-03-23 15:19:13,948 DEBUG: Computing sift with threshold 0.066
2022-03-23 15:19:14,401 INFO: Reading data for image 100_0073_0055.JPG (queue-size=1
2022-03-23 15:19:14,401 INFO: Extracting ROOT_SIFT features for image 100_0073_0054.JPG
2022-03-23 15:19:14,460 DEBUG: Computing sift with threshold 0.066
2022-03-23 15:19:14,505 INFO: Reading data for image 100_0073_0032.JPG (queue-size=1
2022-03-23 15:19:14,506 INFO: Extracting ROOT_SIFT features for image 100_0073_0031.JPG
2022-03-23 15:19:14,534 INFO: Reading data for image 100_0073_0078.JPG (queue-size=1
2022-03-23 15:19:14,535 INFO: Extracting ROOT_SIFT features for image 100_0073_0077.JPG
2022-03-23 15:19:14,549 DEBUG: Computing sift with threshold 0.066
2022-03-23 15:19:14,579 DEBUG: Computing sift with threshold 0.066
2022-03-23 15:19:14,759 INFO: Reading data for image 100_0073_0010.JPG (queue-size=1
2022-03-23 15:19:14,760 INFO: Extracting ROOT_SIFT features for image 100_0073_0009.JPG
2022-03-23 15:19:14,808 DEBUG: Computing sift with threshold 0.066
2022-03-23 15:19:15,327 INFO: Reading data for image 100_0073_0033.JPG (queue-size=1
2022-03-23 15:19:15,329 INFO: Extracting ROOT_SIFT features for image 100_0073_0032.JPG
2022-03-23 15:19:15,337 INFO: Reading data for image 100_0073_0056.JPG (queue-size=1
2022-03-23 15:19:15,338 INFO: Extracting ROOT_SIFT features for image 100_0073_0055.JPG
2022-03-23 15:19:15,403 DEBUG: Computing sift with threshold 0.066
2022-03-23 15:19:15,405 DEBUG: Computing sift with threshold 0.066
2022-03-23 15:19:15,574 INFO: Reading data for image 100_0073_0079.JPG (queue-size=1
2022-03-23 15:19:15,575 INFO: Extracting ROOT_SIFT features for image 100_0073_0078.JPG
2022-03-23 15:19:15,681 DEBUG: Computing sift with threshold 0.066
2022-03-23 15:19:15,844 INFO: Reading data for image 100_0073_0011.JPG (queue-size=1
2022-03-23 15:19:15,845 INFO: Extracting ROOT_SIFT features for image 100_0073_0010.JPG
2022-03-23 15:19:15,908 DEBUG: Computing sift with threshold 0.066
2022-03-23 15:19:16,173 INFO: Reading data for image 100_0073_0034.JPG (queue-size=1
2022-03-23 15:19:16,174 INFO: Extracting ROOT_SIFT features for image 100_0073_0033.JPG
2022-03-23 15:19:16,278 DEBUG: Computing sift with threshold 0.066
2022-03-23 15:19:16,343 INFO: Reading data for image 100_0073_0057.JPG (queue-size=1
2022-03-23 15:19:16,345 INFO: Extracting ROOT_SIFT features for image 100_0073_0056.JPG
2022-03-23 15:19:16,448 DEBUG: Computing sift with threshold 0.066
2022-03-23 15:19:16,738 INFO: Reading data for image 100_0073_0080.JPG (queue-size=1
2022-03-23 15:19:16,739 INFO: Extracting ROOT_SIFT features for image 100_0073_0079.JPG
2022-03-23 15:19:16,856 DEBUG: Computing sift with threshold 0.066
2022-03-23 15:19:17,123 INFO: Reading data for image 100_0073_0012.JPG (queue-size=1
2022-03-23 15:19:17,124 INFO: Extracting ROOT_SIFT features for image 100_0073_0011.JPG
2022-03-23 15:19:17,241 DEBUG: Computing sift with threshold 0.066
2022-03-23 15:19:17,504 INFO: Reading data for image 100_0073_0035.JPG (queue-size=1
2022-03-23 15:19:17,506 INFO: Extracting ROOT_SIFT features for image 100_0073_0034.JPG
2022-03-23 15:19:17,628 DEBUG: Computing sift with threshold 0.066
2022-03-23 15:19:17,637 INFO: Reading data for image 100_0073_0058.JPG (queue-size=1
2022-03-23 15:19:17,638 INFO: Extracting ROOT_SIFT features for image 100_0073_0057.JPG
2022-03-23 15:19:17,791 DEBUG: Computing sift with threshold 0.066
2022-03-23 15:19:18,099 INFO: Reading data for image 100_0073_0081.JPG (queue-size=1
2022-03-23 15:19:18,100 INFO: Extracting ROOT_SIFT features for image 100_0073_0080.JPG
2022-03-23 15:19:18,342 DEBUG: Computing sift with threshold 0.066
2022-03-23 15:19:18,666 INFO: Reading data for image 100_0073_0013.JPG (queue-size=1
2022-03-23 15:19:18,667 INFO: Extracting ROOT_SIFT features for image 100_0073_0012.JPG
2022-03-23 15:19:18,771 INFO: Reading data for image 100_0073_0036.JPG (queue-size=1
2022-03-23 15:19:18,772 INFO: Extracting ROOT_SIFT features for image 100_0073_0035.JPG
2022-03-23 15:19:18,924 DEBUG: Computing sift with threshold 0.066
2022-03-23 15:19:18,947 DEBUG: Computing sift with threshold 0.066
2022-03-23 15:19:26,508 INFO: Reading data for image 100_0073_0082.JPG (queue-size=1
2022-03-23 15:19:26,510 INFO: Reading data for image 100_0073_0059.JPG (queue-size=1
2022-03-23 15:19:26,512 INFO: Extracting ROOT_SIFT features for image 100_0073_0058.JPG
2022-03-23 15:19:26,511 INFO: Extracting ROOT_SIFT features for image 100_0073_0081.JPG
2022-03-23 15:19:26,974 DEBUG: Computing sift with threshold 0.066
2022-03-23 15:19:27,005 DEBUG: Computing sift with threshold 0.066
2022-03-23 15:19:33,929 INFO: Reading data for image 100_0073_0037.JPG (queue-size=1
2022-03-23 15:19:33,930 INFO: Extracting ROOT_SIFT features for image 100_0073_0036.JPG
2022-03-23 15:19:34,211 DEBUG: Computing sift with threshold 0.066
2022-03-23 15:19:34,212 INFO: Reading data for image 100_0073_0014.JPG (queue-size=1
2022-03-23 15:19:34,213 INFO: Extracting ROOT_SIFT features for image 100_0073_0013.JPG
2022-03-23 15:19:34,894 DEBUG: Computing sift with threshold 0.066
2022-03-23 15:19:41,183 INFO: Reading data for image 100_0073_0083.JPG (queue-size=1
2022-03-23 15:19:41,210 INFO: Extracting ROOT_SIFT features for image 100_0073_0059.JPG
2022-03-23 15:19:41,183 INFO: Reading data for image 100_0073_0060.JPG (queue-size=2
2022-03-23 15:19:41,283 INFO: Extracting ROOT_SIFT features for image 100_0073_0082.JPG
2022-03-23 15:19:42,870 DEBUG: Computing sift with threshold 0.066
2022-03-23 15:19:43,191 DEBUG: Computing sift with threshold 0.066
/code/SuperBuild/install/bin/opensfm/bin/opensfm: line 12: 139 Killed "$PYTHON" "$DIR"/opensfm_main.py "$@"
===== Dumping Info for Geeks (developers need this to fix bugs) =====
Child returned 137
Traceback (most recent call last):
File "/code/stages/odm_app.py", line 94, in execute
self.first_stage.run()
File "/code/opendm/types.py", line 346, in run
self.next_stage.run(outputs)
File "/code/opendm/types.py", line 346, in run
self.next_stage.run(outputs)
File "/code/opendm/types.py", line 346, in run
self.next_stage.run(outputs)
File "/code/opendm/types.py", line 327, in run
self.process(self.args, outputs)
File "/code/stages/run_opensfm.py", line 35, in process
octx.feature_matching(self.rerun())
File "/code/opendm/osfm.py", line 376, in feature_matching
raise e
File "/code/opendm/osfm.py", line 366, in feature_matching
self.run('detect_features')
File "/code/opendm/osfm.py", line 34, in run
system.run('"%s" %s "%s"' %
File "/code/opendm/system.py", line 106, in run
raise SubprocessException("Child returned {}".format(retcode), retcode)
opendm.system.SubprocessException: Child returned 137
===== Done, human-readable information to follow... =====
[ERROR] Whoops! You ran out of memory! Add more RAM to your computer, if you're using docker configure it to use more memory, for WSL2 make use of .wslconfig (https://docs.microsoft.com/en-us/windows/wsl/wsl-config#configure-global-options-with-wslconfig), resize your images, lower the quality settings or process the images using a cloud provider (e.g. https://webodm.net).
I could see way more than 16 concurrent processes in htop.
1 Like
You probably need to clean all the files created in the previous run, else ODM could use the opensfm config from the previous run.
1 Like
But I also noticed this problem, if I use concurrency larger than 1, opensfm will always eat up my memory no matter what value I set. The queue size kept increasing and reached a point to crash.
1 Like
I already tried it with a clean start, same result.
1 Like
In my case, a clean start with max-concurrency 1 will work, but if set the value larger than 1, the OOM error will always occur.
1 Like
I dont know why, but I just tried it with 1 and it worked. then I tried it again with 8 and 16 and it also worked…
maybe I didnt delete something on my first test. maybe try with a value of 2 and a clean start?
1 Like
That sounds like a good idea.
With strange seemingly random errors like this, a stable/reliable STR (Steps To Reproduce) are essential for us to track down what is going wrong. So, anything you can do to help in that regard is much, much appreciated.
Just make sure to have a clean start each time if you want to test.
The problem should be caused by feature detection in opensfm stage, which probably has something to do with the queue-size. Normally if I faced this issue, I will use max-concurrency 1 until the process finished feature detection, then terminate the process, go to the dataset folder, go to opensfm folder and edit config.yaml, change processes: 1 to the processes: 8(based on the number of your CPU cores), then in odm argumetns use --max-concurrency 8 as well. ODM will pick up the existing data and keep processing with max concurrency.
2 Likes
Interesting. because I can now verify that it works with a clean start, not just 1.
2 Likes
This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.