Not enough memory in Linux

Hello everybody! I’m new to the forum. I’ve used ODM before successfully on an older computer on similar tasks, with the same RAM I have now (16 GB).
But now I get a not enough memory error.
I am using default options. Trying to process 45 48MP 22 MB photos taken with a DJI Mavic Air.
I am using Endeavour OS (Arch Linux) on a MSI Alpha 17 computer. Librewolf browser. I installed ODM by git clone, following the official installation guide, but then I migrated to a new computer and so I copied my home folder and didn’t GIT clone again. The first time I started ODM on this new computer it took a while downloading what I thing were updates.

What I found in forums about similar issues proposed solutions having to do with WSL which seems to be a Windows thing, or suggested to increase docker memory size but I can’t find where is the docker config file to change that.

Thanks!
I attach log.

[INFO]    Initializing ODM 2.7.0 - Mon Jan 10 01:44:14  2022
[INFO]    ==============
[INFO]    auto_boundary: True
[INFO]    boundary: {}
[INFO]    build_overviews: False
[INFO]    camera_lens: auto
[INFO]    cameras: {}
[INFO]    cog: True
[INFO]    copy_to: None
[INFO]    crop: 3
[INFO]    debug: False
[INFO]    dem_decimation: 1
[INFO]    dem_euclidean_map: False
[INFO]    dem_gapfill_steps: 3
[INFO]    dem_resolution: 5
[INFO]    depthmap_resolution: 640
[INFO]    dsm: True
[INFO]    dtm: False
[INFO]    end_with: odm_postprocess
[INFO]    fast_orthophoto: False
[INFO]    feature_quality: high
[INFO]    feature_type: sift
[INFO]    force_gps: False
[INFO]    gcp: None
[INFO]    geo: None
[INFO]    gps_accuracy: 10
[INFO]    ignore_gsd: False
[INFO]    matcher_neighbors: 8
[INFO]    matcher_type: flann
[INFO]    max_concurrency: 16
[INFO]    merge: all
[INFO]    mesh_octree_depth: 11
[INFO]    mesh_size: 200000
[INFO]    min_num_features: 8000
[INFO]    name: 814dcc26-558f-4a06-a6a0-9110460cb575
[INFO]    optimize_disk_space: False
[INFO]    orthophoto_compression: DEFLATE
[INFO]    orthophoto_cutline: False
[INFO]    orthophoto_kmz: False
[INFO]    orthophoto_no_tiled: False
[INFO]    orthophoto_png: False
[INFO]    orthophoto_resolution: 5
[INFO]    pc_classify: False
[INFO]    pc_csv: False
[INFO]    pc_ept: True
[INFO]    pc_filter: 2.5
[INFO]    pc_geometric: False
[INFO]    pc_las: False
[INFO]    pc_quality: medium
[INFO]    pc_rectify: False
[INFO]    pc_sample: 0
[INFO]    pc_tile: False
[INFO]    primary_band: auto
[INFO]    project_path: /var/www/data
[INFO]    radiometric_calibration: none
[INFO]    rerun: None
[INFO]    rerun_all: False
[INFO]    rerun_from: ['odm_postprocess']
[INFO]    resize_to: 2048
[INFO]    sfm_algorithm: incremental
[INFO]    skip_3dmodel: False
[INFO]    skip_band_alignment: False
[INFO]    skip_orthophoto: False
[INFO]    skip_report: False
[INFO]    sm_cluster: None
[INFO]    smrf_scalar: 1.25
[INFO]    smrf_slope: 0.15
[INFO]    smrf_threshold: 0.5
[INFO]    smrf_window: 18.0
[INFO]    split: 999999
[INFO]    split_image_groups: None
[INFO]    split_overlap: 150
[INFO]    texturing_data_term: gmi
[INFO]    texturing_keep_unseen_faces: False
[INFO]    texturing_outlier_removal_type: gauss_clamping
[INFO]    texturing_skip_global_seam_leveling: False
[INFO]    texturing_skip_local_seam_leveling: False
[INFO]    texturing_tone_mapping: none
[INFO]    tiles: False
[INFO]    time: False
[INFO]    use_3dmesh: False
[INFO]    use_exif: False
[INFO]    use_fixed_camera_params: False
[INFO]    use_hybrid_bundle_adjustment: False
[INFO]    verbose: False
[INFO]    ==============
[INFO]    Running dataset stage
[INFO]    Loading dataset from: /var/www/data/814dcc26-558f-4a06-a6a0-9110460cb575/images
[INFO]    Loading images database: /var/www/data/814dcc26-558f-4a06-a6a0-9110460cb575/images.json
[INFO]    Found 45 usable images
[INFO]    Coordinates file already exist: /var/www/data/814dcc26-558f-4a06-a6a0-9110460cb575/odm_georeferencing/coords.txt
[INFO]    Model geo file already exist: /var/www/data/814dcc26-558f-4a06-a6a0-9110460cb575/odm_georeferencing/odm_georeferencing_model_geo.txt
[INFO]    Parsing SRS header: WGS84 UTM 19S
[INFO]    Finished dataset stage
[INFO]    Running split stage
[INFO]    Normal dataset, will process all at once.
[INFO]    Finished split stage
[INFO]    Running merge stage
[INFO]    Normal dataset, nothing to merge.
[INFO]    Finished merge stage
[INFO]    Running opensfm stage
[WARNING] /var/www/data/814dcc26-558f-4a06-a6a0-9110460cb575/opensfm/image_list.txt already exists, not rerunning OpenSfM setup
[WARNING] /var/www/data/814dcc26-558f-4a06-a6a0-9110460cb575/opensfm/exif already exists, not rerunning photo to metadata
[INFO]    running "/code/SuperBuild/install/bin/opensfm/bin/opensfm" detect_features "/var/www/data/814dcc26-558f-4a06-a6a0-9110460cb575/opensfm"
2022-01-10 01:44:14,864 INFO: Expecting to process 24 images.
2022-01-10 01:44:14,871 INFO: Reading data for image DJI_0374.JPG (queue-size=0
2022-01-10 01:44:14,872 INFO: Reading data for image DJI_0365.JPG (queue-size=0
2022-01-10 01:44:14,872 INFO: Reading data for image DJI_0361.JPG (queue-size=0
2022-01-10 01:44:14,872 INFO: Reading data for image DJI_0377.JPG (queue-size=0
2022-01-10 01:44:15,863 INFO: Reading data for image DJI_0393.JPG (queue-size=1
2022-01-10 01:44:15,863 INFO: Extracting ROOT_SIFT features for image DJI_0374.JPG
2022-01-10 01:44:15,912 INFO: Reading data for image DJI_0385.JPG (queue-size=1
2022-01-10 01:44:15,912 INFO: Extracting ROOT_SIFT features for image DJI_0377.JPG
2022-01-10 01:44:15,917 INFO: Reading data for image DJI_0386.JPG (queue-size=1
2022-01-10 01:44:15,918 INFO: Extracting ROOT_SIFT features for image DJI_0361.JPG
2022-01-10 01:44:15,926 INFO: Reading data for image DJI_0388.JPG (queue-size=1
2022-01-10 01:44:15,927 INFO: Extracting ROOT_SIFT features for image DJI_0365.JPG
2022-01-10 01:44:15,935 DEBUG: Computing sift with threshold 0.066
2022-01-10 01:44:15,996 DEBUG: Computing sift with threshold 0.066
2022-01-10 01:44:16,008 DEBUG: Computing sift with threshold 0.066
2022-01-10 01:44:16,014 DEBUG: Computing sift with threshold 0.066
2022-01-10 01:44:17,074 INFO: Reading data for image DJI_0370.JPG (queue-size=1
2022-01-10 01:44:17,075 INFO: Extracting ROOT_SIFT features for image DJI_0386.JPG
2022-01-10 01:44:17,142 DEBUG: Computing sift with threshold 0.066
2022-01-10 01:44:17,164 INFO: Reading data for image DJI_0356.JPG (queue-size=1
2022-01-10 01:44:17,164 INFO: Extracting ROOT_SIFT features for image DJI_0388.JPG
2022-01-10 01:44:17,209 INFO: Reading data for image DJI_0390.JPG (queue-size=1
2022-01-10 01:44:17,210 INFO: Extracting ROOT_SIFT features for image DJI_0385.JPG
2022-01-10 01:44:17,217 INFO: Reading data for image DJI_0375.JPG (queue-size=1
2022-01-10 01:44:17,220 INFO: Extracting ROOT_SIFT features for image DJI_0393.JPG
2022-01-10 01:44:17,244 DEBUG: Computing sift with threshold 0.066
2022-01-10 01:44:17,308 DEBUG: Computing sift with threshold 0.066
2022-01-10 01:44:17,310 DEBUG: Computing sift with threshold 0.066
2022-01-10 01:45:12,652 INFO: Extracting ROOT_SIFT features for image DJI_0370.JPG
/code/SuperBuild/install/bin/opensfm/bin/opensfm: line 12:   929 Killed                  "$PYTHON" "$DIR"/opensfm_main.py "$@"

===== Dumping Info for Geeks (developers need this to fix bugs) =====
Child returned 137
Traceback (most recent call last):
File "/code/stages/odm_app.py", line 94, in execute
self.first_stage.run()
File "/code/opendm/types.py", line 346, in run
self.next_stage.run(outputs)
File "/code/opendm/types.py", line 346, in run
self.next_stage.run(outputs)
File "/code/opendm/types.py", line 346, in run
self.next_stage.run(outputs)
File "/code/opendm/types.py", line 327, in run
self.process(self.args, outputs)
File "/code/stages/run_opensfm.py", line 35, in process
octx.feature_matching(self.rerun())
File "/code/opendm/osfm.py", line 321, in feature_matching
self.run('detect_features')
File "/code/opendm/osfm.py", line 34, in run
system.run('"%s" %s "%s"' %
File "/code/opendm/system.py", line 106, in run
raise SubprocessException("Child returned {}".format(retcode), retcode)
opendm.system.SubprocessException: Child returned 137

===== Done, human-readable information to follow... =====

[ERROR]   Whoops! You ran out of memory! Add more RAM to your computer, if you're using docker configure it to use more memory, for WSL2 make use of .wslconfig (https://docs.microsoft.com/en-us/windows/wsl/wsl-config#configure-global-options-with-wslconfig), resize your images, lower the quality settings or process the images using a cloud provider (e.g. https://webodm.net).
1 Like

Welcome!

Docker on Linux is a different beast than Docker on Windows backed by WSL2.

You should not really need to tune much, hence no real config file.

That being said, what does your system’s SWAP look like?

You might be able to reduce peak allocation by dropping your --max-concurrency down from 16, and/or adjusting --feature-quality down to medium.

1 Like

thanks!
Here is some info about my SWAP:

Filename				Type		Size		Used		Priority
/dev/nvme0n1p2                          partition	8191996		0		-2
               total        used        free      shared  buff/cache   available
Mem:           15458        3964        8791         111        2703       11055
Swap:           7999           0        7999

My swappiness value is set at 60.

I will try the tweaks you suggested and report back. I don’t like the idea of reducing feature quality though, as I have a pretty good machine and have done the same on a much less potent one. I think something is going wrong or I need to configure something, because AFAIK my computer should be able to do the task at good quality

1 Like

What was the lesser machine? How many threads? If it was fewer threads, that might be why it went through…

Swappiness 60 should be fine, and 8GB swap is okay as well. Do you have room to skooch that up to 16GB?

1 Like

I have 16GB of total installed system RAM
I think I can’t change swap size now because there I am using a specific swap partition (not a swap file)
The lesser machine was a 2014 Lenovo W550s with a 2.4GHz Intel Core i7-5500U (dual-core, 4MB cache, 2.0 GHZ, 2 core, 4 threads) and 16GB RAM
The actual machine is a 2021 MSI Alpha 17 with a AMD Ryzen 7 4800H 2.9GHz 8 core, 16 thread CPU, and 16GB RAM.

Do the GPUs have anything to do here?

So, I tried with --feature-quality down to slow, and --max-concurrency down to 8. Problem persisted. I attach new log.

[INFO]    Initializing ODM 2.7.0 - Mon Jan 10 16:59:20  2022
[INFO]    ==============
[INFO]    auto_boundary: True
[INFO]    boundary: {}
[INFO]    build_overviews: False
[INFO]    camera_lens: auto
[INFO]    cameras: {}
[INFO]    cog: True
[INFO]    copy_to: None
[INFO]    crop: 3
[INFO]    debug: False
[INFO]    dem_decimation: 1
[INFO]    dem_euclidean_map: False
[INFO]    dem_gapfill_steps: 3
[INFO]    dem_resolution: 5
[INFO]    depthmap_resolution: 640
[INFO]    dsm: False
[INFO]    dtm: False
[INFO]    end_with: odm_postprocess
[INFO]    fast_orthophoto: False
[INFO]    feature_quality: low
[INFO]    feature_type: sift
[INFO]    force_gps: False
[INFO]    gcp: None
[INFO]    geo: None
[INFO]    gps_accuracy: 10
[INFO]    ignore_gsd: False
[INFO]    matcher_neighbors: 8
[INFO]    matcher_type: flann
[INFO]    max_concurrency: 8
[INFO]    merge: all
[INFO]    mesh_octree_depth: 11
[INFO]    mesh_size: 200000
[INFO]    min_num_features: 8000
[INFO]    name: 814dcc26-558f-4a06-a6a0-9110460cb575
[INFO]    optimize_disk_space: False
[INFO]    orthophoto_compression: DEFLATE
[INFO]    orthophoto_cutline: False
[INFO]    orthophoto_kmz: False
[INFO]    orthophoto_no_tiled: False
[INFO]    orthophoto_png: False
[INFO]    orthophoto_resolution: 5
[INFO]    pc_classify: False
[INFO]    pc_csv: False
[INFO]    pc_ept: True
[INFO]    pc_filter: 2.5
[INFO]    pc_geometric: False
[INFO]    pc_las: False
[INFO]    pc_quality: medium
[INFO]    pc_rectify: False
[INFO]    pc_sample: 0
[INFO]    pc_tile: False
[INFO]    primary_band: auto
[INFO]    project_path: /var/www/data
[INFO]    radiometric_calibration: none
[INFO]    rerun: None
[INFO]    rerun_all: False
[INFO]    rerun_from: ['odm_postprocess']
[INFO]    resize_to: 2048
[INFO]    sfm_algorithm: incremental
[INFO]    skip_3dmodel: False
[INFO]    skip_band_alignment: False
[INFO]    skip_orthophoto: False
[INFO]    skip_report: False
[INFO]    sm_cluster: None
[INFO]    smrf_scalar: 1.25
[INFO]    smrf_slope: 0.15
[INFO]    smrf_threshold: 0.5
[INFO]    smrf_window: 18.0
[INFO]    split: 999999
[INFO]    split_image_groups: None
[INFO]    split_overlap: 150
[INFO]    texturing_data_term: gmi
[INFO]    texturing_keep_unseen_faces: False
[INFO]    texturing_outlier_removal_type: gauss_clamping
[INFO]    texturing_skip_global_seam_leveling: False
[INFO]    texturing_skip_local_seam_leveling: False
[INFO]    texturing_tone_mapping: none
[INFO]    tiles: False
[INFO]    time: False
[INFO]    use_3dmesh: False
[INFO]    use_exif: False
[INFO]    use_fixed_camera_params: False
[INFO]    use_hybrid_bundle_adjustment: False
[INFO]    verbose: False
[INFO]    ==============
[INFO]    Running dataset stage
[INFO]    Loading dataset from: /var/www/data/814dcc26-558f-4a06-a6a0-9110460cb575/images
[INFO]    Loading images database: /var/www/data/814dcc26-558f-4a06-a6a0-9110460cb575/images.json
[INFO]    Found 45 usable images
[INFO]    Coordinates file already exist: /var/www/data/814dcc26-558f-4a06-a6a0-9110460cb575/odm_georeferencing/coords.txt
[INFO]    Model geo file already exist: /var/www/data/814dcc26-558f-4a06-a6a0-9110460cb575/odm_georeferencing/odm_georeferencing_model_geo.txt
[INFO]    Parsing SRS header: WGS84 UTM 19S
[INFO]    Finished dataset stage
[INFO]    Running split stage
[INFO]    Normal dataset, will process all at once.
[INFO]    Finished split stage
[INFO]    Running merge stage
[INFO]    Normal dataset, nothing to merge.
[INFO]    Finished merge stage
[INFO]    Running opensfm stage
[WARNING] /var/www/data/814dcc26-558f-4a06-a6a0-9110460cb575/opensfm/image_list.txt already exists, not rerunning OpenSfM setup
[WARNING] /var/www/data/814dcc26-558f-4a06-a6a0-9110460cb575/opensfm/exif already exists, not rerunning photo to metadata
[INFO]    running "/code/SuperBuild/install/bin/opensfm/bin/opensfm" detect_features "/var/www/data/814dcc26-558f-4a06-a6a0-9110460cb575/opensfm"
2022-01-10 16:59:21,375 INFO: Expecting to process 27 images.
2022-01-10 16:59:21,381 INFO: Reading data for image DJI_0374.JPG (queue-size=0
2022-01-10 16:59:21,381 INFO: Reading data for image DJI_0365.JPG (queue-size=0
2022-01-10 16:59:21,382 INFO: Reading data for image DJI_0361.JPG (queue-size=0
2022-01-10 16:59:21,382 INFO: Reading data for image DJI_0377.JPG (queue-size=0
2022-01-10 16:59:22,290 INFO: Reading data for image DJI_0388.JPG (queue-size=1
2022-01-10 16:59:22,290 INFO: Extracting ROOT_SIFT features for image DJI_0365.JPG
2022-01-10 16:59:22,327 INFO: Reading data for image DJI_0393.JPG (queue-size=1
2022-01-10 16:59:22,328 INFO: Extracting ROOT_SIFT features for image DJI_0374.JPG
2022-01-10 16:59:22,364 DEBUG: Computing sift with threshold 0.066
2022-01-10 16:59:22,372 DEBUG: Computing sift with threshold 0.066
2022-01-10 16:59:22,392 INFO: Reading data for image DJI_0385.JPG (queue-size=1
2022-01-10 16:59:22,393 INFO: Extracting ROOT_SIFT features for image DJI_0377.JPG
2022-01-10 16:59:22,402 INFO: Reading data for image DJI_0386.JPG (queue-size=1
2022-01-10 16:59:22,402 INFO: Extracting ROOT_SIFT features for image DJI_0361.JPG
2022-01-10 16:59:22,465 DEBUG: Computing sift with threshold 0.066
2022-01-10 16:59:22,469 DEBUG: Computing sift with threshold 0.066
2022-01-10 16:59:23,402 INFO: Reading data for image DJI_0356.JPG (queue-size=1
2022-01-10 16:59:23,402 INFO: Extracting ROOT_SIFT features for image DJI_0388.JPG
2022-01-10 16:59:23,463 DEBUG: Computing sift with threshold 0.066
2022-01-10 16:59:23,641 INFO: Reading data for image DJI_0370.JPG (queue-size=1
2022-01-10 16:59:23,642 INFO: Extracting ROOT_SIFT features for image DJI_0386.JPG
2022-01-10 16:59:23,705 INFO: Reading data for image DJI_0390.JPG (queue-size=1
2022-01-10 16:59:23,705 INFO: Extracting ROOT_SIFT features for image DJI_0385.JPG
2022-01-10 16:59:23,706 DEBUG: Computing sift with threshold 0.066
2022-01-10 16:59:23,738 INFO: Reading data for image DJI_0375.JPG (queue-size=1
2022-01-10 16:59:23,739 INFO: Extracting ROOT_SIFT features for image DJI_0393.JPG
2022-01-10 16:59:23,797 DEBUG: Computing sift with threshold 0.066
2022-01-10 16:59:23,822 DEBUG: Computing sift with threshold 0.066
2022-01-10 16:59:24,562 INFO: Reading data for image DJI_0379.JPG (queue-size=1
2022-01-10 16:59:24,562 INFO: Extracting ROOT_SIFT features for image DJI_0356.JPG
2022-01-10 16:59:24,909 INFO: Reading data for image DJI_0357.JPG (queue-size=1
2022-01-10 16:59:24,911 INFO: Extracting ROOT_SIFT features for image DJI_0370.JPG
2022-01-10 16:59:24,915 DEBUG: Computing sift with threshold 0.066
2022-01-10 16:59:25,226 DEBUG: Computing sift with threshold 0.066
2022-01-10 16:59:25,337 INFO: Reading data for image DJI_0400.JPG (queue-size=1
2022-01-10 16:59:25,338 INFO: Extracting ROOT_SIFT features for image DJI_0390.JPG
2022-01-10 16:59:25,378 INFO: Reading data for image DJI_0397.JPG (queue-size=1
2022-01-10 16:59:25,381 INFO: Extracting ROOT_SIFT features for image DJI_0375.JPG
2022-01-10 16:59:25,581 DEBUG: Computing sift with threshold 0.066
2022-01-10 16:59:30,457 DEBUG: Computing sift with threshold 0.066
2022-01-10 17:00:25,139 INFO: Reading data for image DJI_0398.JPG (queue-size=1
/code/SuperBuild/install/bin/opensfm/bin/opensfm: line 12:   731 Killed                  "$PYTHON" "$DIR"/opensfm_main.py "$@"

===== Dumping Info for Geeks (developers need this to fix bugs) =====
Child returned 137
Traceback (most recent call last):
File "/code/stages/odm_app.py", line 94, in execute
self.first_stage.run()
File "/code/opendm/types.py", line 346, in run
self.next_stage.run(outputs)
File "/code/opendm/types.py", line 346, in run
self.next_stage.run(outputs)
File "/code/opendm/types.py", line 346, in run
self.next_stage.run(outputs)
File "/code/opendm/types.py", line 327, in run
self.process(self.args, outputs)
File "/code/stages/run_opensfm.py", line 35, in process
octx.feature_matching(self.rerun())
File "/code/opendm/osfm.py", line 321, in feature_matching
self.run('detect_features')
File "/code/opendm/osfm.py", line 34, in run
system.run('"%s" %s "%s"' %
File "/code/opendm/system.py", line 106, in run
raise SubprocessException("Child returned {}".format(retcode), retcode)
opendm.system.SubprocessException: Child returned 137

===== Done, human-readable information to follow... =====

[ERROR]   Whoops! You ran out of memory! Add more RAM to your computer, if you're using docker configure it to use more memory, for WSL2 make use of .wslconfig (https://docs.microsoft.com/en-us/windows/wsl/wsl-config#configure-global-options-with-wslconfig), resize your images, lower the quality settings or process the images using a cloud provider (e.g. https://webodm.net).
1 Like

I also note that system gets very slow (unusable for multitasking) when I am processing with ODM. If I remember correctly, this was not like that with my lesser machine.

EDIT: I also tried Firefox but problem persisted.

1 Like

Yes, the will be used for parts of the processing pipeline to help speed things up a bit provided a few factors are met.

Can you run with --max-concurrency 4 to match your older machine better?

You might be able to play with the “nice” or priority values a bit, but yeah when it is processing it will really take up your machine’s active CPU time.

If you need to be using that machine for work, you should probably cap the max-concurrency low to avoid resource contention.

Ok. I tried --max-concurrency 4 but problem persisted. So I deleted WebODM Folder, rebooted computer (I then tried connecting to localhost:8000 and it did open WebODM interface, so I stopped docker.service), and then I git cloned WebODM and started back docker and WebODM.
And Then I added the task and it worked! (low quality, max-concurrency 4). So I created a new project and task, with same settings, and did not work.
So I went back to original task (the one that did work), and ran it again, and it ran again OK!
So now I will try editing this original task and increasing quality to normal and then high and see what happens, and I will report back.

thanks!

ok so now it worked even on medium quality!

I really don´t know what is so special about this project in regards to the other ones…
I will now try high quality. Here is the medium quality succesful process log

... output truncated at undefined lines ...
Decimated faces 3404385 (57.84%, 19s, ETA 14s)...
Decimated faces 3417562 (58.06%, 19s, ETA 14s)...
Decimated faces 3430402 (58.28%, 19s, ETA 14s)...
Decimated faces 3443468 (58.50%, 19s, ETA 14s)...
Decimated faces 3456594 (58.72%, 20s, ETA 14s)...
Decimated faces 3469726 (58.95%, 20s, ETA 14s)...
Decimated faces 3482172 (59.16%, 22s, ETA 15s)...
Decimated faces 3511502 (59.66%, 22s, ETA 15s)...
Decimated faces 3539686 (60.14%, 22s, ETA 14s)...
Decimated faces 3567828 (60.61%, 22s, ETA 14s)...
Decimated faces 3594498 (61.07%, 22s, ETA 14s)...
Decimated faces 3622241 (61.54%, 22s, ETA 14s)...
Decimated faces 3649520 (62.00%, 22s, ETA 13s)...
Decimated faces 3676576 (62.46%, 22s, ETA 13s)...
Decimated faces 3703176 (62.91%, 22s, ETA 13s)...
Decimated faces 3729524 (63.36%, 22s, ETA 13s)...
Decimated faces 3755826 (63.81%, 23s, ETA 13s)...
Decimated faces 3781651 (64.25%, 23s, ETA 12s)...
Decimated faces 3807634 (64.69%, 23s, ETA 12s)...
Decimated faces 3833393 (65.13%, 23s, ETA 12s)...
Decimated faces 3857962 (65.54%, 23s, ETA 12s)...
Decimated faces 3882740 (65.96%, 23s, ETA 12s)...
Decimated faces 3907293 (66.38%, 23s, ETA 12s)...
Decimated faces 3931769 (66.80%, 23s, ETA 11s)...
Decimated faces 3955731 (67.20%, 23s, ETA 11s)...
Decimated faces 3979609 (67.61%, 23s, ETA 11s)...
Decimated faces 4002261 (68.00%, 24s, ETA 11s)...
Decimated faces 4025255 (68.39%, 24s, ETA 11s)...
Decimated faces 4047594 (68.77%, 24s, ETA 11s)...
Decimated faces 4069620 (69.14%, 24s, ETA 10s)...
Decimated faces 4090358 (69.49%, 24s, ETA 10s)...
Decimated faces 4112038 (69.86%, 24s, ETA 10s)...
Decimated faces 4133050 (70.22%, 24s, ETA 10s)...
Decimated faces 4154426 (70.58%, 24s, ETA 10s)...
Decimated faces 4175501 (70.94%, 24s, ETA 10s)...
Decimated faces 4195836 (71.28%, 24s, ETA 10s)...
Decimated faces 4214578 (71.60%, 25s, ETA 9s)...
Decimated faces 4234312 (71.94%, 25s, ETA 9s)...
Decimated faces 4254222 (72.28%, 25s, ETA 9s)...
Decimated faces 4273787 (72.61%, 25s, ETA 9s)...
Decimated faces 4293154 (72.94%, 25s, ETA 9s)...
Decimated faces 4313206 (73.28%, 25s, ETA 9s)...
Decimated faces 4332694 (73.61%, 25s, ETA 9s)...
Decimated faces 4351754 (73.93%, 25s, ETA 9s)...
Decimated faces 4370603 (74.25%, 25s, ETA 8s)...
Decimated faces 4388960 (74.56%, 26s, ETA 8s)...
Decimated faces 4408052 (74.89%, 26s, ETA 8s)...
Decimated faces 4427075 (75.21%, 26s, ETA 8s)...
Decimated faces 4445898 (75.53%, 26s, ETA 8s)...
Decimated faces 4464328 (75.85%, 26s, ETA 8s)...
Decimated faces 4482359 (76.15%, 26s, ETA 8s)...
Decimated faces 4500819 (76.47%, 26s, ETA 8s)...
Decimated faces 4518826 (76.77%, 26s, ETA 8s)...
Decimated faces 4536692 (77.07%, 26s, ETA 7s)...
Decimated faces 4554247 (77.37%, 26s, ETA 7s)...
Decimated faces 4571555 (77.67%, 27s, ETA 7s)...
Decimated faces 4588837 (77.96%, 27s, ETA 7s)...
Decimated faces 4604391 (78.22%, 27s, ETA 7s)...
Decimated faces 4619798 (78.49%, 27s, ETA 7s)...
Decimated faces 4635418 (78.75%, 27s, ETA 7s)...
Decimated faces 4651072 (79.02%, 27s, ETA 7s)...
Decimated faces 4666609 (79.28%, 27s, ETA 7s)...
Decimated faces 4683205 (79.56%, 27s, ETA 7s)...
Decimated faces 4698741 (79.83%, 27s, ETA 7s)...
Decimated faces 4714433 (80.09%, 27s, ETA 6s)...
Decimated faces 4729734 (80.35%, 28s, ETA 6s)...
Decimated faces 4745896 (80.63%, 28s, ETA 6s)...
Decimated faces 4761818 (80.90%, 28s, ETA 6s)...
Decimated faces 4777679 (81.17%, 28s, ETA 6s)...
Decimated faces 4793197 (81.43%, 28s, ETA 6s)...
Decimated faces 4807394 (81.67%, 28s, ETA 6s)...
Decimated faces 4822276 (81.93%, 28s, ETA 6s)...
Decimated faces 4836714 (82.17%, 28s, ETA 6s)...
Decimated faces 4850878 (82.41%, 28s, ETA 6s)...
Decimated faces 4865266 (82.66%, 28s, ETA 6s)...
Decimated faces 4879284 (82.90%, 29s, ETA 5s)...
Decimated faces 4893192 (83.13%, 29s, ETA 5s)...
Decimated faces 4905819 (83.35%, 29s, ETA 5s)...
Decimated faces 4918469 (83.56%, 29s, ETA 5s)...
Decimated faces 4930577 (83.77%, 29s, ETA 5s)...
Decimated faces 4942241 (83.96%, 29s, ETA 5s)...
Decimated faces 4954913 (84.18%, 29s, ETA 5s)...
Decimated faces 4968443 (84.41%, 29s, ETA 5s)...
Decimated faces 4981887 (84.64%, 29s, ETA 5s)...
Decimated faces 4985991 (84.71%, 30s, ETA 5s)...
Decimated faces 5010379 (85.12%, 30s, ETA 5s)...
Decimated faces 5034163 (85.53%, 30s, ETA 5s)...
Decimated faces 5057597 (85.92%, 31s, ETA 5s)...
Decimated faces 5081774 (86.34%, 31s, ETA 4s)...
Decimated faces 5105560 (86.74%, 31s, ETA 4s)...
Decimated faces 5129011 (87.14%, 31s, ETA 4s)...
Decimated faces 5151796 (87.52%, 31s, ETA 4s)...
Decimated faces 5173982 (87.90%, 31s, ETA 4s)...
Decimated faces 5196375 (88.28%, 31s, ETA 4s)...
Decimated faces 5218591 (88.66%, 31s, ETA 4s)...
Decimated faces 5240231 (89.03%, 31s, ETA 3s)...
Decimated faces 5261716 (89.39%, 31s, ETA 3s)...
Decimated faces 5282824 (89.75%, 32s, ETA 3s)...
Decimated faces 5303612 (90.10%, 32s, ETA 3s)...
Decimated faces 5323922 (90.45%, 32s, ETA 3s)...
Decimated faces 5343962 (90.79%, 32s, ETA 3s)...
Decimated faces 5363283 (91.12%, 32s, ETA 3s)...
Decimated faces 5382270 (91.44%, 32s, ETA 3s)...
Decimated faces 5401073 (91.76%, 32s, ETA 2s)...
Decimated faces 5419127 (92.07%, 32s, ETA 2s)...
Decimated faces 5437234 (92.37%, 32s, ETA 2s)...
Decimated faces 5454884 (92.67%, 32s, ETA 2s)...
Decimated faces 5472315 (92.97%, 33s, ETA 2s)...
Decimated faces 5489395 (93.26%, 33s, ETA 2s)...
Decimated faces 5506179 (93.55%, 33s, ETA 2s)...
Decimated faces 5522503 (93.82%, 33s, ETA 2s)...
Decimated faces 5538552 (94.10%, 33s, ETA 2s)...
Decimated faces 5553813 (94.35%, 33s, ETA 2s)...
Decimated faces 5569403 (94.62%, 33s, ETA 1s)...
Decimated faces 5584489 (94.88%, 33s, ETA 1s)...
Decimated faces 5599265 (95.13%, 33s, ETA 1s)...
Decimated faces 5613849 (95.37%, 33s, ETA 1s)...
Decimated faces 5628061 (95.62%, 34s, ETA 1s)...
Decimated faces 5642035 (95.85%, 34s, ETA 1s)...
Decimated faces 5655773 (96.09%, 34s, ETA 1s)...
Decimated faces 5669205 (96.32%, 34s, ETA 1s)...
Decimated faces 5682042 (96.53%, 34s, ETA 1s)...
Decimated faces 5694071 (96.74%, 34s, ETA 1s)...
Decimated faces 5706329 (96.95%, 34s, ETA 1s)...
Decimated faces 5718165 (97.15%, 34s, ETA 1s)...
Decimated faces 5723971 (97.25%, 35s, ETA 996ms)...
Decimated faces 5746090 (97.62%, 35s, ETA 859ms)...
Decimated faces 5767761 (97.99%, 35s, ETA 725ms)...
Decimated faces 5787721 (98.33%, 35s, ETA 602ms)...
Decimated faces 5808809 (98.69%, 35s, ETA 473ms)...
Decimated faces 5829429 (99.04%, 35s, ETA 346ms)...
Decimated faces 5849406 (99.38%, 35s, ETA 224ms)...
Decimated faces 5868818 (99.71%, 35s, ETA 105ms)...
Decimated faces 5886097 (100%, 35s970ms)
22:01:11 [App     ] MEMORYINFO: {
22:01:11 [App     ] 	VmPeak:	 2084084 kB
22:01:11 [App     ] 	VmSize:	  741392 kB
22:01:11 [App     ] } ENDINFO
[INFO]    Writing ODM 2.5D Mesh file in: /var/www/data/3320f153-0262-4c9a-90c5-ba8222f91d50/odm_meshing/odm_25dmesh.ply
[INFO]    ODM 2.5D DSM resolution: 0.05
[INFO]    Created temporary directory: /var/www/data/3320f153-0262-4c9a-90c5-ba8222f91d50/odm_meshing/tmp
[INFO]    Creating DSM for 2.5D mesh
[INFO]    running pdal info "/var/www/data/3320f153-0262-4c9a-90c5-ba8222f91d50/odm_filterpoints/point_cloud.ply" > "/tmp/tmpzgpss7lg.json"
[INFO]    Point cloud bounds are [minx: -88.77968597, maxx: 66.72105408] [miny: -92.36439514, maxy: 78.40961456]
[INFO]    DEM resolution is (3416, 3111), max tile size is 4096, will split DEM generation into 1 tiles
[INFO]    Generating /var/www/data/3320f153-0262-4c9a-90c5-ba8222f91d50/odm_meshing/tmp/mesh_dsm_r0.07071067811865477_x0_y0.tif (max, radius: 0.07071067811865477, resolution: 0.05)
[INFO]    running pdal pipeline -i /tmp/tmpapk5aq1m.json > /dev/null 2>&1
[INFO]    running gdalbuildvrt -input_file_list "/var/www/data/3320f153-0262-4c9a-90c5-ba8222f91d50/odm_meshing/tmp/tiles_list.txt" "/var/www/data/3320f153-0262-4c9a-90c5-ba8222f91d50/odm_meshing/tmp/tiles.vrt"
0...10...20...30...40...50...60...70...80...90...100 - done.
[INFO]    running gdal_translate -co NUM_THREADS=4 -co BIGTIFF=IF_SAFER --config GDAL_CACHEMAX 24.5% "/var/www/data/3320f153-0262-4c9a-90c5-ba8222f91d50/odm_meshing/tmp/tiles.vrt" "/var/www/data/3320f153-0262-4c9a-90c5-ba8222f91d50/odm_meshing/tmp/tiles.tmp.tif"
Input file size is 3111, 3416
0...10...20...30...40...50...60...70...80...90...100 - done.
[INFO]    running gdal_translate -co NUM_THREADS=4 -co BIGTIFF=IF_SAFER --config GDAL_CACHEMAX 24.5% -outsize 10% 0 "/var/www/data/3320f153-0262-4c9a-90c5-ba8222f91d50/odm_meshing/tmp/tiles.tmp.tif" "/var/www/data/3320f153-0262-4c9a-90c5-ba8222f91d50/odm_meshing/tmp/tiles.small.tif"
0
.
.
.
10
.
.
.
20
.
.
.
30
.
.
.
40
.
.
.
50
.
.
.
60
.
.
.
70
.
.
.
80
.
.
.
90
.
.
.
Input file size is 3111, 3416
0...10...20...30...40...50...60...70...80...90...100 - done.
[INFO]    running gdalbuildvrt -resolution highest -r bilinear "/var/www/data/3320f153-0262-4c9a-90c5-ba8222f91d50/odm_meshing/tmp/merged.vrt" "/var/www/data/3320f153-0262-4c9a-90c5-ba8222f91d50/odm_meshing/tmp/tiles.small_filled.tif" "/var/www/data/3320f153-0262-4c9a-90c5-ba8222f91d50/odm_meshing/tmp/tiles.tmp.tif"
0...10...20...30...40...50...60...70...80...90...100 - done.
[INFO]    running gdal_translate -co NUM_THREADS=4 -co TILED=YES -co BIGTIFF=IF_SAFER -co COMPRESS=DEFLATE --config GDAL_CACHEMAX 24.5% "/var/www/data/3320f153-0262-4c9a-90c5-ba8222f91d50/odm_meshing/tmp/merged.vrt" "/var/www/data/3320f153-0262-4c9a-90c5-ba8222f91d50/odm_meshing/tmp/tiles.tif"
Input file size is 3111, 3416
0...10...20...30...40...50...60...70...80...90...100 - done.
[INFO]    Starting smoothing...
[INFO]    Smoothing iteration 1
[INFO]    Completed smoothing to create /var/www/data/3320f153-0262-4c9a-90c5-ba8222f91d50/odm_meshing/tmp/mesh_dsm.tif in 0:00:03.405764
[INFO]    Completed mesh_dsm.tif in 0:00:14.977125
[INFO]    Creating mesh from DSM: /var/www/data/3320f153-0262-4c9a-90c5-ba8222f91d50/odm_meshing/tmp/mesh_dsm.tif
[INFO]    running "/code/SuperBuild/install/bin/dem2mesh" -inputFile "/var/www/data/3320f153-0262-4c9a-90c5-ba8222f91d50/odm_meshing/tmp/mesh_dsm.tif" -outputFile "/var/www/data/3320f153-0262-4c9a-90c5-ba8222f91d50/odm_meshing/odm_25dmesh.dirty.ply" -maxTileLength 2000 -maxVertexCount 200000 -maxConcurrency 4
[INFO]    running "/code/SuperBuild/install/bin/OpenMVS/ReconstructMesh" -i "/var/www/data/3320f153-0262-4c9a-90c5-ba8222f91d50/odm_meshing/odm_25dmesh.dirty.ply" -o "/var/www/data/3320f153-0262-4c9a-90c5-ba8222f91d50/odm_meshing/odm_25dmesh.ply" --remove-spikes 0 --remove-spurious 0 --smooth 0 --target-face-num 400000 -v 0
22:02:00 [App     ] Build date: Jan  9 2022, 00:27:19
22:02:00 [App     ] CPU: AMD Ryzen 7 4800H with Radeon Graphics          (16 cores)
22:02:00 [App     ] RAM: 15.10GB Physical Memory 7.81GB Virtual Memory
22:02:00 [App     ] OS: Linux 5.15.13-arch1-1 (x86_64)
22:02:00 [App     ] SSE & AVX compatible CPU & OS detected
22:02:00 [App     ] Command line: -i /var/www/data/3320f153-0262-4c9a-90c5-ba8222f91d50/odm_meshing/odm_25dmesh.dirty.ply -o /var/www/data/3320f153-0262-4c9a-90c5-ba8222f91d50/odm_meshing/odm_25dmesh.ply --remove-spikes 0 --remove-spurious 0 --smooth 0 --target-face-num 400000 -v 0
22:02:02 [App     ] MEMORYINFO: {
22:02:02 [App     ] 	VmPeak:	  187224 kB
22:02:02 [App     ] 	VmSize:	  105972 kB
22:02:02 [App     ] } ENDINFO
[INFO]    Finished odm_meshing stage
[INFO]    Running mvs_texturing stage
[INFO]    Writing MVS Textured file in: /var/www/data/3320f153-0262-4c9a-90c5-ba8222f91d50/odm_texturing/odm_textured_model_geo.obj
[INFO]    running "/code/SuperBuild/install/bin/texrecon" "/var/www/data/3320f153-0262-4c9a-90c5-ba8222f91d50/opensfm/undistorted/reconstruction.nvm" "/var/www/data/3320f153-0262-4c9a-90c5-ba8222f91d50/odm_meshing/odm_mesh.ply" "/var/www/data/3320f153-0262-4c9a-90c5-ba8222f91d50/odm_texturing/odm_textured_model_geo" -d gmi -o gauss_clamping -t none --no_intermediate_results      --max_texture_size=8192
/code/SuperBuild/install/bin/texrecon (built on Jan  9 2022, 00:26:03)
Load and prepare mesh:
Reading PLY: 191172 verts... 382467 faces... done.
Generating texture views:
NVM: Loading file...
NVM: Number of views: 45
NVM: Number of features: 51035


Loading 100%... done. (Took 0.356s)
Building adjacency graph:


Adding edges 100%... done. (Took 0.542s)
573693 total edges.
View selection:
Building BVH from 382467 faces... done. (Took: 246 ms)


Calculating face qualities 100%... done. (Took 1.334s)


Postprocessing face infos 100%... done. (Took 0.102s)
Maximum quality of a face within an image: 496.909
Clamping qualities to 81.7994 within normalization.
Optimizing:
Time[s]	Energy
0	336072
1	332892
2	326607
2	323414
2	321862
2	321087
3	320548
3	320276
3	319970
4	319764
4	319644
4	319550
4	319483
5	319421
5	319349
63100 faces have not been seen
Took: 7.697s
Generating texture patches:
Running... done. (Took 0.507s)
3913 texture patches.
Running global seam leveling:
Create matrices for optimization...  done.
Lhs dimensionality: 192329 x 192329
Calculating adjustments:
Color channel 0: CG took 91 iterations. Residual is 9.22422e-05
Color channel 1: CG took 91 iterations. Residual is 9.32693e-05
Color channel 2: CG took 91 iterations. Residual is 9.26921e-05
Took 0.342 seconds


Adjusting texture patches 100%... done. (Took 0.318s)
Running local seam leveling:


Blending texture patches 100%... done. (Took 10.16s)
Generating texture atlases:
Sorting texture patches... done.

Working on atlas 18 100%... done.
Finalizing texture atlases... done. (Took: 0s)
Building objmodel:
Saving model... done.
Whole texturing procedure took: 37.28s
[INFO]    Writing MVS Textured file in: /var/www/data/3320f153-0262-4c9a-90c5-ba8222f91d50/odm_texturing_25d/odm_textured_model_geo.obj
[INFO]    running "/code/SuperBuild/install/bin/texrecon" "/var/www/data/3320f153-0262-4c9a-90c5-ba8222f91d50/opensfm/undistorted/reconstruction.nvm" "/var/www/data/3320f153-0262-4c9a-90c5-ba8222f91d50/odm_meshing/odm_25dmesh.ply" "/var/www/data/3320f153-0262-4c9a-90c5-ba8222f91d50/odm_texturing_25d/odm_textured_model_geo" -d gmi -o gauss_clamping -t none --no_intermediate_results    --nadir_mode  --max_texture_size=8192
/code/SuperBuild/install/bin/texrecon (built on Jan  9 2022, 00:26:03)
Load and prepare mesh:
Reading PLY: 200401 verts... 399997 faces... done.
Generating texture views:
NVM: Loading file...
NVM: Number of views: 45
NVM: Number of features: 51035


Loading 100%... done. (Took 0.296s)
Building adjacency graph:


Adding edges 100%... done. (Took 0.444s)
599591 total edges.
View selection:
Building BVH from 399997 faces... done. (Took: 256 ms)


Calculating face qualities 100%... done. (Took 0.933s)


Postprocessing face infos 100%... done. (Took 0.125s)
Maximum quality of a face within an image: 0.99999
Clamping qualities to 0.992589 within normalization.
Optimizing:
Time[s]	Energy
0	121726
0	121348
1	116801
1	114591
1	113614
2	113117
2	112871
2	112799
3	112641
3	112536
3	112485
4	112449
4	112410
4	112371
5	112352
19487 faces have not been seen
Took: 6.836s
Generating texture patches:
Running... done. (Took 0.485s)
3708 texture patches.
Running global seam leveling:
Create matrices for optimization...  done.
Lhs dimensionality: 212508 x 212508
Calculating adjustments:
Color channel 0: CG took 83 iterations. Residual is 9.86271e-05
Color channel 1: CG took 83 iterations. Residual is 9.63827e-05
Color channel 2: CG took 87 iterations. Residual is 9.10043e-05
Took 0.339 seconds


Adjusting texture patches 100%... done. (Took 0.376s)
Running local seam leveling:


Blending texture patches 100%... done. (Took 6.668s)
Generating texture atlases:
Sorting texture patches... done.

Working on atlas 11 100%... done.
Finalizing texture atlases... done. (Took: 0s)
Building objmodel:
Saving model... done.
Whole texturing procedure took: 34.087s
[INFO]    Finished mvs_texturing stage
[INFO]    Running odm_georeferencing stage
[INFO]    Georeferencing point cloud
[INFO]    running pdal translate -i "/var/www/data/3320f153-0262-4c9a-90c5-ba8222f91d50/odm_filterpoints/point_cloud.ply" -o "/var/www/data/3320f153-0262-4c9a-90c5-ba8222f91d50/odm_georeferencing/odm_georeferenced_model.laz" ferry transformation --filters.ferry.dimensions="views => UserData" --writers.las.compression="lazip" --filters.transformation.matrix="1 0 0 287688.0 0 1 0 6348801.0 0 0 1 0 0 0 0 1" --writers.las.offset_x=287688.0 --writers.las.offset_y=6348801.0 --writers.las.offset_z=0 --writers.las.a_srs="+proj=utm +zone=19 +south +datum=WGS84 +units=m +no_defs +type=crs"
[INFO]    Calculating cropping area and generating bounds shapefile from point cloud
[INFO]    running pdal translate -i "/var/www/data/3320f153-0262-4c9a-90c5-ba8222f91d50/odm_georeferencing/odm_georeferenced_model.laz" -o "/var/www/data/3320f153-0262-4c9a-90c5-ba8222f91d50/odm_georeferencing/odm_georeferenced_model.decimated.las" decimation --filters.decimation.step=40
[INFO]    running pdal info --boundary --filters.hexbin.edge_size=1 --filters.hexbin.threshold=0 "/var/www/data/3320f153-0262-4c9a-90c5-ba8222f91d50/odm_georeferencing/odm_georeferenced_model.decimated.las" > "/var/www/data/3320f153-0262-4c9a-90c5-ba8222f91d50/odm_georeferencing/odm_georeferenced_model.boundary.json"
[INFO]    running pdal info --summary "/var/www/data/3320f153-0262-4c9a-90c5-ba8222f91d50/odm_georeferencing/odm_georeferenced_model.laz" > "/var/www/data/3320f153-0262-4c9a-90c5-ba8222f91d50/odm_georeferencing/odm_georeferenced_model.summary.json"
[INFO]    running ogr2ogr -overwrite -f GPKG -a_srs "+proj=utm +zone=19 +south +datum=WGS84 +units=m +no_defs" /var/www/data/3320f153-0262-4c9a-90c5-ba8222f91d50/odm_georeferencing/odm_georeferenced_model.bounds.gpkg /var/www/data/3320f153-0262-4c9a-90c5-ba8222f91d50/odm_georeferencing/odm_georeferenced_model.bounds.geojson
[INFO]    Creating Entwine Point Tile output
[WARNING] Removing previous EPT directory: /var/www/data/3320f153-0262-4c9a-90c5-ba8222f91d50/entwine_pointcloud
[INFO]    running entwine build --threads 4 --tmp "/var/www/data/3320f153-0262-4c9a-90c5-ba8222f91d50/entwine_pointcloud-tmp" -i /var/www/data/3320f153-0262-4c9a-90c5-ba8222f91d50/odm_georeferencing/odm_georeferenced_model.laz -o "/var/www/data/3320f153-0262-4c9a-90c5-ba8222f91d50/entwine_pointcloud"
1/1: /var/www/data/3320f153-0262-4c9a-90c5-ba8222f91d50/odm_georeferencing/odm_georeferenced_model.laz
Dimensions: [
X:int32, Y:int32, Z:int32, Intensity:uint16, ReturnNumber:uint8,
NumberOfReturns:uint8, ScanDirectionFlag:uint8, EdgeOfFlightLine:uint8,
Classification:uint8, ScanAngleRank:float32, UserData:uint8,
PointSourceId:uint16, GpsTime:float64, Red:uint16, Green:uint16, Blue:uint16
]
Points: 4,989,988
Bounds: [(287599, 6348708, -11), (287755, 6348880, 18)]
Scale: 0.01
SRS: EPSG:32719

Adding 0 - /var/www/data/3320f153-0262-4c9a-90c5-ba8222f91d50/odm_georeferencing/odm_georeferenced_model.laz
Joining
Done 0
00:10 - 100% - 4,989,988 - 1,796 (1,796) M/h - 114W - 0R - 17A
Saving
Wrote 4,989,988 points.
[INFO]    Finished odm_georeferencing stage
[INFO]    Running odm_dem stage
[INFO]    Classify: False
[INFO]    Create DSM: False
[INFO]    Create DTM: False
[INFO]    DEM input file /var/www/data/3320f153-0262-4c9a-90c5-ba8222f91d50/odm_georeferencing/odm_georeferenced_model.laz found: True
[WARNING] DEM will not be generated
[INFO]    Finished odm_dem stage
[INFO]    Running odm_orthophoto stage
[INFO]    running "/code/SuperBuild/install/bin/odm_orthophoto" -inputFiles /var/www/data/3320f153-0262-4c9a-90c5-ba8222f91d50/odm_texturing_25d/odm_textured_model_geo.obj -logFile "/var/www/data/3320f153-0262-4c9a-90c5-ba8222f91d50/odm_orthophoto/odm_orthophoto_log.txt" -outputFile "/var/www/data/3320f153-0262-4c9a-90c5-ba8222f91d50/odm_orthophoto/odm_orthophoto_render.tif" -resolution 20.0  -outputCornerFile "/var/www/data/3320f153-0262-4c9a-90c5-ba8222f91d50/odm_orthophoto/odm_orthophoto_corners.txt"
[INFO]    Creating GeoTIFF
[INFO]    running gdal_translate -a_ullr 287599.2203140259 6348879.43560791 287754.6703109741 6348708.68560791 -co TILED=YES -co COMPRESS=DEFLATE -co PREDICTOR=2 -co BIGTIFF=IF_SAFER -co BLOCKXSIZE=512 -co BLOCKYSIZE=512 -co NUM_THREADS=4 -a_srs "+proj=utm +zone=19 +south +datum=WGS84 +units=m +no_defs +type=crs" --config GDAL_CACHEMAX 25.15% --config GDAL_TIFF_INTERNAL_MASK YES "/var/www/data/3320f153-0262-4c9a-90c5-ba8222f91d50/odm_orthophoto/odm_orthophoto_render.tif" "/var/www/data/3320f153-0262-4c9a-90c5-ba8222f91d50/odm_orthophoto/odm_orthophoto.tif" > "/var/www/data/3320f153-0262-4c9a-90c5-ba8222f91d50/odm_orthophoto/gdal_translate_log.txt"
[INFO]    Cropping /var/www/data/3320f153-0262-4c9a-90c5-ba8222f91d50/odm_orthophoto/odm_orthophoto.tif
[INFO]    running gdalwarp -cutline /var/www/data/3320f153-0262-4c9a-90c5-ba8222f91d50/odm_georeferencing/odm_georeferenced_model.bounds.gpkg -crop_to_cutline -co TILED=YES -co COMPRESS=DEFLATE -co PREDICTOR=2 -co BIGTIFF=IF_SAFER -co BLOCKXSIZE=512 -co BLOCKYSIZE=512 -co NUM_THREADS=4 -dstalpha /var/www/data/3320f153-0262-4c9a-90c5-ba8222f91d50/odm_orthophoto/odm_orthophoto.original.tif /var/www/data/3320f153-0262-4c9a-90c5-ba8222f91d50/odm_orthophoto/odm_orthophoto.tif --config GDAL_CACHEMAX 25.15%
Using band 4 of source image as alpha.
Creating output file that is 3038P x 3334L.
Processing /var/www/data/3320f153-0262-4c9a-90c5-ba8222f91d50/odm_orthophoto/odm_orthophoto.original.tif [1/1] : 0...10...20...30...40...50...60...70...80...90...100 - done.
[INFO]    Optimizing /var/www/data/3320f153-0262-4c9a-90c5-ba8222f91d50/odm_orthophoto/odm_orthophoto.tif as Cloud Optimized GeoTIFF
[INFO]    running gdal_translate -of COG -co NUM_THREADS=4 -co BLOCKSIZE=256 -co COMPRESS=DEFLATE -co PREDICTOR=2 -co BIGTIFF=IF_SAFER -co RESAMPLING=NEAREST --config GDAL_CACHEMAX 25.2% --config GDAL_NUM_THREADS 4 "/var/www/data/3320f153-0262-4c9a-90c5-ba8222f91d50/odm_orthophoto/odm_orthophoto.tif" "/var/www/data/3320f153-0262-4c9a-90c5-ba8222f91d50/odm_orthophoto/odm_orthophoto_cogeo.tif"
Input file size is 3038, 3334
0...10...20...30...40...50...60...70...80...90...100 - done.
[INFO]    Finished odm_orthophoto stage
[INFO]    Running odm_report stage
[INFO]    Exporting shots.geojson
[INFO]    Wrote /var/www/data/3320f153-0262-4c9a-90c5-ba8222f91d50/odm_report/shots.geojson
[INFO]    running pdal info --dimensions "X,Y,Z" "/var/www/data/3320f153-0262-4c9a-90c5-ba8222f91d50/odm_georeferencing/odm_georeferenced_model.laz" > "/var/www/data/3320f153-0262-4c9a-90c5-ba8222f91d50/odm_georeferencing/odm_georeferenced_model.info.json"
[INFO]    running pdal translate -i "/var/www/data/3320f153-0262-4c9a-90c5-ba8222f91d50/odm_georeferencing/odm_georeferenced_model.laz" -o "/var/www/data/3320f153-0262-4c9a-90c5-ba8222f91d50/opensfm/stats/overlap.tif" --writer gdal --writers.gdal.resolution=0.12197857142891735 --writers.gdal.data_type=uint8_t --writers.gdal.dimension=UserData --writers.gdal.output_type=max --writers.gdal.radius=0.17250375003367024
[INFO]    Cropping /var/www/data/3320f153-0262-4c9a-90c5-ba8222f91d50/opensfm/stats/overlap.tif
[INFO]    running gdalwarp -cutline /var/www/data/3320f153-0262-4c9a-90c5-ba8222f91d50/odm_georeferencing/odm_georeferenced_model.bounds.gpkg -crop_to_cutline -co TILED=YES -co COMPRESS=DEFLATE -co PREDICTOR=2 -co BIGTIFF=IF_SAFER -co BLOCKXSIZE=512 -co BLOCKYSIZE=512 -co NUM_THREADS=4  /var/www/data/3320f153-0262-4c9a-90c5-ba8222f91d50/opensfm/stats/overlap.original.tif /var/www/data/3320f153-0262-4c9a-90c5-ba8222f91d50/opensfm/stats/overlap.tif --config GDAL_CACHEMAX 25.0%
Creating output file that is 1244P x 1366L.
Processing /var/www/data/3320f153-0262-4c9a-90c5-ba8222f91d50/opensfm/stats/overlap.original.tif [1/1] : 0Using internal nodata values (e.g. 255) for image /var/www/data/3320f153-0262-4c9a-90c5-ba8222f91d50/opensfm/stats/overlap.original.tif.
Copying nodata values from source /var/www/data/3320f153-0262-4c9a-90c5-ba8222f91d50/opensfm/stats/overlap.original.tif to destination /var/www/data/3320f153-0262-4c9a-90c5-ba8222f91d50/opensfm/stats/overlap.tif.
...10...20...30...40...50...60...70...80...90...100 - done.
[INFO]    running gdaldem color-relief "/var/www/data/3320f153-0262-4c9a-90c5-ba8222f91d50/opensfm/stats/overlap.tif" "/code/opendm/report/overlap_color_map.txt" "/var/www/data/3320f153-0262-4c9a-90c5-ba8222f91d50/opensfm/stats/overlap.png" -of PNG -alpha
0...10...20...30...40...50...60...70...80...90...100 - done.
[INFO]    running gdal_translate -of png "/var/www/data/3320f153-0262-4c9a-90c5-ba8222f91d50/odm_orthophoto/odm_orthophoto.tif" "/var/www/data/3320f153-0262-4c9a-90c5-ba8222f91d50/opensfm/stats/ortho.png" -outsize 1400 0  --config GDAL_CACHEMAX 25.0%
Input file size is 3038, 3334
0...10...20...30...40...50...60...70...80...90...100 - done.
[INFO]    Exporting report to /var/www/data/3320f153-0262-4c9a-90c5-ba8222f91d50/odm_report/report.pdf
2022-01-10 22:04:25,811 DEBUG: Final doc sections size summary:
2022-01-10 22:04:25,812 DEBUG: - header.size: 9.0B
2022-01-10 22:04:25,813 DEBUG: - pages.size: 4.0KiB
2022-01-10 22:04:25,813 DEBUG: - resources.fonts.size: 199.0B
2022-01-10 22:04:25,818 DEBUG: - resources.images.size: 6.2MiB
2022-01-10 22:04:25,818 DEBUG: - resources.dict.size: 204.0B
2022-01-10 22:04:25,818 DEBUG: - info.size: 55.0B
2022-01-10 22:04:25,818 DEBUG: - catalog.size: 104.0B
2022-01-10 22:04:25,818 DEBUG: - xref.size: 550.0B
2022-01-10 22:04:25,818 DEBUG: - trailer.size: 67.0B
[INFO]    Finished odm_report stage
[INFO]    Running odm_postprocess stage
[INFO]    Post Processing
[INFO]    Finished odm_postprocess stage
[INFO]    No more stages to run
[INFO]    MMMMMMMMMMMNNNMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMNNNMMMMMMMMMMM
[INFO]    MMMMMMdo:..---../sNMMMMMMMMMMMMMMMMMMMMMMMMMMNs/..---..:odMMMMMM
[INFO]    MMMMy-.odNMMMMMNy/`/mMMMMMMMMMMMMMMMMMMMMMMm/`/hNMMMMMNdo.-yMMMM
[INFO]    MMN/`sMMMMMMMMMNNMm/`yMMMMMMMMMMMMMMMMMMMMy`/mMNNMMMMMMMMNs`/MMM
[INFO]    MM/ hMMMMMMMMNs.+MMM/ dMMMMMMMMMMMMMMMMMMh +MMM+.sNMMMMMMMMh +MM
[INFO]    MN /MMMMMMNo/./mMMMMN :MMMMMMMMMMMMMMMMMM: NMMMMm/./oNMMMMMM: NM
[INFO]    Mm +MMMMMN+ `/MMMMMMM`-MMMMMMMMMMMMMMMMMM-`MMMMMMM:` oNMMMMM+ mM
[INFO]    MM..NMMNs./mNMMMMMMMy sMMMMMMMMMMMMMMMMMMo hMMMMMMMNm/.sNMMN`-MM
[INFO]    MMd`:mMNomMMMMMMMMMy`:MMMMMMMNmmmmNMMMMMMN:`hMMMMMMMMMdoNMm-`dMM
[INFO]    MMMm:.omMMMMMMMMNh/  sdmmho/.`..`-``-/sddh+  /hNMMMMMMMMdo.:mMMM
[INFO]    MMMMMd+--/osss+:-:/`  ```:- .ym+ hmo``:-`   `+:-:ossso/-:+dMMMMM
[INFO]    MMMMMMMNmhysosydmNMo   /ds`/NMM+ hMMd..dh.  sMNmdysosyhmNMMMMMMM
[INFO]    MMMMMMMMMMMMMMMMMMMs .:-:``hmmN+ yNmds -:.:`-NMMMMMMMMMMMMMMMMMM
[INFO]    MMMMMMMMMMMMMMMMMMN.-mNm- //:::. -:://: +mMd`-NMMMMMMMMMMMMMMMMM
[INFO]    MMMMMMMMMMMMMMMMMM+ dMMN -MMNNN+ yNNNMN :MMMs sMMMMMMMMMMMMMMMMM
[INFO]    MMMMMMMMMMMMMMMMMM`.mmmy /mmmmm/ smmmmm``mmmh :MMMMMMMMMMMMMMMMM
[INFO]    MMMMMMMMMMMMMMMMMM``:::- ./////. -:::::` :::: -MMMMMMMMMMMMMMMMM
[INFO]    MMMMMMMMMMMMMMMMMM:`mNNd /NNNNN+ hNNNNN .NNNy +MMMMMMMMMMMMMMMMM
[INFO]    MMMMMMMMMMMMMMMMMMd`/MMM.`ys+//. -/+oso +MMN.`mMMMMMMMMMMMMMMMMM
[INFO]    MMMMMMMMMMMMMMMMMMMy /o:- `oyhd/ shys+ `-:s-`hMMMMMMMMMMMMMMMMMM
[INFO]    MMMMMMMMNmdhhhdmNMMM`  +d+ sMMM+ hMMN:`hh-  sMMNmdhhhdmNMMMMMMMM
[INFO]    MMMMMms:::/++//::+ho    .+- /dM+ hNh- +/`   -h+:://++/::/smMMMMM
[INFO]    MMMN+./hmMMMMMMNds-  ./oso:.``:. :-``.:os+-  -sdNMMMMMMmy:.oNMMM
[INFO]    MMm-.hMNhNMMMMMMMMNo`/MMMMMNdhyyyyhhdNMMMM+`oNMMMMMMMMNhNMh.-mMM
[INFO]    MM:`mMMN/-sNNMMMMMMMo yMMMMMMMMMMMMMMMMMMy sMMMMMMMNNs-/NMMm`:MM
[INFO]    Mm /MMMMMd/.-oMMMMMMN :MMMMMMMMMMMMMMMMMM-`MMMMMMMo-./dMMMMM/ NM
[INFO]    Mm /MMMMMMm:-`sNMMMMN :MMMMMMMMMMMMMMMMMM-`MMMMMNs`-/NMMMMMM/ NM
[INFO]    MM:`mMMMMMMMMd/-sMMMo yMMMMMMMMMMMMMMMMMMy sMMMs-/dMMMMMMMMd`:MM
[INFO]    MMm-.hMMMMMMMMMdhMNo`+MMMMMMMMMMMMMMMMMMMM+`oNMhdMMMMMMMMMh.-mMM
[INFO]    MMMNo./hmNMMMMMNms--yMMMMMMMMMMMMMMMMMMMMMMy--smNMMMMMNmy/.oNMMM
[INFO]    MMMMMms:-:/+++/:-+hMMMMMMMMMMMMMMMMMMMMMMMMMNh+-:/+++/:-:smMMMMM
[INFO]    MMMMMMMMNdhhyhdmMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMmdhyhhmNMMMMMMMM
[INFO]    MMMMMMMMMMMMMMMNNNNNMMMMMMNNNNNNMMMMMMMMNNMMMMMMMNNMMMMMMMMMMMMM
[INFO]    MMMMMMMMMMMMMh/-...-+dMMMm......:+hMMMMs../MMMMMo..sMMMMMMMMMMMM
[INFO]    MMMMMMMMMMMM/  /yhy-  sMMm  -hhy/  :NMM+   oMMMy   /MMMMMMMMMMMM
[INFO]    MMMMMMMMMMMy  /MMMMN`  NMm  /MMMMo  +MM: .` yMd``` :MMMMMMMMMMMM
[INFO]    MMMMMMMMMMM+  sMMMMM:  hMm  /MMMMd  -MM- /s `h.`d- -MMMMMMMMMMMM
[INFO]    MMMMMMMMMMMs  +MMMMM.  mMm  /MMMMy  /MM. +M/   yM: `MMMMMMMMMMMM
[INFO]    MMMMMMMMMMMN-  smNm/  +MMm  :NNdo` .mMM` oMM+/yMM/  MMMMMMMMMMMM
[INFO]    MMMMMMMMMMMMNo-    `:yMMMm      `:sNMMM` sMMMMMMM+  NMMMMMMMMMMM
[INFO]    MMMMMMMMMMMMMMMNmmNMMMMMMMNmmmmNMMMMMMMNNMMMMMMMMMNNMMMMMMMMMMMM
[INFO]    ODM app finished - Mon Jan 10 22:04:25  2022
100 - done.
Postprocessing: /var/www/data/3320f153-0262-4c9a-90c5-ba8222f91d50
Found point cloud: odm_georeferencing/odm_georeferenced_model.laz

Entwine point cloud is already built.
Entwine point cloud is already built.
Postprocessing: done (•̀ᴗ•́)و!

Compressing all.zip

Done!

What else are you running on the machine while processing? Chrome? Other heavy stuff? Try to minimize all other usage to keep your free memory/swap pool more consistent across runs.

I am running LibreWolf (a Firefox fork), but I doubt it has to do with the problem because I tested shutting down all apps and had the out of memory problem, and now while I am using Librewolf, with Signal, Flameshot, Nextcloud and VPN open and it works OK.

wow. this is so strange.

I now created a new project for a new set of images. Same happened (not enough memory.)
And then I created that tasks in my immaculate “First Project”, which has always finished (in different qualities) OK.
And it goes OK.
What is happening here? (image attached)
image

EDIT: It halted with not enough memory after one hour of processing.

... output truncated at undefined lines ...
2022-01-11 00:13:09,767 INFO: DJI_0310.JPG resection inliers: 97 / 108
2022-01-11 00:13:09,769 INFO: Adding DJI_0310.JPG to the reconstruction
2022-01-11 00:13:09,914 INFO: -------------------------------------------------------
2022-01-11 00:13:09,920 INFO: DJI_0309.JPG resection inliers: 144 / 157
2022-01-11 00:13:09,922 INFO: Adding DJI_0309.JPG to the reconstruction
2022-01-11 00:13:10,050 INFO: -------------------------------------------------------
2022-01-11 00:13:10,060 INFO: DJI_0301.JPG resection inliers: 38 / 119
2022-01-11 00:13:10,061 INFO: Adding DJI_0301.JPG to the reconstruction
2022-01-11 00:13:10,185 INFO: -------------------------------------------------------
2022-01-11 00:13:10,191 INFO: DJI_0300.JPG resection inliers: 62 / 105
2022-01-11 00:13:10,192 INFO: Adding DJI_0300.JPG to the reconstruction
2022-01-11 00:13:10,325 INFO: -------------------------------------------------------
2022-01-11 00:13:10,332 INFO: DJI_0302.JPG resection inliers: 113 / 213
2022-01-11 00:13:10,334 INFO: Adding DJI_0302.JPG to the reconstruction
2022-01-11 00:13:10,465 INFO: -------------------------------------------------------
2022-01-11 00:13:10,472 INFO: DJI_0303.JPG resection inliers: 62 / 139
2022-01-11 00:13:10,473 INFO: Adding DJI_0303.JPG to the reconstruction
2022-01-11 00:13:10,594 INFO: -------------------------------------------------------
2022-01-11 00:13:10,600 INFO: DJI_0308.JPG resection inliers: 82 / 103
2022-01-11 00:13:10,602 INFO: Adding DJI_0308.JPG to the reconstruction
2022-01-11 00:13:10,726 INFO: -------------------------------------------------------
2022-01-11 00:13:10,734 INFO: DJI_0307.JPG resection inliers: 109 / 120
2022-01-11 00:13:10,736 INFO: Adding DJI_0307.JPG to the reconstruction
2022-01-11 00:13:10,864 INFO: -------------------------------------------------------
2022-01-11 00:13:10,889 INFO: DJI_0306.JPG resection inliers: 96 / 99
2022-01-11 00:13:10,891 INFO: Adding DJI_0306.JPG to the reconstruction
2022-01-11 00:13:11,177 INFO: -------------------------------------------------------
2022-01-11 00:13:11,204 INFO: DJI_0305.JPG resection inliers: 282 / 284
2022-01-11 00:13:11,207 INFO: Adding DJI_0305.JPG to the reconstruction
2022-01-11 00:13:11,680 INFO: -------------------------------------------------------
2022-01-11 00:13:11,688 INFO: DJI_0304.JPG resection inliers: 231 / 259
2022-01-11 00:13:11,691 INFO: Adding DJI_0304.JPG to the reconstruction
2022-01-11 00:13:11,886 INFO: -------------------------------------------------------
2022-01-11 00:13:11,892 INFO: DJI_0299.JPG resection inliers: 9 / 21
2022-01-11 00:13:11,893 INFO: Some images can not be added
2022-01-11 00:13:11,893 INFO: -------------------------------------------------------
2022-01-11 00:13:11,896 INFO: Shots and/or GCPs are well-conditioned. Using naive 3D-3D alignment.
2022-01-11 00:13:32,494 DEBUG: Ceres Solver Report: Iterations: 29, Initial cost: 8.816128e+04, Final cost: 5.503584e+04, Termination: CONVERGENCE
2022-01-11 00:13:34,077 INFO: Removed outliers: 6825
2022-01-11 00:13:35,158 INFO: Reconstruction 0: 120 images, 150973 points
2022-01-11 00:13:35,158 INFO: 1 partial reconstructions in total.
[INFO]    Export reconstruction stats
[INFO]    running "/code/SuperBuild/install/bin/opensfm/bin/opensfm" compute_statistics --diagram_max_points 100000 "/var/www/data/94f16ce4-2eb0-4953-8d65-6cbe31e75691/opensfm"
2022-01-11 00:14:07,323 DEBUG: locator: <matplotlib.colorbar._ColorbarAutoLocator object at 0x7fdf64760490>
2022-01-11 00:14:07,324 DEBUG: Using auto colorbar locator <matplotlib.colorbar._ColorbarAutoLocator object at 0x7fdf64760490> on colorbar
2022-01-11 00:14:07,324 DEBUG: Setting pcolormesh
2022-01-11 00:14:09,296 DEBUG: locator: <matplotlib.colorbar._ColorbarAutoLocator object at 0x7fdf646a98b0>
2022-01-11 00:14:09,296 DEBUG: Using auto colorbar locator <matplotlib.colorbar._ColorbarAutoLocator object at 0x7fdf646a98b0> on colorbar
2022-01-11 00:14:09,296 DEBUG: Setting pcolormesh
[INFO]    running "/code/SuperBuild/install/bin/opensfm/bin/opensfm" export_geocoords --reconstruction --proj "+proj=utm +zone=19 +south +datum=WGS84 +units=m +no_defs +type=crs" --offset-x 287647.0 --offset-y 6348792.0 "/var/www/data/94f16ce4-2eb0-4953-8d65-6cbe31e75691/opensfm"
[INFO]    Updating /var/www/data/94f16ce4-2eb0-4953-8d65-6cbe31e75691/opensfm/config.yaml
[INFO]    undistorted_image_max_size: 3202
[INFO]    Undistorting /var/www/data/94f16ce4-2eb0-4953-8d65-6cbe31e75691/opensfm ...
2022-01-11 00:14:38,462 DEBUG: Undistorting image DJI_0343.JPG
2022-01-11 00:14:38,462 DEBUG: Undistorting image DJI_0238.JPG
2022-01-11 00:14:38,462 DEBUG: Undistorting image DJI_0237.JPG
2022-01-11 00:14:38,463 DEBUG: Undistorting image DJI_0324.JPG
2022-01-11 00:14:50,752 DEBUG: Undistorting image DJI_0248.JPG
2022-01-11 00:14:51,280 DEBUG: Undistorting image DJI_0269.JPG
2022-01-11 00:14:51,680 DEBUG: Undistorting image DJI_0346.JPG
2022-01-11 00:14:52,008 DEBUG: Undistorting image DJI_0280.JPG
2022-01-11 00:15:03,753 DEBUG: Undistorting image DJI_0353.JPG
2022-01-11 00:15:04,265 DEBUG: Undistorting image DJI_0268.JPG
2022-01-11 00:15:04,792 DEBUG: Undistorting image DJI_0264.JPG
2022-01-11 00:15:05,126 DEBUG: Undistorting image DJI_0249.JPG
2022-01-11 00:15:16,736 DEBUG: Undistorting image DJI_0258.JPG
2022-01-11 00:15:17,284 DEBUG: Undistorting image DJI_0243.JPG
2022-01-11 00:15:18,644 DEBUG: Undistorting image DJI_0298.JPG
2022-01-11 00:15:18,651 DEBUG: Undistorting image DJI_0338.JPG
2022-01-11 00:15:29,732 DEBUG: Undistorting image DJI_0271.JPG
2022-01-11 00:15:30,164 DEBUG: Undistorting image DJI_0274.JPG
2022-01-11 00:15:31,224 DEBUG: Undistorting image DJI_0293.JPG
2022-01-11 00:15:31,618 DEBUG: Undistorting image DJI_0337.JPG
2022-01-11 00:15:42,009 DEBUG: Undistorting image DJI_0294.JPG
2022-01-11 00:15:43,027 DEBUG: Undistorting image DJI_0277.JPG
2022-01-11 00:15:43,757 DEBUG: Undistorting image DJI_0290.JPG
2022-01-11 00:15:44,557 DEBUG: Undistorting image DJI_0314.JPG
2022-01-11 00:15:54,863 DEBUG: Undistorting image DJI_0275.JPG
2022-01-11 00:15:55,351 DEBUG: Undistorting image DJI_0254.JPG
2022-01-11 00:15:56,127 DEBUG: Undistorting image DJI_0278.JPG
2022-01-11 00:15:56,489 DEBUG: Undistorting image DJI_0325.JPG
2022-01-11 00:16:07,473 DEBUG: Undistorting image DJI_0305.JPG
2022-01-11 00:16:08,386 DEBUG: Undistorting image DJI_0267.JPG
2022-01-11 00:16:08,389 DEBUG: Undistorting image DJI_0316.JPG
2022-01-11 00:16:08,813 DEBUG: Undistorting image DJI_0319.JPG
2022-01-11 00:16:19,521 DEBUG: Undistorting image DJI_0302.JPG
2022-01-11 00:16:20,230 DEBUG: Undistorting image DJI_0309.JPG
2022-01-11 00:16:20,705 DEBUG: Undistorting image DJI_0355.JPG
2022-01-11 00:16:21,098 DEBUG: Undistorting image DJI_0320.JPG
2022-01-11 00:16:31,499 DEBUG: Undistorting image DJI_0270.JPG
2022-01-11 00:16:31,967 DEBUG: Undistorting image DJI_0306.JPG
2022-01-11 00:16:32,826 DEBUG: Undistorting image DJI_0315.JPG
2022-01-11 00:16:33,381 DEBUG: Undistorting image DJI_0304.JPG
2022-01-11 00:16:43,172 DEBUG: Undistorting image DJI_0307.JPG
2022-01-11 00:16:43,683 DEBUG: Undistorting image DJI_0335.JPG
2022-01-11 00:16:45,134 DEBUG: Undistorting image DJI_0241.JPG
2022-01-11 00:16:45,563 DEBUG: Undistorting image DJI_0247.JPG
2022-01-11 00:16:54,400 DEBUG: Undistorting image DJI_0354.JPG
2022-01-11 00:16:55,790 DEBUG: Undistorting image DJI_0296.JPG
2022-01-11 00:16:56,912 DEBUG: Undistorting image DJI_0273.JPG
2022-01-11 00:16:57,457 DEBUG: Undistorting image DJI_0291.JPG
2022-01-11 00:17:06,405 DEBUG: Undistorting image DJI_0318.JPG
2022-01-11 00:17:07,219 DEBUG: Undistorting image DJI_0351.JPG
2022-01-11 00:17:08,588 DEBUG: Undistorting image DJI_0336.JPG
2022-01-11 00:17:09,544 DEBUG: Undistorting image DJI_0282.JPG
2022-01-11 00:17:18,260 DEBUG: Undistorting image DJI_0281.JPG
2022-01-11 00:17:19,239 DEBUG: Undistorting image DJI_0303.JPG
2022-01-11 00:17:20,737 DEBUG: Undistorting image DJI_0334.JPG
2022-01-11 00:17:21,288 DEBUG: Undistorting image DJI_0331.JPG
2022-01-11 00:17:29,748 DEBUG: Undistorting image DJI_0279.JPG
2022-01-11 00:17:30,243 DEBUG: Undistorting image DJI_0321.JPG
2022-01-11 00:17:32,224 DEBUG: Undistorting image DJI_0297.JPG
2022-01-11 00:17:32,619 DEBUG: Undistorting image DJI_0256.JPG
2022-01-11 00:17:40,909 DEBUG: Undistorting image DJI_0235.JPG
2022-01-11 00:17:41,475 DEBUG: Undistorting image DJI_0308.JPG
2022-01-11 00:17:43,757 DEBUG: Undistorting image DJI_0300.JPG
2022-01-11 00:17:44,236 DEBUG: Undistorting image DJI_0289.JPG
2022-01-11 00:17:53,022 DEBUG: Undistorting image DJI_0239.JPG
2022-01-11 00:17:53,973 DEBUG: Undistorting image DJI_0327.JPG
2022-01-11 00:17:56,108 DEBUG: Undistorting image DJI_0287.JPG
2022-01-11 00:17:56,651 DEBUG: Undistorting image DJI_0350.JPG
2022-01-11 00:18:05,200 DEBUG: Undistorting image DJI_0285.JPG
2022-01-11 00:18:05,808 DEBUG: Undistorting image DJI_0326.JPG
2022-01-11 00:18:07,790 DEBUG: Undistorting image DJI_0292.JPG
2022-01-11 00:18:08,230 DEBUG: Undistorting image DJI_0301.JPG
2022-01-11 00:18:16,841 DEBUG: Undistorting image DJI_0340.JPG
2022-01-11 00:18:18,770 DEBUG: Undistorting image DJI_0272.JPG
2022-01-11 00:18:20,258 DEBUG: Undistorting image DJI_0259.JPG
2022-01-11 00:18:20,706 DEBUG: Undistorting image DJI_0242.JPG
2022-01-11 00:18:29,414 DEBUG: Undistorting image DJI_0257.JPG
2022-01-11 00:18:30,800 DEBUG: Undistorting image DJI_0286.JPG
2022-01-11 00:18:32,544 DEBUG: Undistorting image DJI_0332.JPG
2022-01-11 00:18:33,153 DEBUG: Undistorting image DJI_0328.JPG
2022-01-11 00:18:40,536 DEBUG: Undistorting image DJI_0288.JPG
2022-01-11 00:18:40,940 DEBUG: Undistorting image DJI_0344.JPG
2022-01-11 00:18:45,592 DEBUG: Undistorting image DJI_0311.JPG
2022-01-11 00:18:45,989 DEBUG: Undistorting image DJI_0244.JPG
2022-01-11 00:18:52,105 DEBUG: Undistorting image DJI_0339.JPG
2022-01-11 00:18:52,436 DEBUG: Undistorting image DJI_0342.JPG
2022-01-11 00:18:56,979 DEBUG: Undistorting image DJI_0266.JPG
2022-01-11 00:18:57,315 DEBUG: Undistorting image DJI_0330.JPG
2022-01-11 00:19:03,648 DEBUG: Undistorting image DJI_0341.JPG
2022-01-11 00:19:04,005 DEBUG: Undistorting image DJI_0265.JPG
2022-01-11 00:19:08,059 DEBUG: Undistorting image DJI_0329.JPG
2022-01-11 00:19:08,458 DEBUG: Undistorting image DJI_0284.JPG
2022-01-11 00:19:15,095 DEBUG: Undistorting image DJI_0263.JPG
2022-01-11 00:19:15,524 DEBUG: Undistorting image DJI_0313.JPG
2022-01-11 00:19:19,766 DEBUG: Undistorting image DJI_0255.JPG
2022-01-11 00:19:20,271 DEBUG: Undistorting image DJI_0345.JPG
2022-01-11 00:19:26,391 DEBUG: Undistorting image DJI_0252.JPG
2022-01-11 00:19:27,331 DEBUG: Undistorting image DJI_0317.JPG
2022-01-11 00:19:31,348 DEBUG: Undistorting image DJI_0253.JPG
2022-01-11 00:19:31,800 DEBUG: Undistorting image DJI_0283.JPG
2022-01-11 00:19:38,644 DEBUG: Undistorting image DJI_0262.JPG
2022-01-11 00:19:39,143 DEBUG: Undistorting image DJI_0261.JPG
2022-01-11 00:19:42,893 DEBUG: Undistorting image DJI_0250.JPG
2022-01-11 00:19:43,317 DEBUG: Undistorting image DJI_0246.JPG
2022-01-11 00:19:50,091 DEBUG: Undistorting image DJI_0251.JPG
2022-01-11 00:19:50,644 DEBUG: Undistorting image DJI_0276.JPG
2022-01-11 00:19:54,588 DEBUG: Undistorting image DJI_0236.JPG
2022-01-11 00:19:54,955 DEBUG: Undistorting image DJI_0312.JPG
2022-01-11 00:20:01,759 DEBUG: Undistorting image DJI_0260.JPG
2022-01-11 00:20:02,422 DEBUG: Undistorting image DJI_0240.JPG
2022-01-11 00:20:06,483 DEBUG: Undistorting image DJI_0348.JPG
2022-01-11 00:20:06,883 DEBUG: Undistorting image DJI_0347.JPG
2022-01-11 00:20:13,339 DEBUG: Undistorting image DJI_0295.JPG
2022-01-11 00:20:13,705 DEBUG: Undistorting image DJI_0322.JPG
2022-01-11 00:20:17,741 DEBUG: Undistorting image DJI_0349.JPG
2022-01-11 00:20:18,126 DEBUG: Undistorting image DJI_0333.JPG
2022-01-11 00:20:24,809 DEBUG: Undistorting image DJI_0310.JPG
2022-01-11 00:20:25,203 DEBUG: Undistorting image DJI_0323.JPG
2022-01-11 00:20:28,916 DEBUG: Undistorting image DJI_0352.JPG
2022-01-11 00:20:29,335 DEBUG: Undistorting image DJI_0245.JPG
[INFO]    running "/code/SuperBuild/install/bin/opensfm/bin/opensfm" export_visualsfm --points "/var/www/data/94f16ce4-2eb0-4953-8d65-6cbe31e75691/opensfm"
[INFO]    Finished opensfm stage
[INFO]    Running openmvs stage
[INFO]    running "/code/SuperBuild/install/bin/opensfm/bin/opensfm" export_openmvs "/var/www/data/94f16ce4-2eb0-4953-8d65-6cbe31e75691/opensfm"
[INFO]    Running dense reconstruction. This might take a while.
[INFO]    Estimating depthmaps
[INFO]    running "/code/SuperBuild/install/bin/OpenMVS/DensifyPointCloud" "/var/www/data/94f16ce4-2eb0-4953-8d65-6cbe31e75691/opensfm/undistorted/openmvs/scene.mvs"  --resolution-level 2 --min-resolution 1000 --max-resolution 3202 --max-threads 4 --number-views-fuse 2 -w "/var/www/data/94f16ce4-2eb0-4953-8d65-6cbe31e75691/opensfm/undistorted/openmvs/depthmaps" -v 0 --geometric-iters 0 --cuda-device -1
00:20:59 [App     ] Build date: Jan  9 2022, 00:27:19
00:20:59 [App     ] CPU: AMD Ryzen 7 4800H with Radeon Graphics          (16 cores)
00:20:59 [App     ] RAM: 15.10GB Physical Memory 7.81GB Virtual Memory
00:20:59 [App     ] OS: Linux 5.15.13-arch1-1 (x86_64)
00:20:59 [App     ] SSE & AVX compatible CPU & OS detected
00:20:59 [App     ] Command line: /var/www/data/94f16ce4-2eb0-4953-8d65-6cbe31e75691/opensfm/undistorted/openmvs/scene.mvs --resolution-level 2 --min-resolution 1000 --max-resolution 3202 --max-threads 4 --number-views-fuse 2 -w /var/www/data/94f16ce4-2eb0-4953-8d65-6cbe31e75691/opensfm/undistorted/openmvs/depthmaps -v 0 --geometric-iters 0 --cuda-device -1
00:21:04 [App     ] Preparing images for dense reconstruction completed: 120 images (4s821ms)
00:21:04 [App     ] Selecting images for dense reconstruction completed: 120 images (217ms)

Estimated depth-maps 1 (0.83%, 14s, ETA 0ms)...
Estimated depth-maps 2 (1.67%, 31s, ETA 31m)...
Estimated depth-maps 3 (2.50%, 47s, ETA 30m)...
Estimated depth-maps 4 (3.33%, 1m2s, ETA 30m)...
Estimated depth-maps 5 (4.17%, 1m16s, ETA 29m)...
Estimated depth-maps 6 (5.00%, 1m32s, ETA 29m)...
Estimated depth-maps 7 (5.83%, 1m49s, ETA 29m)...
Estimated depth-maps 8 (6.67%, 2m2s, ETA 28m)...
Estimated depth-maps 9 (7.50%, 2m18s, ETA 28m)...
Estimated depth-maps 10 (8.33%, 2m33s, ETA 28m)...
Estimated depth-maps 11 (9.17%, 2m48s, ETA 27m)...
Estimated depth-maps 12 (10.00%, 3m2s, ETA 27m)...
Estimated depth-maps 13 (10.83%, 3m14s, ETA 26m)...
Estimated depth-maps 14 (11.67%, 3m29s, ETA 26m)...
Estimated depth-maps 15 (12.50%, 3m41s, ETA 25m)...
Estimated depth-maps 16 (13.33%, 3m57s, ETA 25m)...
Estimated depth-maps 17 (14.17%, 4m12s, ETA 25m)...
Estimated depth-maps 18 (15.00%, 4m22s, ETA 24m)...
Estimated depth-maps 19 (15.83%, 4m36s, ETA 24m)...
Estimated depth-maps 20 (16.67%, 4m52s, ETA 24m)...
Estimated depth-maps 21 (17.50%, 5m4s, ETA 23m)...
Estimated depth-maps 22 (18.33%, 5m20s, ETA 23m)...
Estimated depth-maps 23 (19.17%, 5m36s, ETA 23m)...
Estimated depth-maps 24 (20.00%, 5m52s, ETA 23m)...
Estimated depth-maps 25 (20.83%, 6m8s, ETA 23m)...
Estimated depth-maps 26 (21.67%, 6m20s, ETA 22m)...
Estimated depth-maps 27 (22.50%, 6m37s, ETA 22m)...
Estimated depth-maps 28 (23.33%, 6m47s, ETA 22m)...
Estimated depth-maps 29 (24.17%, 7m3s, ETA 22m)...
Estimated depth-maps 30 (25.00%, 7m19s, ETA 21m)...
Estimated depth-maps 31 (25.83%, 7m36s, ETA 21m)...
Estimated depth-maps 32 (26.67%, 7m53s, ETA 21m)...
Estimated depth-maps 33 (27.50%, 8m9s, ETA 21m)...
Estimated depth-maps 34 (28.33%, 8m26s, ETA 21m)...
Estimated depth-maps 35 (29.17%, 8m43s, ETA 21m)...
Estimated depth-maps 36 (30.00%, 8m59s, ETA 20m)...
Estimated depth-maps 37 (30.83%, 9m16s, ETA 20m)...
Estimated depth-maps 38 (31.67%, 9m35s, ETA 20m)...
Estimated depth-maps 39 (32.50%, 9m51s, ETA 20m)...
Estimated depth-maps 40 (33.33%, 10m5s, ETA 20m)...
Estimated depth-maps 41 (34.17%, 10m22s, ETA 19m)...
Estimated depth-maps 42 (35.00%, 10m37s, ETA 19m)...
Estimated depth-maps 43 (35.83%, 10m53s, ETA 19m)...
Estimated depth-maps 44 (36.67%, 11m10s, ETA 19m)...
Estimated depth-maps 45 (37.50%, 11m23s, ETA 18m)...
Estimated depth-maps 46 (38.33%, 11m40s, ETA 18m)...
Estimated depth-maps 47 (39.17%, 11m53s, ETA 18m)...
Estimated depth-maps 48 (40.00%, 12m9s, ETA 18m)...
Estimated depth-maps 49 (40.83%, 12m25s, ETA 18m)...
Estimated depth-maps 50 (41.67%, 12m41s, ETA 17m)...
Estimated depth-maps 51 (42.50%, 12m57s, ETA 17m)...
Estimated depth-maps 52 (43.33%, 13m11s, ETA 17m)...
Estimated depth-maps 53 (44.17%, 13m25s, ETA 16m)...
Estimated depth-maps 54 (45.00%, 13m43s, ETA 16m)...
Estimated depth-maps 55 (45.83%, 14m1s, ETA 16m)...
Estimated depth-maps 56 (46.67%, 14m17s, ETA 16m)...
Estimated depth-maps 57 (47.50%, 14m30s, ETA 16m)...
Estimated depth-maps 58 (48.33%, 14m46s, ETA 15m)...
Estimated depth-maps 59 (49.17%, 15m3s, ETA 15m)...
Estimated depth-maps 60 (50.00%, 15m19s, ETA 15m)...
Estimated depth-maps 61 (50.83%, 15m36s, ETA 15m)...
Estimated depth-maps 62 (51.67%, 15m53s, ETA 14m)...
Estimated depth-maps 63 (52.50%, 16m8s, ETA 14m)...
Estimated depth-maps 64 (53.33%, 16m23s, ETA 14m)...
Estimated depth-maps 65 (54.17%, 16m34s, ETA 14m)...
Estimated depth-maps 66 (55.00%, 16m49s, ETA 13m)...
Estimated depth-maps 67 (55.83%, 17m1s, ETA 13m)...
Estimated depth-maps 68 (56.67%, 17m15s, ETA 13m)...
Estimated depth-maps 69 (57.50%, 17m31s, ETA 12m)...
Estimated depth-maps 70 (58.33%, 17m46s, ETA 12m)...
Estimated depth-maps 71 (59.17%, 18m1s, ETA 12m)...
Estimated depth-maps 72 (60.00%, 18m19s, ETA 12m)...
Estimated depth-maps 73 (60.83%, 18m35s, ETA 11m)...
Estimated depth-maps 74 (61.67%, 18m50s, ETA 11m)...
Estimated depth-maps 75 (62.50%, 19m6s, ETA 11m)...
Estimated depth-maps 76 (63.33%, 19m17s, ETA 11m)...
Estimated depth-maps 77 (64.17%, 19m34s, ETA 10m)...
Estimated depth-maps 78 (65.00%, 19m48s, ETA 10m)...
Estimated depth-maps 79 (65.83%, 20m4s, ETA 10m)...
Estimated depth-maps 80 (66.67%, 20m19s, ETA 10m)...
Estimated depth-maps 81 (67.50%, 20m36s, ETA 9m)...
Estimated depth-maps 82 (68.33%, 20m47s, ETA 9m)...
Estimated depth-maps 83 (69.17%, 21m, ETA 9m)...
Estimated depth-maps 84 (70.00%, 21m15s, ETA 9m)...
Estimated depth-maps 85 (70.83%, 21m31s, ETA 8m)...
Estimated depth-maps 86 (71.67%, 21m45s, ETA 8m)...
Estimated depth-maps 87 (72.50%, 22m, ETA 8m)...
Estimated depth-maps 88 (73.33%, 22m17s, ETA 8m)...
Estimated depth-maps 89 (74.17%, 22m33s, ETA 7m)...
Estimated depth-maps 90 (75.00%, 22m49s, ETA 7m)...
Estimated depth-maps 91 (75.83%, 23m5s, ETA 7m)...
Estimated depth-maps 92 (76.67%, 23m19s, ETA 7m)...
Estimated depth-maps 93 (77.50%, 23m36s, ETA 6m)...
Estimated depth-maps 94 (78.33%, 23m50s, ETA 6m)...
Estimated depth-maps 95 (79.17%, 24m4s, ETA 6m)...
Estimated depth-maps 96 (80.00%, 24m20s, ETA 6m)...
Estimated depth-maps 97 (80.83%, 24m36s, ETA 5m)...
Estimated depth-maps 98 (81.67%, 24m54s, ETA 5m)...
Estimated depth-maps 99 (82.50%, 25m11s, ETA 5m)...
Estimated depth-maps 100 (83.33%, 25m26s, ETA 5m)...
Estimated depth-maps 101 (84.17%, 25m42s, ETA 4m)...
Estimated depth-maps 102 (85.00%, 25m52s, ETA 4m)...
Estimated depth-maps 103 (85.83%, 26m9s, ETA 4m)...
Estimated depth-maps 104 (86.67%, 26m25s, ETA 4m)...
Estimated depth-maps 105 (87.50%, 26m38s, ETA 3m)...
Estimated depth-maps 106 (88.33%, 26m56s, ETA 3m)...
Estimated depth-maps 107 (89.17%, 27m13s, ETA 3m)...
Estimated depth-maps 108 (90.00%, 27m29s, ETA 3m)...
Estimated depth-maps 109 (90.83%, 27m44s, ETA 2m)...
Estimated depth-maps 110 (91.67%, 27m59s, ETA 2m)...
Estimated depth-maps 111 (92.50%, 28m15s, ETA 2m)...
Estimated depth-maps 112 (93.33%, 28m31s, ETA 2m)...
Estimated depth-maps 113 (94.17%, 28m47s, ETA 1m)...
Estimated depth-maps 114 (95.00%, 29m4s, ETA 1m)...
Estimated depth-maps 115 (95.83%, 29m19s, ETA 1m)...
Estimated depth-maps 116 (96.67%, 29m36s, ETA 1m)...
Estimated depth-maps 117 (97.50%, 29m51s, ETA 45s)...
Estimated depth-maps 118 (98.33%, 29m57s, ETA 30s)...
Estimated depth-maps 119 (99.17%, 30m10s, ETA 15s)...
Estimated depth-maps 120 (100.00%, 30m27s, ETA 0ms)...
Estimated depth-maps 120 (100%, 30m27s325ms)

Filtered depth-maps 1 (0.83%, 33s, ETA 0ms)...
Filtered depth-maps 2 (1.67%, 33s, ETA 33m)...
Filtered depth-maps 3 (2.50%, 34s, ETA 22m)...
Filtered depth-maps 4 (3.33%, 34s, ETA 16m)...
Filtered depth-maps 5 (4.17%, 35s, ETA 13m)...
Filtered depth-maps 6 (5.00%, 35s, ETA 11m)...
Filtered depth-maps 7 (5.83%, 36s, ETA 9m)...
Filtered depth-maps 8 (6.67%, 37s, ETA 8m)...
Filtered depth-maps 9 (7.50%, 37s, ETA 7m)...
Filtered depth-maps 10 (8.33%, 38s, ETA 7m)...
Filtered depth-maps 11 (9.17%, 38s, ETA 6m)...
Filtered depth-maps 12 (10.00%, 39s, ETA 5m)...
Filtered depth-maps 13 (10.83%, 40s, ETA 5m)...
Filtered depth-maps 14 (11.67%, 40s, ETA 5m)...
Filtered depth-maps 15 (12.50%, 41s, ETA 4m)...
Filtered depth-maps 16 (13.33%, 41s, ETA 4m)...
Filtered depth-maps 17 (14.17%, 42s, ETA 4m)...
Filtered depth-maps 18 (15.00%, 43s, ETA 4m)...
Filtered depth-maps 19 (15.83%, 43s, ETA 3m)...
Filtered depth-maps 20 (16.67%, 44s, ETA 3m)...
Filtered depth-maps 21 (17.50%, 45s, ETA 3m)...
Filtered depth-maps 22 (18.33%, 47s, ETA 3m)...
Filtered depth-maps 23 (19.17%, 48s, ETA 3m)...
Filtered depth-maps 24 (20.00%, 49s, ETA 3m)...
Filtered depth-maps 25 (20.83%, 50s, ETA 3m)...
Filtered depth-maps 26 (21.67%, 51s, ETA 3m)...
Filtered depth-maps 27 (22.50%, 51s, ETA 2m)...
Filtered depth-maps 28 (23.33%, 52s, ETA 2m)...
Filtered depth-maps 29 (24.17%, 53s, ETA 2m)...
Filtered depth-maps 30 (25.00%, 53s, ETA 2m)...
Filtered depth-maps 31 (25.83%, 54s, ETA 2m)...
Filtered depth-maps 32 (26.67%, 54s, ETA 2m)...
Filtered depth-maps 33 (27.50%, 55s, ETA 2m)...
Filtered depth-maps 34 (28.33%, 56s, ETA 2m)...
Filtered depth-maps 35 (29.17%, 56s, ETA 2m)...
Filtered depth-maps 36 (30.00%, 57s, ETA 2m)...
Filtered depth-maps 37 (30.83%, 57s, ETA 2m)...
Filtered depth-maps 38 (31.67%, 58s, ETA 2m)...
Filtered depth-maps 39 (32.50%, 59s, ETA 2m)...
Filtered depth-maps 40 (33.33%, 59s, ETA 1m)...
Filtered depth-maps 41 (34.17%, 1m, ETA 1m)...
Filtered depth-maps 42 (35.00%, 1m1s, ETA 1m)...
Filtered depth-maps 43 (35.83%, 1m2s, ETA 1m)...
Filtered depth-maps 44 (36.67%, 1m2s, ETA 1m)...
Filtered depth-maps 45 (37.50%, 1m3s, ETA 1m)...
Filtered depth-maps 46 (38.33%, 1m3s, ETA 1m)...
Filtered depth-maps 47 (39.17%, 1m4s, ETA 1m)...
Filtered depth-maps 48 (40.00%, 1m4s, ETA 1m)...
Filtered depth-maps 49 (40.83%, 1m5s, ETA 1m)...
Filtered depth-maps 50 (41.67%, 1m5s, ETA 1m)...
Filtered depth-maps 51 (42.50%, 1m6s, ETA 1m)...
Filtered depth-maps 52 (43.33%, 1m7s, ETA 1m)...
Filtered depth-maps 53 (44.17%, 1m7s, ETA 1m)...
Filtered depth-maps 54 (45.00%, 1m8s, ETA 1m)...
Filtered depth-maps 55 (45.83%, 1m8s, ETA 1m)...
Filtered depth-maps 56 (46.67%, 1m8s, ETA 1m)...
Filtered depth-maps 57 (47.50%, 1m9s, ETA 1m)...
Filtered depth-maps 58 (48.33%, 1m9s, ETA 1m)...
Filtered depth-maps 59 (49.17%, 1m10s, ETA 1m)...
Filtered depth-maps 60 (50.00%, 1m11s, ETA 1m)...
Filtered depth-maps 61 (50.83%, 1m11s, ETA 1m)...
Filtered depth-maps 62 (51.67%, 1m12s, ETA 1m)...
Filtered depth-maps 63 (52.50%, 1m13s, ETA 1m)...
Filtered depth-maps 64 (53.33%, 1m13s, ETA 1m)...
Filtered depth-maps 65 (54.17%, 1m14s, ETA 1m)...
Filtered depth-maps 66 (55.00%, 1m14s, ETA 1m)...
Filtered depth-maps 67 (55.83%, 1m15s, ETA 59s)...
Filtered depth-maps 68 (56.67%, 1m16s, ETA 58s)...
Filtered depth-maps 69 (57.50%, 1m17s, ETA 57s)...
Filtered depth-maps 70 (58.33%, 1m17s, ETA 55s)...
Filtered depth-maps 71 (59.17%, 1m18s, ETA 54s)...
Filtered depth-maps 72 (60.00%, 1m19s, ETA 52s)...
Filtered depth-maps 73 (60.83%, 1m19s, ETA 51s)...
Filtered depth-maps 74 (61.67%, 1m20s, ETA 49s)...
Filtered depth-maps 75 (62.50%, 1m21s, ETA 48s)...
Filtered depth-maps 76 (63.33%, 1m21s, ETA 47s)...
Filtered depth-maps 77 (64.17%, 1m21s, ETA 45s)...
Filtered depth-maps 78 (65.00%, 1m22s, ETA 44s)...
Filtered depth-maps 79 (65.83%, 1m23s, ETA 43s)...
Filtered depth-maps 80 (66.67%, 1m23s, ETA 41s)...
Filtered depth-maps 81 (67.50%, 1m24s, ETA 40s)...
Filtered depth-maps 82 (68.33%, 1m24s, ETA 39s)...
Filtered depth-maps 83 (69.17%, 1m25s, ETA 38s)...
Filtered depth-maps 84 (70.00%, 1m26s, ETA 37s)...
Filtered depth-maps 85 (70.83%, 1m26s, ETA 35s)...
Filtered depth-maps 86 (71.67%, 1m27s, ETA 34s)...
Filtered depth-maps 87 (72.50%, 1m27s, ETA 33s)...
Filtered depth-maps 88 (73.33%, 1m28s, ETA 32s)...
Filtered depth-maps 89 (74.17%, 1m29s, ETA 31s)...
Filtered depth-maps 90 (75.00%, 1m29s, ETA 29s)...
Filtered depth-maps 91 (75.83%, 1m30s, ETA 28s)...
Filtered depth-maps 92 (76.67%, 1m30s, ETA 27s)...
Filtered depth-maps 93 (77.50%, 1m31s, ETA 26s)...
Filtered depth-maps 94 (78.33%, 1m32s, ETA 25s)...
Filtered depth-maps 95 (79.17%, 1m32s, ETA 24s)...
Filtered depth-maps 96 (80.00%, 1m33s, ETA 23s)...
Filtered depth-maps 97 (80.83%, 1m34s, ETA 22s)...
Filtered depth-maps 98 (81.67%, 1m34s, ETA 21s)...
Filtered depth-maps 99 (82.50%, 1m35s, ETA 20s)...
Filtered depth-maps 100 (83.33%, 1m35s, ETA 19s)...
Filtered depth-maps 101 (84.17%, 1m36s, ETA 18s)...
Filtered depth-maps 102 (85.00%, 1m36s, ETA 17s)...
Filtered depth-maps 103 (85.83%, 1m36s, ETA 15s)...
Filtered depth-maps 104 (86.67%, 1m37s, ETA 14s)...
Filtered depth-maps 105 (87.50%, 1m38s, ETA 14s)...
Filtered depth-maps 106 (88.33%, 1m38s, ETA 13s)...
Filtered depth-maps 107 (89.17%, 1m38s, ETA 11s)...
Filtered depth-maps 108 (90.00%, 1m39s, ETA 11s)...
Filtered depth-maps 109 (90.83%, 1m39s, ETA 10s)...
Filtered depth-maps 110 (91.67%, 1m40s, ETA 9s)...
Filtered depth-maps 111 (92.50%, 1m40s, ETA 8s)...
Filtered depth-maps 112 (93.33%, 1m40s, ETA 7s)...
Filtered depth-maps 113 (94.17%, 1m41s, ETA 6s)...
Filtered depth-maps 114 (95.00%, 1m41s, ETA 5s)...
Filtered depth-maps 115 (95.83%, 1m42s, ETA 4s)...
Filtered depth-maps 116 (96.67%, 1m42s, ETA 3s)...
Filtered depth-maps 117 (97.50%, 1m43s, ETA 2s)...
Filtered depth-maps 118 (98.33%, 1m43s, ETA 1s)...
Filtered depth-maps 119 (99.17%, 1m44s, ETA 876ms)...
Filtered depth-maps 120 (100.00%, 1m44s, ETA 0ms)...
Filtered depth-maps 120 (100%, 1m45s31ms)

Fused depth-maps 0 (0.00%, 3s, ETA 0ms)...
Fused depth-maps 1 (0.83%, 4s, ETA 0ms)...
Fused depth-maps 2 (1.67%, 5s, ETA 5m)...
Fused depth-maps 3 (2.50%, 6s, ETA 4m)...
Fused depth-maps 4 (3.33%, 8s, ETA 4m)...
Fused depth-maps 5 (4.17%, 9s, ETA 3m)...
Fused depth-maps 6 (5.00%, 10s, ETA 3m)...
Fused depth-maps 7 (5.83%, 11s, ETA 3m)...
Fused depth-maps 8 (6.67%, 11s, ETA 2m)...
Fused depth-maps 9 (7.50%, 12s, ETA 2m)...
Fused depth-maps 10 (8.33%, 12s, ETA 2m)...
Fused depth-maps 11 (9.17%, 12s, ETA 2m)...
Fused depth-maps 13 (10.83%, 13s, ETA 1m)...
Fused depth-maps 14 (11.67%, 13s, ETA 1m)...
Fused depth-maps 15 (12.50%, 13s, ETA 1m)...
Fused depth-maps 16 (13.33%, 14s, ETA 1m)...
Fused depth-maps 17 (14.17%, 14s, ETA 1m)...
Fused depth-maps 18 (15.00%, 16s, ETA 1m)...
Fused depth-maps 19 (15.83%, 16s, ETA 1m)...
Fused depth-maps 20 (16.67%, 16s, ETA 1m)...
Fused depth-maps 21 (17.50%, 17s, ETA 1m)...
Fused depth-maps 22 (18.33%, 18s, ETA 1m)...
Fused depth-maps 23 (19.17%, 18s, ETA 1m)...
Fused depth-maps 24 (20.00%, 18s, ETA 1m)...
Fused depth-maps 25 (20.83%, 19s, ETA 1m)...
Fused depth-maps 26 (21.67%, 19s, ETA 1m)...
Fused depth-maps 27 (22.50%, 20s, ETA 1m)...
Fused depth-maps 28 (23.33%, 21s, ETA 1m)...
Fused depth-maps 29 (24.17%, 21s, ETA 1m)...
Fused depth-maps 30 (25.00%, 21s, ETA 1m)...
Fused depth-maps 31 (25.83%, 24s, ETA 1m)...
Fused depth-maps 32 (26.67%, 25s, ETA 1m)...
Fused depth-maps 33 (27.50%, 26s, ETA 1m)...
Fused depth-maps 34 (28.33%, 30s, ETA 1m)...
Fused depth-maps 35 (29.17%, 32s, ETA 1m)...
Fused depth-maps 36 (30.00%, 33s, ETA 1m)...
Fused depth-maps 37 (30.83%, 33s, ETA 1m)...
Fused depth-maps 38 (31.67%, 33s, ETA 1m)...
Fused depth-maps 39 (32.50%, 34s, ETA 1m)...Killed

===== Dumping Info for Geeks (developers need this to fix bugs) =====
Child returned 137
Traceback (most recent call last):
File "/code/stages/odm_app.py", line 94, in execute
self.first_stage.run()
File "/code/opendm/types.py", line 346, in run
self.next_stage.run(outputs)
File "/code/opendm/types.py", line 346, in run
self.next_stage.run(outputs)
File "/code/opendm/types.py", line 346, in run
self.next_stage.run(outputs)
[Previous line repeated 1 more time]
File "/code/opendm/types.py", line 327, in run
self.process(self.args, outputs)
File "/code/stages/openmvs.py", line 100, in process
raise e
File "/code/stages/openmvs.py", line 91, in process
run_densify()
File "/code/stages/openmvs.py", line 86, in run_densify
system.run('"%s" "%s" %s' % (context.omvs_densify_path,
File "/code/opendm/system.py", line 106, in run
raise SubprocessException("Child returned {}".format(retcode), retcode)
opendm.system.SubprocessException: Child returned 137

===== Done, human-readable information to follow... =====

[ERROR]   Whoops! You ran out of memory! Add more RAM to your computer, if you're using docker configure it to use more memory, for WSL2 make use of .wslconfig (https://docs.microsoft.com/en-us/windows/wsl/wsl-config#configure-global-options-with-wslconfig), resize your images, lower the quality settings or process the images using a cloud provider (e.g. https://webodm.net).

What node is each task using? Same one?

Hi Geoma,

out of curiosity ,
did you actually see the ram in your pc fill up to 16gb?

how big is the partition you have your linux installed ? could it be you are filling out your docker ?

Best!

1 Like

yes, I only have one node available
node-odm-1

1 Like

thanks @firebblade !
The RAM Usage goes high but not higher than 90- 91%
CPU does go higher though.
I didn´t monitor it thoroughly though, so I will next time
My root partition is 70GB
my /home partition is about 900 GB

Could I be filling my docker?
Probably, I have no idea how to check that, I have no experience with docker and I am not sure if docker/ODM are using my root partition or my home folder for cache while working nor for storing the final files built once the process is done
image

1 Like

Docker should be storing the images in /var/lib/docker, which sounds like your root partition, though WebODM looks like it is reading your home partition.

Please check your ODM/WebODM/NodeODM version, its same?

Aside that, we can resize your images when processing in WebODM, the default is 2048 px.

I have 16 GB RAM too. If I dont resize it, it also out of memory, If I resize it to 2048px, most of times, it was success. (using Default preset, processing about 100-150 photos)

Sometimes high quality preset + no resize will works if it just small dataset under (50 photos)

1 Like

i think you would need to learn some basics for docker if you wanna go the linux way .

to check the size of your docker sudo du -sh /var/lib/docker/
check your docker info docker info
check images docker image ls
inspect an image docker image inspect opendronemap/webodm_webapp

i would suggest watching a 101 on docker or searching for some docker cheat sheet like this one The Ultimate Docker Cheat Sheet | dockerlabs