Processing large dataset (part ii)

Hi community,
This is a continuation of a previous discussion…

I am attempting to process 65k images over an area with buildings and a few trees.
Images were captured using a drone flying in 8 directions with 80% overlap and at a height of 35m above the ground.
I gained access to a second VM. So now I have two VMs with 256GB RAM and 256GB SWAP in each
I have set up NodeODM on the 2 machines and ClusterODM on one of them.
My first test with a sample of 250 images is running but seems to be only using one node. I used the defaults…
Info from tasks.json:

status: {Code 20}
webhook: null
skippostprocessing: true
outputs: []

Maybe it is only using one node as it does not require more than that?
Or I need to tell it to use both nodes…
Any advice is appreciated!
I hope to start it with approx 8k images once the first test has completed.
Does ClusterODM only work with a split merge? In that case, I would not get the textured mesh. Is this correct?

1 Like

Correct, same for the Report PDF.

2 Likes

So…
I just ran cluster ODM on 4k images with a split of 2000 and a split-overlap of 25
Going to try again with 2000 images and a split of 800.
THIS failed with the following error… (Any advice is appreciated)

[INFO]    running "/code/SuperBuild/install/bin/opensfm/bin/opensfm" export_visualsfm --points "/var/www/data/8ebe40ef-be26-4fbf-a2cd-5acf352f9f16/opensfm"
[INFO]    Finished opensfm stage
[INFO]    Running openmvs stage
[INFO]    running "/code/SuperBuild/install/bin/opensfm/bin/opensfm" export_openmvs "/var/www/data/8ebe40ef-be26-4fbf-a2cd-5acf352f9f16/opensfm"
[INFO]    Depthmap resolution set to: 507px
[INFO]    Running dense reconstruction. This might take a while.
[INFO]    Estimating depthmaps
[INFO]    nvidia-smi not found in PATH, using CPU
[INFO]    running "/code/SuperBuild/install/bin/OpenMVS/DensifyPointCloud" "/var/www/data/8ebe40ef-be26-4fbf-a2cd-5acf352f9f16/opensfm/undistorted/openmvs/scene.mvs"  --resolution-level 2 --dense-config-file "/var/www/data/8ebe40ef-be26-4fbf-a2cd-5acf352f9f16/opensfm/undistorted/openmvs/Densify.ini" --min-resolution 507 --max-resolution 2635 --max-threads 32 --number-views-fuse 2 --sub-resolution-levels 2 -w "/var/www/data/8ebe40ef-be26-4fbf-a2cd-5acf352f9f16/opensfm/undistorted/openmvs/depthmaps" -v 0 --cuda-device -2 --geometric-iters 0
20:44:39 [App     ] Build date: Jun 24 2022, 23:48:50
20:44:39 [App     ] CPU: Intel(R) Xeon(R) Gold 6150 CPU @ 2.70GHz (32 cores)
20:44:39 [App     ] RAM: 251.73GB Physical Memory 256.00GB Virtual Memory
20:44:39 [App     ] OS: Linux 3.10.0-1160.66.1.el7.x86_64 (x86_64)
20:44:39 [App     ] Disk: 485.98GB (511.75GB) space
20:44:39 [App     ] SSE & AVX compatible CPU & OS detected
20:44:39 [App     ] Command line: DensifyPointCloud /var/www/data/8ebe40ef-be26-4fbf-a2cd-5acf352f9f16/opensfm/undistorted/openmvs/scene.mvs --resolution-level 2 --dense-config-file /var/www/data/8ebe40ef-be26-4fbf-a2cd-5acf352f9f16/opensfm/undistorted/openmvs/Densify.ini --min-resolution 507 --max-resolution 2635 --max-threads 32 --number-views-fuse 2 --sub-resolution-levels 2 -w /var/www/data/8ebe40ef-be26-4fbf-a2cd-5acf352f9f16/opensfm/undistorted/openmvs/depthmaps -v 0 --cuda-device -2 --geometric-iters 0
20:44:42 [App     ] Find a camera not pointing towards the scene center; the scene will be considered unbounded (no ROI)
20:44:53 [App     ] Preparing images for dense reconstruction completed: 2666 images (10s894ms)
20:45:37 [App     ] Selecting images for dense reconstruction completed: 0 images (44s277ms)

Fused depth-maps 0 (100%, 0ms)
20:45:38 [App     ] Densifying point-cloud completed: 0 points (55s666ms)
20:45:38 [App     ] MEMORYINFO: {
20:45:38 [App     ] 	VmPeak:	 7329912 kB
20:45:38 [App     ] 	VmSize:	 6290764 kB
20:45:38 [App     ] } ENDINFO
[INFO]    running "/code/SuperBuild/install/bin/OpenMVS/DensifyPointCloud" --filter-point-cloud -1 -i "/var/www/data/8ebe40ef-be26-4fbf-a2cd-5acf352f9f16/opensfm/undistorted/openmvs/scene_dense.mvs" -v 0 --cuda-device -2 --geometric-iters 0
20:45:38 [App     ] Build date: Jun 24 2022, 23:48:50
20:45:38 [App     ] CPU: Intel(R) Xeon(R) Gold 6150 CPU @ 2.70GHz (32 cores)
20:45:38 [App     ] RAM: 251.73GB Physical Memory 256.00GB Virtual Memory
20:45:38 [App     ] OS: Linux 3.10.0-1160.66.1.el7.x86_64 (x86_64)
20:45:38 [App     ] Disk: 50.46GB (69.98GB) space
20:45:38 [App     ] SSE & AVX compatible CPU & OS detected
20:45:38 [App     ] Command line: DensifyPointCloud --filter-point-cloud -1 -i /var/www/data/8ebe40ef-be26-4fbf-a2cd-5acf352f9f16/opensfm/undistorted/openmvs/scene_dense.mvs -v 0 --cuda-device -2 --geometric-iters 0
20:45:38 [App     ] error: no valid point-cloud for the ROI estimation

===== Dumping Info for Geeks (developers need this to fix bugs) =====
Child returned 1
Traceback (most recent call last):
File "/code/stages/odm_app.py", line 94, in execute
self.first_stage.run()
File "/code/opendm/types.py", line 347, in run
self.next_stage.run(outputs)
File "/code/opendm/types.py", line 347, in run
self.next_stage.run(outputs)
File "/code/opendm/types.py", line 347, in run
self.next_stage.run(outputs)
[Previous line repeated 1 more time]
File "/code/opendm/types.py", line 328, in run
self.process(self.args, outputs)
File "/code/stages/openmvs.py", line 206, in process
system.run('"%s" %s' % (context.omvs_densify_path, ' '.join(config + gpu_config + extra_config)))
File "/code/opendm/system.py", line 106, in run
raise SubprocessException("Child returned {}".format(retcode), retcode)
opendm.system.SubprocessException: Child returned 1

===== Done, human-readable information to follow... =====

[ERROR]   Uh oh! Processing stopped because of strange values in the reconstruction. This is often a sign that the input data has some issues or the software cannot deal with it. Have you followed best practices for data acquisition? See https://docs.opendronemap.org/flying/
1 Like

No depth maps were generated… Maybe you need to turn pc-quality and feature-quality up a bit?

1 Like

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.