Densify Point Cloud | Error loading libcuda.so.1

sin-2pi · February 8, 2024, 12:52am

Not sure what to do when encountering this error. I am using a singularity image to run it as well. Does anyone have insight on how to load this library correctly?

I am running it on two NVIDIA A30 GPUs and using the linux script from here. Tutorials — OpenDroneMap 3.3.4 documentation

[INFO]    running "/code/SuperBuild/install/bin/OpenMVS/DensifyPointCloud" "/project/drone_ai/nyoung8/odm/code/opensfm/undistorted/openmvs/scene.mvs" --resolution-level 3 --dense-config-file "/project/drone_ai/nyoung8/odm/code/opensfm/undistorted/openmvs/Densify.ini" --max-resolution 4864 --max-threads 16 --number-views-fuse 2 --sub-resolution-levels 2 --archive-type 3 -w "/project/drone_ai/nyoung8/odm/code/opensfm/undistorted/openmvs/depthmaps" -v 0 --cuda-device -2
/code/SuperBuild/install/bin/OpenMVS/DensifyPointCloud: error while loading shared libraries: libcuda.so.1: cannot open shared object file: No such file or directory

===== Dumping Info for Geeks (developers need this to fix bugs) =====
Child returned 127
Traceback (most recent call last):
  File "/code/stages/odm_app.py", line 82, in execute
    self.first_stage.run()
  File "/code/opendm/types.py", line 438, in run
    self.next_stage.run(outputs)
  File "/code/opendm/types.py", line 438, in run
    self.next_stage.run(outputs)
  File "/code/opendm/types.py", line 438, in run
    self.next_stage.run(outputs)
  [Previous line repeated 1 more time]
  File "/code/opendm/types.py", line 417, in run
    self.process(self.args, outputs)
  File "/code/stages/openmvs.py", line 119, in process
    raise e
  File "/code/stages/openmvs.py", line 105, in process
    run_densify()
  File "/code/stages/openmvs.py", line 100, in run_densify
    system.run('"%s" "%s" %s' % (context.omvs_densify_path,
  File "/code/opendm/system.py", line 112, in run
    raise SubprocessException("Child returned {}".format(retcode), retcode)
opendm.system.SubprocessException: Child returned 127

===== Done, human-readable information to follow... =====

smathermather · February 8, 2024, 7:12pm

Are you running on an HPC?

sin-2pi · February 8, 2024, 10:32pm

Yes I am.

smathermather · February 9, 2024, 5:01am

Interesting. I think you have to run the --nv flag to enable GPU in your singularity container. Can you try and revert whether or not it works? If it works, we can update the docs to reflect:

images_dir=/path_to_image_dir/
name=`basename $images_dir`
output_dir=/path_to_output_directories/$name
mkdir -p $output_dir

singularity run
--nv \
--bind $images_dir:/$output_dir/code/images,\
--writable-tmpfs odm_latest.sif  \
--orthophoto-png --mesh-octree-depth 12 --ignore-gsd --dtm \
--smrf-threshold 0.4 --smrf-window 24 --dsm --pc-csv --pc-las --orthophoto-kmz \
--ignore-gsd  --matcher-type flann --feature-quality ultra --max-concurrency 16 \
--use-hybrid-bundle-adjustment --build-overviews --time --min-num-features 10000 \
--project-path $output_dir

sin-2pi · February 9, 2024, 8:37pm

It appeared to fix the last issue, however in turn I got a segmentation fault

13:36:19 [App     ] error: no valid point-cloud for the ROI estimation
Segmentation fault (core dumped)

===== Dumping Info for Geeks (developers need this to fix bugs) =====
Child returned 139
Traceback (most recent call last):
  File "/code/stages/odm_app.py", line 82, in execute
    self.first_stage.run()
  File "/code/opendm/types.py", line 438, in run
    self.next_stage.run(outputs)
  File "/code/opendm/types.py", line 438, in run
    self.next_stage.run(outputs)
  File "/code/opendm/types.py", line 438, in run
    self.next_stage.run(outputs)
  [Previous line repeated 1 more time]
  File "/code/opendm/types.py", line 417, in run
    self.process(self.args, outputs)
  File "/code/stages/openmvs.py", line 230, in process
    raise e
  File "/code/stages/openmvs.py", line 224, in process
    system.run('"%s" %s' % (context.omvs_densify_path, ' '.join(config + gpu_config + extra_config)))
  File "/code/opendm/system.py", line 112, in run
    raise SubprocessException("Child returned {}".format(retcode), retcode)
opendm.system.SubprocessException: Child returned 139

===== Done, human-readable information to follow... =====

smathermather · February 12, 2024, 1:49pm

At first blush, it looks like an OOM. What are your processing settings and what are your machine sizes? Always best to do testing on defaults and then turn the knobs from there.

sin-2pi · February 12, 2024, 9:59pm

That is what I thought as well, however I am running on two NVIDIA A30 GPUs, so I can’t imagine memory in that area would be a problem. I will try playing with the settings.

I am using the base processing settings from the singularity linux script in the documentation.

smathermather · February 12, 2024, 10:30pm

What is your image dataset size and the size of the memory on your CPU? If it OOMs on GPU, it just fails back to CPU, but I suspect this is not a GPU stage that’s OOMing but a CPU stage.

Also, if you can include more of the log, it’s hard to see what is happening without more of the context for the OOM.

sin-2pi · February 13, 2024, 12:27am

I am running it on 10 images with a size of 75mb. From the log it is showing just over a terabyte of physical memory and 8GB of virtual memory.

17:23:24 [App     ] OpenMVS x64 v2.2.0
17:23:24 [App     ] Build date: Dec 12 2023, 21:01:30
17:23:24 [App     ] CPU: Intel(R) Xeon(R) Gold 6330 CPU @ 2.00GHz (56 cores)
17:23:24 [App     ] RAM: 1007.34GB Physical Memory 8.00GB Virtual Memory
17:23:24 [App     ] OS: Linux 4.18.0-513.11.1.el8_9.x86_64 (x86_64)
17:23:24 [App     ] Disk:
17:23:24 [App     ] SSE & AVX compatible CPU & OS detected
17:23:24 [App     ] Command line: DensifyPointCloud /project/drone_ai/nyoung8/odm/code/opensfm/undistorted/openmvs/scene.mvs --resolution-level 3 --dense-config-file /project/drone_ai/nyoung8/odm/code/opensfm/undistorted/openmvs/Densify.ini --max-resolution 4864 --max-threads 16 --number-views-fuse 2 --sub-resolution-levels 2 --archive-type 3 -w /project/drone_ai/nyoung8/odm/code/opensfm/undistorted/openmvs/depthmaps -v 0 --cuda-device -1
17:23:24 [App     ] MapSMtoCores for SM 8.0 is undefined; default to use 64 cores/SM
17:23:24 [App     ] MapSMtoCores for SM 8.0 is undefined; default to use 64 cores/SM
17:23:25 [App     ] The camera directions mean is unbalanced; the scene will be considered unbounded (no ROI)
17:23:25 [App     ] Preparing images for dense reconstruction completed: 8 images (587ms)
17:23:25 [App     ] Selecting images for dense reconstruction completed: 0 images (0ms)

Fused depth-maps 0 (100%, 0ms)
17:23:25 [App     ] Densifying point-cloud completed: 0 points (587ms)
17:23:25 [App     ] MEMORYINFO: {
17:23:25 [App     ] 	VmPeak:	12313208 kB
17:23:25 [App     ] 	VmSize:	11769280 kB
17:23:25 [App     ] } ENDINFO
[INFO]    running "/code/SuperBuild/install/bin/OpenMVS/DensifyPointCloud" --filter-point-cloud -20 -i "/project/drone_ai/nyoung8/odm/code/opensfm/undistorted/openmvs/scene_dense.mvs" -v 0 --cuda-device -1
17:23:25 [App     ] OpenMVS x64 v2.2.0
17:23:25 [App     ] Build date: Dec 12 2023, 21:01:30
17:23:25 [App     ] CPU: Intel(R) Xeon(R) Gold 6330 CPU @ 2.00GHz (56 cores)
17:23:25 [App     ] RAM: 1007.34GB Physical Memory 8.00GB Virtual Memory
17:23:25 [App     ] OS: Linux 4.18.0-513.11.1.el8_9.x86_64 (x86_64)
17:23:25 [App     ] Disk:
17:23:25 [App     ] SSE & AVX compatible CPU & OS detected
17:23:25 [App     ] Command line: DensifyPointCloud --filter-point-cloud -20 -i /project/drone_ai/nyoung8/odm/code/opensfm/undistorted/openmvs/scene_dense.mvs -v 0 --cuda-device -1
17:23:25 [App     ] MapSMtoCores for SM 8.0 is undefined; default to use 64 cores/SM
17:23:25 [App     ] MapSMtoCores for SM 8.0 is undefined; default to use 64 cores/SM
17:23:26 [App     ] error: no valid point-cloud for the ROI estimation
Segmentation fault (core dumped)

===== Dumping Info for Geeks (developers need this to fix bugs) =====
Child returned 139
Traceback (most recent call last):
  File "/code/stages/odm_app.py", line 82, in execute
    self.first_stage.run()
  File "/code/opendm/types.py", line 438, in run
    self.next_stage.run(outputs)
  File "/code/opendm/types.py", line 438, in run
    self.next_stage.run(outputs)
  File "/code/opendm/types.py", line 438, in run
    self.next_stage.run(outputs)
  [Previous line repeated 1 more time]
  File "/code/opendm/types.py", line 417, in run
    self.process(self.args, outputs)
  File "/code/stages/openmvs.py", line 230, in process
    raise e
  File "/code/stages/openmvs.py", line 224, in process
    system.run('"%s" %s' % (context.omvs_densify_path, ' '.join(config + gpu_config + extra_config)))
  File "/code/opendm/system.py", line 112, in run
    raise SubprocessException("Child returned {}".format(retcode), retcode)
opendm.system.SubprocessException: Child returned 139

===== Done, human-readable information to follow... =====

[ERROR]   Uh oh! Processing stopped because of strange values in the reconstruction. This is often a sign that the input data has some issues or the software cannot deal with it. Have you followed best practices for data acquisition? See https://docs.opendronemap.org/flying/

smathermather · February 13, 2024, 2:44am

Can you share the dataset you are running?

sin-2pi · February 13, 2024, 8:05am

Sure. Here are the ten images. I appreciate the help!
https://drive.google.com/drive/folders/1k_pLr619YFwLBKrOnKFAGQWAg6bTdtCr?usp=sharing

Gordon · February 13, 2024, 8:38am

First go with 10000 min-num-features looks the same, so I suspect not enough overlap, plus it’s just a single line of images. The DEMs are more complete, but with artifacts. Single reconstruction reported, but it is a number of just off E-W stripes.

Gordon · February 13, 2024, 10:25pm

Running this task with various settings I discovered that DSPSIFT completes (the ortho is broken), but SIFT fails, with everything else identical.

[Copy of Copy of Copy of Montezuma County - 20/07/2018](javascript:void(0)

00:01:34

Processing failed (4294967295)

|Options:|dem-resolution: 1.0, dsm: true, dtm: true, feature-quality: ultra, gps-accuracy: 10, mesh-octree-depth: 12, mesh-size: 500000, min-num-features: 30000, optimize-disk-space: true, orthophoto-resolution: 0.5, pc-filter: 4, pc-quality: ultra, pc-rectify: true, rerun-from: dataset, use-3dmesh: true|

…

00:04:12 Completed

|Options:|dsm: true, dtm: true, feature-quality: ultra, feature-type: dspsift, gps-accuracy: 10, mesh-octree-depth: 12, mesh-size: 500000, min-num-features: 30000, optimize-disk-space: true, orthophoto-resolution: 0.5, pc-filter: 4, pc-quality: ultra, pc-rectify: true, rerun-from: dataset, use-3dmesh: true|
|Average GSD:|0.32 cm|
|Area:|57.98 m²|
|Reconstructed Points:|1,236,397|

sin-2pi · February 14, 2024, 1:11am

Thank you! I will try these settings. Just to clarify, would I put them in this script?

images_dir=/path_to_image_dir/
name=`basename $images_dir`
output_dir=/path_to_output_directories/$name
mkdir -p $output_dir

singularity run
--nv \
--bind $images_dir:/$output_dir/code/images,\
--writable-tmpfs odm_latest.sif  \
--orthophoto-png --mesh-octree-depth 12 --ignore-gsd --dtm \
--smrf-threshold 0.4 --smrf-window 24 --dsm --pc-csv --pc-las --orthophoto-kmz \
--ignore-gsd  --matcher-type flann --feature-quality ultra --max-concurrency 16 \
--use-hybrid-bundle-adjustment --build-overviews --time --min-num-features 10000 \
--project-path $output_dir