How did you install ODM? (Docker, installer, natively, …)?
Singularity
singularity pull --disable-cache docker://opendronemap/odm:latest
singularity pull --disable-cache docker://opendronemap/odm:2.8.6
Using singularity 3.9.6
Currently Loaded Modulefiles:
- singularity/3.9.6(default)
What is the problem?
I was testing out using Singularity to run the container for jobs on a HPC cluster but for some reason when I try to execute a simple test job I get the error below:
[[email protected] odm_processing_code]$ singularity run --bind $images_dir:/$output_dir/code/images --writable-tmpfs odm_latest.sif --orthophoto-png --dtm --orthophoto-kmz --project-path $output_dir
[INFO] DTM is turned on, automatically turning on point cloud classification
[INFO] Initializing ODM ? - Tue Jul 19 04:12:41 2022
[INFO] ==============
[INFO] 3d_tiles: False
[INFO] auto_boundary: False
[INFO] auto_boundary_distance: 0
[INFO] boundary: {}
[INFO] build_overviews: False
[INFO] camera_lens: auto
[INFO] cameras: {}
[INFO] cog: False
[INFO] copy_to: None
[INFO] crop: 3
[INFO] debug: False
[INFO] dem_decimation: 1
[INFO] dem_euclidean_map: False
[INFO] dem_gapfill_steps: 3
[INFO] dem_resolution: 5
[INFO] depthmap_resolution: 640
[INFO] dsm: False
[INFO] dtm: True
[INFO] end_with: odm_postprocess
[INFO] fast_orthophoto: False
[INFO] feature_quality: high
[INFO] feature_type: sift
[INFO] force_gps: False
[INFO] gcp: None
[INFO] geo: None
[INFO] gps_accuracy: 10
[INFO] ignore_gsd: False
[INFO] matcher_neighbors: 0
[INFO] matcher_type: flann
[INFO] max_concurrency: 36
[INFO] merge: all
[INFO] mesh_octree_depth: 11
[INFO] mesh_size: 200000
[INFO] min_num_features: 10000
[INFO] name: code
[INFO] no_gpu: False
[INFO] optimize_disk_space: False
[INFO] orthophoto_compression: DEFLATE
[INFO] orthophoto_cutline: False
[INFO] orthophoto_kmz: True
[INFO] orthophoto_no_tiled: False
[INFO] orthophoto_png: True
[INFO] orthophoto_resolution: 5
[INFO] pc_classify: True
[INFO] pc_copc: False
[INFO] pc_csv: False
[INFO] pc_ept: False
[INFO] pc_filter: 2.5
[INFO] pc_geometric: False
[INFO] pc_las: False
[INFO] pc_quality: medium
[INFO] pc_rectify: False
[INFO] pc_sample: 0
[INFO] pc_tile: False
[INFO] primary_band: auto
[INFO] project_path: /data2/RS_GIS_Data/UAS_Data/NGEEArctic/UAS_Flights/Seward_2021/Skydio2/Kougarok/20210816/Flight_2/odm
[INFO] radiometric_calibration: none
[INFO] rerun: None
[INFO] rerun_all: False
[INFO] rerun_from: None
[INFO] resize_to: 2048
[INFO] rolling_shutter: False
[INFO] rolling_shutter_readout: 0
[INFO] sfm_algorithm: incremental
[INFO] skip_3dmodel: False
[INFO] skip_band_alignment: False
[INFO] skip_orthophoto: False
[INFO] skip_report: False
[INFO] sky_removal: False
[INFO] sm_cluster: None
[INFO] smrf_scalar: 1.25
[INFO] smrf_slope: 0.15
[INFO] smrf_threshold: 0.5
[INFO] smrf_window: 18.0
[INFO] split: 999999
[INFO] split_image_groups: None
[INFO] split_overlap: 150
[INFO] texturing_data_term: gmi
[INFO] texturing_keep_unseen_faces: False
[INFO] texturing_outlier_removal_type: gauss_clamping
[INFO] texturing_skip_global_seam_leveling: False
[INFO] texturing_skip_local_seam_leveling: False
[INFO] texturing_tone_mapping: none
[INFO] tiles: False
[INFO] time: False
[INFO] use_3dmesh: False
[INFO] use_exif: False
[INFO] use_fixed_camera_params: False
[INFO] use_hybrid_bundle_adjustment: False
[INFO] verbose: False
[INFO] ==============
[INFO] Running dataset stage
[INFO] Loading dataset from: /data2/RS_GIS_Data/UAS_Data/NGEEArctic/UAS_Flights/Seward_2021/Skydio2/Kougarok/20210816/Flight_2/odm/code/images
[INFO] Loading 187 images
[INFO] Wrote images database: /data2/RS_GIS_Data/UAS_Data/NGEEArctic/UAS_Flights/Seward_2021/Skydio2/Kougarok/20210816/Flight_2/odm/code/images.json
[INFO] Found 187 usable images
[INFO] Parsing SRS header: WGS84 UTM 3N
[INFO] Finished dataset stage
[INFO] Running split stage
[INFO] Normal dataset, will process all at once.
[INFO] Finished split stage
[INFO] Running merge stage
[INFO] Normal dataset, nothing to merge.
[INFO] Finished merge stage
[INFO] Running opensfm stage
[INFO] Maximum photo dimensions: 4056px
[INFO] Photo dimensions for feature extraction: 2028px
[INFO] nvidia-smi not found in PATH, using CPU
[INFO] Altitude data detected, enabling it for GPS alignment
[INFO] ['use_exif_size: no', 'flann_algorithm: KDTREE', 'feature_process_size: 2028', 'feature_min_frames: 10000', 'processes: 36', 'matching_gps_neighbors: 0', 'matching_gps_distance: 0', 'matching_graph_rounds: 50', 'optimize_camera_parameters: yes', 'reconstruction_algorithm: incremental', 'undistorted_image_format: tif', 'bundle_outlier_filtering_type: AUTO', 'sift_peak_threshold: 0.066', 'align_orientation_prior: vertical', 'triangulation_type: ROBUST', 'retriangulation_ratio: 2', 'matcher_type: FLANN', 'feature_type: SIFT', 'use_altitude_tag: yes', 'align_method: auto', 'local_bundle_radius: 0']
[INFO] Wrote reference_lla.json
[INFO] running "/code/SuperBuild/install/bin/opensfm/bin/opensfm" detect_features "/data2/RS_GIS_Data/UAS_Data/NGEEArctic/UAS_Flights/Seward_2021/Skydio2/Kougarok/20210816/Flight_2/odm/code/opensfm"
Traceback (most recent call last):
File "/code/SuperBuild/install/bin/opensfm/bin/opensfm_main.py", line 9, in <module>
from opensfm import commands
File "/code/SuperBuild/install/bin/opensfm/opensfm/commands/__init__.py", line 1, in <module>
from . import (
File "/code/SuperBuild/install/bin/opensfm/opensfm/commands/compute_statistics.py", line 4, in <module>
from opensfm.actions import compute_statistics
File "/code/SuperBuild/install/bin/opensfm/opensfm/actions/compute_statistics.py", line 5, in <module>
from opensfm import stats
File "/code/SuperBuild/install/bin/opensfm/opensfm/stats.py", line 13, in <module>
import matplotlib as mpl
File "/home/sserbin/.local/lib/python3.9/site-packages/matplotlib/__init__.py", line 105, in <module>
from packaging.version import parse as parse_version
ModuleNotFoundError: No module named 'packaging'
===== Dumping Info for Geeks (developers need this to fix bugs) =====
Child returned 1
Traceback (most recent call last):
File "/code/stages/odm_app.py", line 94, in execute
self.first_stage.run()
File "/code/opendm/types.py", line 347, in run
self.next_stage.run(outputs)
File "/code/opendm/types.py", line 347, in run
self.next_stage.run(outputs)
File "/code/opendm/types.py", line 347, in run
self.next_stage.run(outputs)
File "/code/opendm/types.py", line 328, in run
self.process(self.args, outputs)
File "/code/stages/run_opensfm.py", line 35, in process
octx.feature_matching(self.rerun())
File "/code/opendm/osfm.py", line 410, in feature_matching
raise e
File "/code/opendm/osfm.py", line 400, in feature_matching
self.run('detect_features')
File "/code/opendm/osfm.py", line 34, in run
system.run('"%s" %s "%s"' %
File "/code/opendm/system.py", line 106, in run
raise SubprocessException("Child returned {}".format(retcode), retcode)
opendm.system.SubprocessException: Child returned 1
===== Done, human-readable information to follow... =====
[ERROR] Uh oh! Processing stopped because of strange values in the reconstruction. This is often a sign that the input data has some issues or the software cannot deal with it. Have you followed best practices for data acquisition? See https://docs.opendronemap.org/flying/
I’ve even simplified more
singularity run --bind $images_dir:/$output_dir/code/images --writable-tmpfs odm_latest.sif --orthophoto-png --dtm --orthophoto-kmz --project-path $output_dir
Same error
Interestingly the same source containers mostly work in podman but fail for another reason late in the processing, but I dont yet know why that is. I think perhaps an issue with trying to create the DTM. The example above does seem to generally work with podman/docker.
Not really sure why in Singularity it would be telling me a package is missing.
What should be the expected behavior? If this is a feature request, please describe in detail the changes you think should be made to the code, citing files and lines where changes should be made, if possible.
I would expect the job to execute and generate an orthophoto.
How can we reproduce this? What steps did you do to trigger the problem? If this is an issue with processing a dataset, YOU MUST include a copy of your dataset uploaded on Google Drive or Dropbox (otherwise we cannot reproduce this).
images_dir=/data2/RS_GIS_Data/UAS_Data/NGEEArctic/UAS_Flights/Seward_2021/Skydio2/Kougarok/20210816/Flight_2/images
name=`basename $images_dir`
output_dir=/data2/RS_GIS_Data/UAS_Data/NGEEArctic/UAS_Flights/Seward_2021/Skydio2/Kougarok/20210816/Flight_2/odm
mkdir -p $output_dir
singularity run --bind $images_dir:/$output_dir/code/images \
--writable-tmpfs odm_latest.sif \
--orthophoto-png --dtm --orthophoto-kmz --feature-quality ultra --max-concurrency 16 \
--use-hybrid-bundle-adjustment --build-overviews --min-num-features 10000 \
--project-path $output_dir
I suspect I may not be invoking the run correctly. Any insight would be much appreciated. Thanks!
I use/create a lot of containerized environments that run in Docker and Singularity but I seem to not really understand how the code expects to be provided the location of the source image files and output(s). I think its also trying to create or point to the code folder which might be where its looking for the python code. Perhaps I just need a little more help on how to structure my folders and the singularity run command to find the folders and have the data in the expected location(s)