ODM + ClusterODM + NodeODM + Submodels

ODM
#1

Hi,

I have a configuration like this:

<NODE-ODM> <----------><CLUSTER-ODM><------------> <NODE-ODM>
_Ubuntu 18.06___________Ubuntu 18.06_____________Ubuntu 18.06
_ machine 1______________machine 1________________machine 2

So I executed this command on machine 2 where the images were present.

docker run -ti --rm -v /$(pwd)/hinduja:/datasets/code opendronemap/odm --project-path /datasets --verbose --resize-to -1 --split 200 --split-overlap 70 --sm-cluster http://<clsuter-odm-ip>:3001 --dsm --dtm --depthmap-resolution 1000 --dem-resolution 2 --orthophoto-resolution 2 --ignore-gsd --crop 0

where <cluster-odm-ip> was my cluster odm IP(IP of 1st machine).

I got the below report along with error.

azureuser@odm2:~/Downloads$ docker run -ti --rm -v /$(pwd)/hinduja:/datasets/code opendronemap/odm --project-path /datasets --verbose --resize-to -1 --split 200 --split-overlap 70 --sm-cluster http://20.184.5.64:3001 --dsm --dtm --depthmap-resolution 1000 --dem-resolution 2 --orthophoto-resolution 2 --ignore-gsd --crop 0
/usr/local/lib/python2.7/dist-packages/OpenSSL/crypto.py:12: CryptographyDeprecationWarning: Python 2 is no longer supported by the Python core team. Support for it is now deprecated in cryptography, and will be removed in a future release.
  from cryptography import x509
[INFO]    DTM is turned on, automatically turning on point cloud classification
[INFO]    Initializing ODM - Wed Sep 23 16:36:41  2020
[INFO]    ==============
[INFO]    build_overviews: False
[INFO]    camera_lens: auto
[INFO]    cameras: {}
[INFO]    crop: 0.0
[INFO]    debug: False
[INFO]    dem_decimation: 1
[INFO]    dem_euclidean_map: False
[INFO]    dem_gapfill_steps: 3
[INFO]    dem_resolution: 2.0
[INFO]    depthmap_resolution: 1000.0
[INFO]    dsm: True
[INFO]    dtm: True
[INFO]    end_with: odm_report
[INFO]    fast_orthophoto: False
[INFO]    feature_type: sift
[INFO]    force_gps: False
[INFO]    gcp: None
[INFO]    gps_accuracy: 15
[INFO]    ignore_gsd: True
[INFO]    matcher_distance: 0
[INFO]    matcher_neighbors: 8
[INFO]    max_concurrency: 16
[INFO]    merge: all
[INFO]    mesh_octree_depth: 10
[INFO]    mesh_point_weight: 4
[INFO]    mesh_samples: 1.0
[INFO]    mesh_size: 200000
[INFO]    min_num_features: 8000
[INFO]    mve_confidence: 0.6
[INFO]    name: code
[INFO]    opensfm_depthmap_method: PATCH_MATCH
[INFO]    opensfm_depthmap_min_consistent_views: 3
[INFO]    opensfm_depthmap_min_patch_sd: 1
[INFO]    optimize_disk_space: False
[INFO]    orthophoto_compression: DEFLATE
[INFO]    orthophoto_cutline: False
[INFO]    orthophoto_no_tiled: False
[INFO]    orthophoto_png: False
[INFO]    orthophoto_resolution: 2.0
[INFO]    pc_classify: True
[INFO]    pc_csv: False
[INFO]    pc_ept: False
[INFO]    pc_filter: 2.5
[INFO]    pc_las: False
[INFO]    pc_rectify: False
[INFO]    pc_sample: 0
[INFO]    project_path: /datasets
[INFO]    radiometric_calibration: none
[INFO]    rerun: None
[INFO]    rerun_all: False
[INFO]    rerun_from: None
[INFO]    resize_to: -1
[INFO]    skip_3dmodel: False
[INFO]    sm_cluster: True
[INFO]    smrf_scalar: 1.25
[INFO]    smrf_slope: 0.15
[INFO]    smrf_threshold: 0.5
[INFO]    smrf_window: 18.0
[INFO]    split: 200
[INFO]    split_multitracks: False
[INFO]    split_overlap: 70.0
[INFO]    texturing_data_term: gmi
[INFO]    texturing_keep_unseen_faces: False
[INFO]    texturing_nadir_weight: 16
[INFO]    texturing_outlier_removal_type: gauss_clamping
[INFO]    texturing_skip_global_seam_leveling: False
[INFO]    texturing_skip_hole_filling: False
[INFO]    texturing_skip_local_seam_leveling: False
[INFO]    texturing_skip_visibility_test: False
[INFO]    texturing_tone_mapping: none
[INFO]    time: False
[INFO]    use_3dmesh: False
[INFO]    use_exif: False
[INFO]    use_fixed_camera_params: False
[INFO]    use_hybrid_bundle_adjustment: False
[INFO]    use_opensfm_dense: False
[INFO]    verbose: True
[INFO]    ==============
[INFO]    Running dataset stage
[INFO]    Loading dataset from: /datasets/code/images
[INFO]    Loading images database: /datasets/code/images.json
[INFO]    Found 452 usable images
[INFO]    Coordinates file already exist: /datasets/code/odm_georeferencing/coords.txt
[INFO]    Parsing SRS header: WGS84 UTM 44N
[INFO]    Finished dataset stage
[INFO]    Running split stage
[INFO]    Setting max-concurrency to 15 to better handle remote splits
[INFO]    Large dataset detected (452 photos) and split set at 200. Preparing split merge.
[WARNING] /datasets/code/opensfm/image_list.txt already exists, not rerunning OpenSfM setup
[INFO]    running /usr/bin/env python2 /code/SuperBuild/src/opensfm/bin/opensfm create_submodels "/datasets/code/opensfm"
[INFO]    Dataset has been split into 6 submodels. Reconstructing each submodel...
[INFO]    LRE: Initializing using cluster node 20.184.5.64:3001
[INFO]    LRE: Node is online and running odm version 1.0.2
[INFO]    LRE: Adding to queue /datasets/code/opensfm/submodels
[INFO]    LRE: Adding to queue /datasets/code/opensfm/submodels
[INFO]    LRE: Adding to queue /datasets/code/opensfm/submodels
[INFO]    LRE: Adding to queue /datasets/code/opensfm/submodels
[INFO]    LRE: Adding to queue /datasets/code/opensfm/submodels
[INFO]    LRE: Adding to queue /datasets/code/opensfm/submodels
[INFO]    LRE: About to process submodels locally
[INFO]    ==================================
[INFO]    Local Reconstruction submodels
[INFO]    ==================================
[INFO]    running /usr/bin/env python2 /code/SuperBuild/src/opensfm/bin/opensfm detect_features "/datasets/code/opensfm/submodels/opensfm"
[INFO]    LRE: About to process submodels remotely
[INFO]    LRE: Upload of submodels at [100%]
[WARNING] LRE: submodels failed with: Not enough images (1 files uploaded)
[INFO]    LRE: Re-queueing submodels (retries: 1)
[INFO]    LRE: About to process submodels remotely
[INFO]    LRE: Upload of submodels at [100%]
[WARNING] LRE: submodels failed with: Not enough images (1 files uploaded)
[INFO]    LRE: Re-queueing submodels (retries: 1)
[INFO]    LRE: About to process submodels remotely
[INFO]    LRE: Upload of submodels at [100%]
[WARNING] LRE: submodels failed with: Not enough images (1 files uploaded)
[INFO]    LRE: Re-queueing submodels (retries: 1)
[INFO]    LRE: About to process submodels remotely
[INFO]    LRE: Upload of submodels at [100%]
[WARNING] LRE: submodels failed with: Not enough images (1 files uploaded)
[INFO]    LRE: Re-queueing submodels (retries: 1)
[INFO]    LRE: About to process submodels remotely
[INFO]    LRE: Upload of submodels at [100%]
[WARNING] LRE: submodels failed with: Not enough images (1 files uploaded)
[INFO]    LRE: Re-queueing submodels (retries: 1)
[INFO]    LRE: About to process submodels remotely
[INFO]    LRE: Waiting 10 seconds before processing submodels
Traceback (most recent call last):
  File "/code/SuperBuild/src/opensfm/bin/opensfm", line 34, in <module>
    command.run(args)
  File "/code/SuperBuild/src/opensfm/opensfm/commands/detect_features.py", line 33, in run
    with open(data.profile_log(), 'a') as fout:
IOError: [Errno 2] No such file or directory: '/datasets/code/opensfm/submodels/opensfm/profile.log'
[WARNING] LRE: submodels failed with: Child returned 1
[INFO]    LRE: Re-queueing submodels (retries: 1)
[INFO]    LRE: About to process submodels locally
[INFO]    ==================================
[INFO]    Local Reconstruction submodels
[INFO]    ==================================
[INFO]    running /usr/bin/env python2 /code/SuperBuild/src/opensfm/bin/opensfm detect_features "/datasets/code/opensfm/submodels/opensfm"
Traceback (most recent call last):
  File "/code/SuperBuild/src/opensfm/bin/opensfm", line 34, in <module>
    command.run(args)
  File "/code/SuperBuild/src/opensfm/opensfm/commands/detect_features.py", line 33, in run
    with open(data.profile_log(), 'a') as fout:
IOError: [Errno 2] No such file or directory: '/datasets/code/opensfm/submodels/opensfm/profile.log'
[WARNING] LRE: submodels failed with: Child returned 1
[INFO]    LRE: Re-queueing submodels (retries: 2)
[INFO]    LRE: About to process submodels locally
[INFO]    ==================================
[INFO]    Local Reconstruction submodels
[INFO]    ==================================
[INFO]    running /usr/bin/env python2 /code/SuperBuild/src/opensfm/bin/opensfm detect_features "/datasets/code/opensfm/submodels/opensfm"
Traceback (most recent call last):
  File "/code/SuperBuild/src/opensfm/bin/opensfm", line 34, in <module>
    command.run(args)
  File "/code/SuperBuild/src/opensfm/opensfm/commands/detect_features.py", line 33, in run
    with open(data.profile_log(), 'a') as fout:
IOError: [Errno 2] No such file or directory: '/datasets/code/opensfm/submodels/opensfm/profile.log'
[WARNING] LRE: submodels failed with: Child returned 1
[INFO]    LRE: Re-queueing submodels (retries: 2)
[INFO]    LRE: About to process submodels locally
[INFO]    ==================================
[INFO]    Local Reconstruction submodels
[INFO]    ==================================
[INFO]    running /usr/bin/env python2 /code/SuperBuild/src/opensfm/bin/opensfm detect_features "/datasets/code/opensfm/submodels/opensfm"
Traceback (most recent call last):
  File "/code/SuperBuild/src/opensfm/bin/opensfm", line 34, in <module>
    command.run(args)
  File "/code/SuperBuild/src/opensfm/opensfm/commands/detect_features.py", line 33, in run
    with open(data.profile_log(), 'a') as fout:
IOError: [Errno 2] No such file or directory: '/datasets/code/opensfm/submodels/opensfm/profile.log'
[WARNING] LRE: submodels failed with: Child returned 1
[INFO]    LRE: Re-queueing submodels (retries: 2)
[INFO]    LRE: About to process submodels locally
[INFO]    ==================================
[INFO]    Local Reconstruction submodels
[INFO]    ==================================
[INFO]    running /usr/bin/env python2 /code/SuperBuild/src/opensfm/bin/opensfm detect_features "/datasets/code/opensfm/submodels/opensfm"
Traceback (most recent call last):
  File "/code/SuperBuild/src/opensfm/bin/opensfm", line 34, in <module>
    command.run(args)
  File "/code/SuperBuild/src/opensfm/opensfm/commands/detect_features.py", line 33, in run
    with open(data.profile_log(), 'a') as fout:
IOError: [Errno 2] No such file or directory: '/datasets/code/opensfm/submodels/opensfm/profile.log'
[WARNING] LRE: submodels failed with: Child returned 1
[INFO]    LRE: Re-queueing submodels (retries: 2)
[INFO]    LRE: About to process submodels locally
[INFO]    ==================================
[INFO]    Local Reconstruction submodels
[INFO]    ==================================
[INFO]    running /usr/bin/env python2 /code/SuperBuild/src/opensfm/bin/opensfm detect_features "/datasets/code/opensfm/submodels/opensfm"
Traceback (most recent call last):
  File "/code/SuperBuild/src/opensfm/bin/opensfm", line 34, in <module>
    command.run(args)
  File "/code/SuperBuild/src/opensfm/opensfm/commands/detect_features.py", line 33, in run
    with open(data.profile_log(), 'a') as fout:
IOError: [Errno 2] No such file or directory: '/datasets/code/opensfm/submodels/opensfm/profile.log'
[WARNING] LRE: submodels failed with: Child returned 1
[INFO]    LRE: Re-queueing submodels (retries: 2)
[INFO]    LRE: About to process submodels locally
[INFO]    ==================================
[INFO]    Local Reconstruction submodels
[INFO]    ==================================
[INFO]    running /usr/bin/env python2 /code/SuperBuild/src/opensfm/bin/opensfm detect_features "/datasets/code/opensfm/submodels/opensfm"
Traceback (most recent call last):
  File "/code/SuperBuild/src/opensfm/bin/opensfm", line 34, in <module>
    command.run(args)
  File "/code/SuperBuild/src/opensfm/opensfm/commands/detect_features.py", line 33, in run
    with open(data.profile_log(), 'a') as fout:
IOError: [Errno 2] No such file or directory: '/datasets/code/opensfm/submodels/opensfm/profile.log'
[WARNING] LRE: submodels failed with: Child returned 1
[INFO]    LRE: Re-queueing submodels (retries: 3)
[INFO]    LRE: About to process submodels locally
[INFO]    ==================================
[INFO]    Local Reconstruction submodels
[INFO]    ==================================
[INFO]    running /usr/bin/env python2 /code/SuperBuild/src/opensfm/bin/opensfm detect_features "/datasets/code/opensfm/submodels/opensfm"
Traceback (most recent call last):
  File "/code/SuperBuild/src/opensfm/bin/opensfm", line 34, in <module>
    command.run(args)
  File "/code/SuperBuild/src/opensfm/opensfm/commands/detect_features.py", line 33, in run
    with open(data.profile_log(), 'a') as fout:
IOError: [Errno 2] No such file or directory: '/datasets/code/opensfm/submodels/opensfm/profile.log'
[WARNING] LRE: submodels failed with: Child returned 1
[INFO]    LRE: Re-queueing submodels (retries: 3)
[INFO]    LRE: About to process submodels locally
[INFO]    ==================================
[INFO]    Local Reconstruction submodels
[INFO]    ==================================
[INFO]    running /usr/bin/env python2 /code/SuperBuild/src/opensfm/bin/opensfm detect_features "/datasets/code/opensfm/submodels/opensfm"
Traceback (most recent call last):
  File "/code/SuperBuild/src/opensfm/bin/opensfm", line 34, in <module>
    command.run(args)
  File "/code/SuperBuild/src/opensfm/opensfm/commands/detect_features.py", line 33, in run
    with open(data.profile_log(), 'a') as fout:
IOError: [Errno 2] No such file or directory: '/datasets/code/opensfm/submodels/opensfm/profile.log'
[WARNING] LRE: submodels failed with: Child returned 1
[INFO]    LRE: Re-queueing submodels (retries: 3)
[INFO]    LRE: About to process submodels locally
[INFO]    ==================================
[INFO]    Local Reconstruction submodels
[INFO]    ==================================
[INFO]    running /usr/bin/env python2 /code/SuperBuild/src/opensfm/bin/opensfm detect_features "/datasets/code/opensfm/submodels/opensfm"
Traceback (most recent call last):
  File "/code/SuperBuild/src/opensfm/bin/opensfm", line 34, in <module>
    command.run(args)
  File "/code/SuperBuild/src/opensfm/opensfm/commands/detect_features.py", line 33, in run
    with open(data.profile_log(), 'a') as fout:
IOError: [Errno 2] No such file or directory: '/datasets/code/opensfm/submodels/opensfm/profile.log'
[WARNING] LRE: submodels failed with: Child returned 1
[INFO]    LRE: Re-queueing submodels (retries: 3)
[INFO]    LRE: About to process submodels locally
[INFO]    ==================================
[INFO]    Local Reconstruction submodels
[INFO]    ==================================
[INFO]    running /usr/bin/env python2 /code/SuperBuild/src/opensfm/bin/opensfm detect_features "/datasets/code/opensfm/submodels/opensfm"
Traceback (most recent call last):
  File "/code/SuperBuild/src/opensfm/bin/opensfm", line 34, in <module>
    command.run(args)
  File "/code/SuperBuild/src/opensfm/opensfm/commands/detect_features.py", line 33, in run
    with open(data.profile_log(), 'a') as fout:
IOError: [Errno 2] No such file or directory: '/datasets/code/opensfm/submodels/opensfm/profile.log'
[WARNING] LRE: submodels failed with: Child returned 1
[INFO]    LRE: Re-queueing submodels (retries: 3)
[INFO]    LRE: About to process submodels locally
[INFO]    ==================================
[INFO]    Local Reconstruction submodels
[INFO]    ==================================
[INFO]    running /usr/bin/env python2 /code/SuperBuild/src/opensfm/bin/opensfm detect_features "/datasets/code/opensfm/submodels/opensfm"
Traceback (most recent call last):
  File "/code/SuperBuild/src/opensfm/bin/opensfm", line 34, in <module>
    command.run(args)
  File "/code/SuperBuild/src/opensfm/opensfm/commands/detect_features.py", line 33, in run
    with open(data.profile_log(), 'a') as fout:
IOError: [Errno 2] No such file or directory: '/datasets/code/opensfm/submodels/opensfm/profile.log'
[WARNING] LRE: submodels failed with: Child returned 1
[INFO]    LRE: Re-queueing submodels (retries: 4)
[INFO]    LRE: About to process submodels locally
[INFO]    ==================================
[INFO]    Local Reconstruction submodels
[INFO]    ==================================
[INFO]    running /usr/bin/env python2 /code/SuperBuild/src/opensfm/bin/opensfm detect_features "/datasets/code/opensfm/submodels/opensfm"
[INFO]    LRE: Upload of submodels at [100%]
[WARNING] LRE: submodels failed with: Not enough images (1 files uploaded)
[INFO]    LRE: Re-queueing submodels (retries: 2)
[INFO]    LRE: About to process submodels remotely
[INFO]    LRE: Waiting 27 seconds before processing submodels
Traceback (most recent call last):
  File "/code/SuperBuild/src/opensfm/bin/opensfm", line 34, in <module>
    command.run(args)
  File "/code/SuperBuild/src/opensfm/opensfm/commands/detect_features.py", line 33, in run
    with open(data.profile_log(), 'a') as fout:
IOError: [Errno 2] No such file or directory: '/datasets/code/opensfm/submodels/opensfm/profile.log'
[WARNING] LRE: submodels failed with: Child returned 1
[INFO]    LRE: Re-queueing submodels (retries: 4)
[INFO]    LRE: About to process submodels locally
[INFO]    ==================================
[INFO]    Local Reconstruction submodels
[INFO]    ==================================
[INFO]    running /usr/bin/env python2 /code/SuperBuild/src/opensfm/bin/opensfm detect_features "/datasets/code/opensfm/submodels/opensfm"
Traceback (most recent call last):
  File "/code/SuperBuild/src/opensfm/bin/opensfm", line 34, in <module>
    command.run(args)
  File "/code/SuperBuild/src/opensfm/opensfm/commands/detect_features.py", line 33, in run
    with open(data.profile_log(), 'a') as fout:
IOError: [Errno 2] No such file or directory: '/datasets/code/opensfm/submodels/opensfm/profile.log'
[WARNING] LRE: submodels failed with: Child returned 1
[INFO]    LRE: Re-queueing submodels (retries: 4)
[INFO]    LRE: About to process submodels locally
[INFO]    ==================================
[INFO]    Local Reconstruction submodels
[INFO]    ==================================
[INFO]    running /usr/bin/env python2 /code/SuperBuild/src/opensfm/bin/opensfm detect_features "/datasets/code/opensfm/submodels/opensfm"
Traceback (most recent call last):
  File "/code/SuperBuild/src/opensfm/bin/opensfm", line 34, in <module>
    command.run(args)
  File "/code/SuperBuild/src/opensfm/opensfm/commands/detect_features.py", line 33, in run
    with open(data.profile_log(), 'a') as fout:
IOError: [Errno 2] No such file or directory: '/datasets/code/opensfm/submodels/opensfm/profile.log'
[WARNING] LRE: submodels failed with: Child returned 1
[INFO]    LRE: Re-queueing submodels (retries: 4)
[INFO]    LRE: About to process submodels locally
[INFO]    ==================================
[INFO]    Local Reconstruction submodels
[INFO]    ==================================
[INFO]    running /usr/bin/env python2 /code/SuperBuild/src/opensfm/bin/opensfm detect_features "/datasets/code/opensfm/submodels/opensfm"
Traceback (most recent call last):
  File "/code/SuperBuild/src/opensfm/bin/opensfm", line 34, in <module>
    command.run(args)
  File "/code/SuperBuild/src/opensfm/opensfm/commands/detect_features.py", line 33, in run
    with open(data.profile_log(), 'a') as fout:
IOError: [Errno 2] No such file or directory: '/datasets/code/opensfm/submodels/opensfm/profile.log'
[WARNING] LRE: submodels failed with: Child returned 1
[INFO]    LRE: Re-queueing submodels (retries: 5)
[INFO]    LRE: About to process submodels locally
[INFO]    ==================================
[INFO]    Local Reconstruction submodels
[INFO]    ==================================
[INFO]    running /usr/bin/env python2 /code/SuperBuild/src/opensfm/bin/opensfm detect_features "/datasets/code/opensfm/submodels/opensfm"
Traceback (most recent call last):
  File "/code/SuperBuild/src/opensfm/bin/opensfm", line 34, in <module>
    command.run(args)
  File "/code/SuperBuild/src/opensfm/opensfm/commands/detect_features.py", line 33, in run
    with open(data.profile_log(), 'a') as fout:
IOError: [Errno 2] No such file or directory: '/datasets/code/opensfm/submodels/opensfm/profile.log'
[WARNING] LRE: submodels failed with: Child returned 1
[INFO]    LRE: Re-queueing submodels (retries: 3)
[INFO]    LRE: About to process submodels locally
[INFO]    ==================================
[INFO]    Local Reconstruction submodels
[INFO]    ==================================
[INFO]    running /usr/bin/env python2 /code/SuperBuild/src/opensfm/bin/opensfm detect_features "/datasets/code/opensfm/submodels/opensfm"
Traceback (most recent call last):
  File "/code/SuperBuild/src/opensfm/bin/opensfm", line 34, in <module>
    command.run(args)
  File "/code/SuperBuild/src/opensfm/opensfm/commands/detect_features.py", line 33, in run
    with open(data.profile_log(), 'a') as fout:
IOError: [Errno 2] No such file or directory: '/datasets/code/opensfm/submodels/opensfm/profile.log'
[WARNING] LRE: submodels failed with: Child returned 1
[INFO]    LRE: Re-queueing submodels (retries: 5)
[INFO]    LRE: About to process submodels locally
[INFO]    ==================================
[INFO]    Local Reconstruction submodels
[INFO]    ==================================
[INFO]    running /usr/bin/env python2 /code/SuperBuild/src/opensfm/bin/opensfm detect_features "/datasets/code/opensfm/submodels/opensfm"
Traceback (most recent call last):
  File "/code/SuperBuild/src/opensfm/bin/opensfm", line 34, in <module>
    command.run(args)
  File "/code/SuperBuild/src/opensfm/opensfm/commands/detect_features.py", line 33, in run
    with open(data.profile_log(), 'a') as fout:
IOError: [Errno 2] No such file or directory: '/datasets/code/opensfm/submodels/opensfm/profile.log'
[WARNING] LRE: submodels failed with: Child returned 1
[INFO]    LRE: Re-queueing submodels (retries: 5)
[INFO]    LRE: About to process submodels locally
[INFO]    ==================================
[INFO]    Local Reconstruction submodels
[INFO]    ==================================
[INFO]    running /usr/bin/env python2 /code/SuperBuild/src/opensfm/bin/opensfm detect_features "/datasets/code/opensfm/submodels/opensfm"
Traceback (most recent call last):
  File "/code/SuperBuild/src/opensfm/bin/opensfm", line 34, in <module>
    command.run(args)
  File "/code/SuperBuild/src/opensfm/opensfm/commands/detect_features.py", line 33, in run
    with open(data.profile_log(), 'a') as fout:
IOError: [Errno 2] No such file or directory: '/datasets/code/opensfm/submodels/opensfm/profile.log'
[WARNING] LRE: submodels failed with: Child returned 1
[INFO]    LRE: Re-queueing submodels (retries: 5)
[INFO]    LRE: About to process submodels locally
[INFO]    ==================================
[INFO]    Local Reconstruction submodels
[INFO]    ==================================
[INFO]    running /usr/bin/env python2 /code/SuperBuild/src/opensfm/bin/opensfm detect_features "/datasets/code/opensfm/submodels/opensfm"
Traceback (most recent call last):
  File "/code/SuperBuild/src/opensfm/bin/opensfm", line 34, in <module>
    command.run(args)
  File "/code/SuperBuild/src/opensfm/opensfm/commands/detect_features.py", line 33, in run
    with open(data.profile_log(), 'a') as fout:
IOError: [Errno 2] No such file or directory: '/datasets/code/opensfm/submodels/opensfm/profile.log'
[WARNING] LRE: submodels failed with: Child returned 1
[INFO]    LRE: Upload of submodels at [100%]
[WARNING] LRE: submodels failed with: Not enough images (1 files uploaded)
[INFO]    LRE: Re-queueing submodels (retries: 4)
[INFO]    LRE: No remote tasks left to cleanup
Traceback (most recent call last):
  File "/code/run.py", line 62, in <module>
    app.execute()
  File "/code/stages/odm_app.py", line 95, in execute
    self.first_stage.run()
  File "/code/opendm/types.py", line 350, in run
    self.next_stage.run(outputs)
  File "/code/opendm/types.py", line 331, in run
    self.process(self.args, outputs)
  File "/code/stages/splitmerge.py", line 103, in process
    lre.run_reconstruction()
  File "/code/opendm/remote.py", line 55, in run_reconstruction
    self.run(ReconstructionTask)
  File "/code/opendm/remote.py", line 252, in run
    raise nonloc.error
Exception: Child returned 1

Why am I getting this error? Does it have something to do with machine 2?
Also, how can we find that how many images have gone to 1st and 2nd node odm instances for processing, even though I know that the --split parameter is set at 200 ?

Thanks,
Ayush

#2

Gentle reminder to community members, please respond, so that this issue can be resolved.

#3

Its a shot in the dark ,
but can you try with --crop as default value ?

#4

Hi @rumenchooo,

So I executed this command from machine 2, just like last time where the images are present.

docker run -ti --rm -v /$(pwd)/hinduja:/datasets/code opendronemap/odm --project-path /datasets --verbose --resize-to -1 --split 200 --split-overlap 70 --sm-cluster http://custer-odm-ip:3001 --dsm --dtm --depthmap-resolution 1000 --dem-resolution 2 --orthophoto-resolution 2 --ignore-gsd

I tried what you mentioned, and this time even more weird issue. Hear me out.

So, last time three sub-models were created and out of them, submodel000 was not executed, and rest 001 and 002 were executing on both the nodes(machine_1 and machine_2). But this time, only machine_1 is the one executing the task, machine_2 is not doing anything as of now and it errored at the end.

Initial
image

and machine_1 is processing submodel that contains only 119 images, just like time.

Can you look at the logs and tell me if something is wrong.(@smathermather-cm any input helpful).

azureuser@testodm:~/Downloads$ docker run -ti --rm -v /$(pwd)/hinduja:/datasets/code opendronemap/odm --project-path /datasets --verbose --resize-to -1 --split 200 --split-overlap 70 --sm-cluster http://xx.xx.xx.xx:3001 --dsm --dtm --depthmap-resolution 1000 --dem-resolution 2 --orthophoto-resolution 2 --ignore-gsd
Unable to find image 'opendronemap/odm:latest' locally
latest: Pulling from opendronemap/odm
f08d8e2a3ba1: Already exists 
3baa9cb2483b: Already exists 
94e5ff4c0b15: Already exists 
1860925334f9: Already exists 
c9294bb993aa: Already exists 
Digest: sha256:fff9e77230131007b8e9f561b90e538719c8a43195faac24b1267f3b84468543
Status: Downloaded newer image for opendronemap/odm:latest
[INFO]    DTM is turned on, automatically turning on point cloud classification
[INFO]    Initializing ODM - Wed Sep 30 12:16:35  2020
[INFO]    ==============
[INFO]    build_overviews: False
[INFO]    camera_lens: auto
[INFO]    cameras: {}
[INFO]    crop: 3
[INFO]    debug: False
[INFO]    dem_decimation: 1
[INFO]    dem_euclidean_map: False
[INFO]    dem_gapfill_steps: 3
[INFO]    dem_resolution: 2.0
[INFO]    depthmap_resolution: 1000.0
[INFO]    dsm: True
[INFO]    dtm: True
[INFO]    end_with: odm_report
[INFO]    fast_orthophoto: False
[INFO]    feature_quality: high
[INFO]    feature_type: sift
[INFO]    force_gps: False
[INFO]    gcp: None
[INFO]    geo: None
[INFO]    gps_accuracy: 15
[INFO]    ignore_gsd: True
[INFO]    matcher_distance: 0
[INFO]    matcher_neighbors: 8
[INFO]    max_concurrency: 16
[INFO]    merge: all
[INFO]    mesh_octree_depth: 10
[INFO]    mesh_point_weight: 4
[INFO]    mesh_samples: 1.0
[INFO]    mesh_size: 200000
[INFO]    min_num_features: 8000
[INFO]    mve_confidence: 0.6
[INFO]    name: code
[INFO]    opensfm_depthmap_method: PATCH_MATCH
[INFO]    opensfm_depthmap_min_consistent_views: 3
[INFO]    opensfm_depthmap_min_patch_sd: 1
[INFO]    optimize_disk_space: False
[INFO]    orthophoto_compression: DEFLATE
[INFO]    orthophoto_cutline: False
[INFO]    orthophoto_no_tiled: False
[INFO]    orthophoto_png: False
[INFO]    orthophoto_resolution: 2.0
[INFO]    pc_classify: True
[INFO]    pc_csv: False
[INFO]    pc_ept: False
[INFO]    pc_filter: 2.5
[INFO]    pc_las: False
[INFO]    pc_rectify: False
[INFO]    pc_sample: 0
[INFO]    project_path: /datasets
[INFO]    radiometric_calibration: none
[INFO]    rerun: None
[INFO]    rerun_all: False
[INFO]    rerun_from: None
[INFO]    resize_to: -1
[INFO]    skip_3dmodel: False
[INFO]    sm_cluster: True
[INFO]    smrf_scalar: 1.25
[INFO]    smrf_slope: 0.15
[INFO]    smrf_threshold: 0.5
[INFO]    smrf_window: 18.0
[INFO]    split: 200
[INFO]    split_multitracks: False
[INFO]    split_overlap: 70.0
[INFO]    texturing_data_term: gmi
[INFO]    texturing_keep_unseen_faces: False
[INFO]    texturing_nadir_weight: 16
[INFO]    texturing_outlier_removal_type: gauss_clamping
[INFO]    texturing_skip_global_seam_leveling: False
[INFO]    texturing_skip_hole_filling: False
[INFO]    texturing_skip_local_seam_leveling: False
[INFO]    texturing_skip_visibility_test: False
[INFO]    texturing_tone_mapping: none
[INFO]    tiles: False
[INFO]    time: False
[INFO]    use_3dmesh: False
[INFO]    use_exif: False
[INFO]    use_fixed_camera_params: False
[INFO]    use_hybrid_bundle_adjustment: False
[INFO]    use_opensfm_dense: False
[INFO]    verbose: True
[INFO]    ==============
[INFO]    Running dataset stage
[INFO]    Loading dataset from: /datasets/code/images
[INFO]    Loading 452 images
[INFO]    Wrote images database: /datasets/code/images.json
[INFO]    Found 452 usable images
[INFO]    Parsing SRS header: WGS84 UTM 44N
[INFO]    Finished dataset stage
[INFO]    Running split stage
[INFO]    Setting max-concurrency to 15 to better handle remote splits
[INFO]    Large dataset detected (452 photos) and split set at 200. Preparing split merge.
[INFO]    Writing exif overrides
[WARNING] Legacy option --resize-to (this might be removed in a future version). Use --feature-quality instead.
[INFO]    Altitude data detected, enabling it for GPS alignment
[INFO]    ['use_exif_size: no', 'flann_algorithm: KDTREE', 'feature_process_size: -1', 'feature_min_frames: 8000', 'processes: 15', 'matching_gps_neighbors: 8', 'matching_gps_distance: 0', 'depthmap_method: PATCH_MATCH', 'depthmap_resolution: 1000.0', 'depthmap_min_patch_sd: 1', 'depthmap_min_consistent_views: 3', 'optimize_camera_parameters: yes', 'undistorted_image_format: tif', 'bundle_outlier_filtering_type: AUTO', 'align_orientation_prior: vertical', 'triangulation_type: ROBUST', 'bundle_common_position_constraints: no', 'feature_type: SIFT', 'use_altitude_tag: yes', 'align_method: auto', 'local_bundle_radius: 0', 'submodels_relpath: ../submodels/opensfm', 'submodel_relpath_template: ../submodels/submodel_%04d/opensfm', 'submodel_images_relpath_template: ../submodels/submodel_%04d/images', 'submodel_size: 200', 'submodel_overlap: 70.0']
[INFO]    running /usr/bin/env python3 /code/SuperBuild/src/opensfm/bin/opensfm extract_metadata "/datasets/code/opensfm"
2020-09-30 12:16:49,636 INFO: Extracting EXIF for DJI_0260.JPG
...
2020-09-30 12:17:35,786 INFO: Extracting EXIF for DJI_0165.JPG
2020-09-30 12:17:35,879 INFO: Extracting EXIF for DJI_0076.JPG
[INFO]    running /usr/bin/env python3 /code/SuperBuild/src/opensfm/bin/opensfm create_submodels "/datasets/code/opensfm"
[INFO]    Dataset has been split into 3 submodels. Reconstructing each submodel...
[INFO]    LRE: Initializing using cluster node 20.184.5.64:3001
[INFO]    LRE: Node is online and running odm version 2.0.0
[INFO]    LRE: Adding to queue /datasets/code/submodels/submodel_0000
[INFO]    LRE: Adding to queue /datasets/code/submodels/submodel_0001
[INFO]    LRE: Adding to queue /datasets/code/submodels/submodel_0002
[INFO]    LRE: About to process submodel_0000 locally
[INFO]    ==================================
[INFO]    LRE: About to process submodel_0001 remotely
[INFO]    Local Reconstruction submodel_0000
[INFO]    ==================================
[INFO]    running /usr/bin/env python3 /code/SuperBuild/src/opensfm/bin/opensfm detect_features "/datasets/code/submodels/submodel_0000/opensfm"
[INFO]    LRE: Upload of submodel_0001 at [0%]
2020-09-30 12:17:40,374 INFO: Extracting ROOT_SIFT features for image DJI_0388.JPG
2020-09-30 12:17:40,401 INFO: Extracting ROOT_
..... sif logs
2020-09-30 12:17:42,111 DEBUG: Computing sift with threshold 0.1
[INFO]    LRE: Upload of submodel_0001 at [34%]
[INFO]    LRE: Upload of submodel_0001 at [47%]
[INFO]    LRE: Upload of submodel_0001 at [60%]
[INFO]    LRE: Upload of submodel_0001 at [74%]
[INFO]    LRE: Upload of submodel_0001 at [86%]
[INFO]    LRE: Upload of submodel_0001 at [100%]
[INFO]    LRE: About to process submodel_0002 remotely
[INFO]    LRE: Upload of submodel_0002 at [0%]
2020-09-30 12:17:54,726 DEBUG: Found 10305 points in 12.826160192489624s
2020-09-30 12:17:54,726 DEBUG: done
.....(exif extraction logs[truncated because of letter limitation 32000])
2020-09-30 12:19:02,464 DEBUG: No segmentation for DJI_0101.JPG, no features masked.
/usr/local/lib/python3.6/dist-packages/joblib/externals/loky/process_executor.py:706: UserWarning: A worker stopped while some jobs were given to the executor. This can be caused by a too short worker timeout or by a memory leak.
  "timeout or by a memory leak.", UserWarning
2020-09-30 12:19:04,878 INFO: Extracting ROOT_SIFT features for image DJI_0353.JPG
2020-09-30 12:19:06,421 DEBUG: Computing sift with threshold 0.1
Traceback (most recent call last):
  File "/code/SuperBuild/src/opensfm/bin/opensfm", line 34, in <module>
    command.run(args)
  File "/code/SuperBuild/src/opensfm/opensfm/commands/detect_features.py", line 31, in run
    parallel_map(detect, arguments, processes, 1)
  File "/code/SuperBuild/src/opensfm/opensfm/context.py", line 41, in parallel_map
    return Parallel(batch_size=batch_size)(delayed(func)(arg) for arg in args)
  File "/usr/local/lib/python3.6/dist-packages/joblib/parallel.py", line 934, in __call__
    self.retrieve()
  File "/usr/local/lib/python3.6/dist-packages/joblib/parallel.py", line 833, in retrieve
    self._output.extend(job.get(timeout=self.timeout))
  File "/usr/local/lib/python3.6/dist-packages/joblib/_parallel_backends.py", line 521, in wrap_future_result
    return future.result(timeout=timeout)
  File "/usr/lib/python3.6/concurrent/futures/_base.py", line 432, in result
    return self.__get_result()
  File "/usr/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
joblib.externals.loky.process_executor.TerminatedWorkerError: A worker process managed by the executor was unexpectedly terminated. This could be caused by a segmentation fault while calling the function or by an excessive memory usage causing the Operating System to kill the worker. The exit codes of the workers are {SIGKILL(-9), SIGKILL(-9)}
[WARNING] LRE: submodel_0000 failed with: Child returned 1
[INFO]    LRE: Re-queueing submodel_0000 (retries: 1)
[INFO]    LRE: About to process submodel_0000 locally
[INFO]    ==================================
[INFO]    Local Reconstruction submodel_0000
[INFO]    ==================================
[WARNING] Detect features already done: /datasets/code/submodels/submodel_0000/opensfm/features exists
[INFO]    running /usr/bin/env python3 /code/SuperBuild/src/opensfm/bin/opensfm match_features "/datasets/code/submodels/submodel_0000/opensfm"
2020-09-30 12:19:11,674 INFO: Matching 922 image pairs
2020-09-30 12:19:11,689 INFO: Computing pair matching with 15 processes
[WARNING] LRE: submodel_0002 failed with: (c94d63d6-5330-444c-b10f-fd4fdd049413) failed with task output: self.process(self.args, outputs)
File "/code/stages/run_opensfm.py", line 30, in process
octx.feature_matching(self.rerun())
File "/code/opendm/osfm.py", line 272, in feature_matching
self.run('detect_features')
File "/code/opendm/osfm.py", line 22, in run
(context.opensfm_path, command, self.opensfm_project_path))
File "/code/opendm/system.py", line 79, in run
raise Exception("Child returned {}".format(retcode))
Exception: Child returned 1
Full log saved at /datasets/code/submodels/submodel_0002/error.log
[INFO]    LRE: Cleaning up remote task (c94d63d6-5330-444c-b10f-fd4fdd049413)... OK
joblib.externals.loky.process_executor._RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/joblib/externals/loky/process_executor.py", line 418, in _process_worker
    r = call_item()
  File "/usr/local/lib/python3.6/dist-packages/joblib/externals/loky/process_executor.py", line 272, in __call__
    return self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.6/dist-packages/joblib/_parallel_backends.py", line 567, in __call__
    return self.func(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/joblib/parallel.py", line 225, in __call__
    for func, args, kwargs in self.items]
  File "/usr/local/lib/python3.6/dist-packages/joblib/parallel.py", line 225, in <listcomp>
    for func, args, kwargs in self.items]
  File "/code/SuperBuild/src/opensfm/opensfm/matching.py", line 145, in match_unwrap_args
    p1, f1, _ = feature_loader.instance.load_points_features_colors(ctx.data, im1)
  File "/code/SuperBuild/src/opensfm/opensfm/feature_loading.py", line 64, in load_points_features_colors
    points, features, colors = self._load_features_nocache(data, image)
  File "/code/SuperBuild/src/opensfm/opensfm/feature_loading.py", line 100, in _load_features_nocache
    points, features, colors = data.load_features(image)
  File "/code/SuperBuild/src/opensfm/opensfm/dataset.py", line 317, in load_features
    return features.load_features(self._feature_file(image), self.config)
  File "/code/SuperBuild/src/opensfm/opensfm/features.py", line 327, in load_features
    s = np.load(filepath)
  File "/usr/local/lib/python3.6/dist-packages/numpy/lib/npyio.py", line 416, in load
    fid = stack.enter_context(open(os_fspath(file), "rb"))
FileNotFoundError: [Errno 2] No such file or directory: '/datasets/code/submodels/submodel_0000/opensfm/features/DJI_0160.JPG.features.npz'
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/code/SuperBuild/src/opensfm/bin/opensfm", line 34, in <module>
    command.run(args)
  File "/code/SuperBuild/src/opensfm/opensfm/commands/match_features.py", line 29, in run
    pairs_matches, preport = matching.match_images(data, images, images)
  File "/code/SuperBuild/src/opensfm/opensfm/matching.py", line 43, in match_images
    return match_images_with_pairs(data, exifs, ref_images, pairs), preport
  File "/code/SuperBuild/src/opensfm/opensfm/matching.py", line 67, in match_images_with_pairs
    matches = context.parallel_map(match_unwrap_args, args, processes, jobs_per_process)
  File "/code/SuperBuild/src/opensfm/opensfm/context.py", line 41, in parallel_map
    return Parallel(batch_size=batch_size)(delayed(func)(arg) for arg in args)
  File "/usr/local/lib/python3.6/dist-packages/joblib/parallel.py", line 934, in __call__
    self.retrieve()
  File "/usr/local/lib/python3.6/dist-packages/joblib/parallel.py", line 833, in retrieve
    self._output.extend(job.get(timeout=self.timeout))
  File "/usr/local/lib/python3.6/dist-packages/joblib/_parallel_backends.py", line 521, in wrap_future_result
    return future.result(timeout=timeout)
  File "/usr/lib/python3.6/concurrent/futures/_base.py", line 432, in result
    return self.__get_result()
  File "/usr/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
FileNotFoundError: [Errno 2] No such file or directory: '/datasets/code/submodels/submodel_0000/opensfm/features/DJI_0160.JPG.features.npz'
[WARNING] LRE: submodel_0000 failed with: Child returned 1
[INFO]    LRE: Re-queueing submodel_0000 (retries: 2)
[INFO]    LRE: submodel_0001 (cdabe829-b5c4-4184-95ee-4d6e9e95989e) is still running
[INFO]    LRE: submodel_0001 (cdabe829-b5c4-4184-95ee-4d6e9e95989e) is still running
[INFO]    LRE: Downloading assets for submodel_0001
[INFO]    LRE: Download of submodel_0001 at [0%]
[INFO]    LRE: Download of submodel_0001 at [23%]
[INFO]    LRE: Download of submodel_0001 at [47%]
[INFO]    LRE: Download of submodel_0001 at [70%]
[INFO]    LRE: Download of submodel_0001 at [94%]
[INFO]    LRE: Download of submodel_0001 at [100%]
[INFO]    LRE: Downloaded and extracted assets for submodel_0001
[INFO]    LRE: submodel_0001 finished successfully
[INFO]    LRE: Cleaning up remote task (cdabe829-b5c4-4184-95ee-4d6e9e95989e)... OK
[INFO]    LRE: No remote tasks left to cleanup
Traceback (most recent call last):
  File "/code/run.py", line 68, in <module>
    app.execute()
  File "/code/stages/odm_app.py", line 95, in execute
    self.first_stage.run()
  File "/code/opendm/types.py", line 356, in run
    self.next_stage.run(outputs)
  File "/code/opendm/types.py", line 337, in run
    self.process(self.args, outputs)
  File "/code/stages/splitmerge.py", line 103, in process
    lre.run_reconstruction()
  File "/code/opendm/remote.py", line 55, in run_reconstruction
    self.run(ReconstructionTask)
  File "/code/opendm/remote.py", line 252, in run
    raise nonloc.error
pyodm.exceptions.TaskFailedError: (c94d63d6-5330-444c-b10f-fd4fdd049413) failed with task output: self.process(self.args, outputs)
File "/code/stages/run_opensfm.py", line 30, in process
octx.feature_matching(self.rerun())
File "/code/opendm/osfm.py", line 272, in feature_matching
self.run('detect_features')
File "/code/opendm/osfm.py", line 22, in run
(context.opensfm_path, command, self.opensfm_project_path))
File "/code/opendm/system.py", line 79, in run
raise Exception("Child returned {}".format(retcode))
Exception: Child returned 1
Full log saved at /datasets/code/submodels/submodel_0002/error.log
1 Like