ClusterODM "[WARNING] LRE: submodel_0001 failed with"

Its not working somehow.
Sub models are only distributed to local node even that ClusterODM has 2 online nodes.
And it fails with :slight_smile: [WARNING] LRE: submodel_0002 failed with: [{“name”:“verbose”,“error”:“Cannot convert 9a3aa963-a5f9-4742-a589-907945610be5 to boolean”}]

console:

[INFO]    Initializing OpenDroneMap app - Mon Oct 28 08:24:10  2019
[INFO]    ==============
[INFO]    build_overviews: False
[INFO]    camera_lens: auto
[INFO]    cameras: {}
[INFO]    crop: 3
[INFO]    debug: True
[INFO]    dem_decimation: 1
[INFO]    dem_euclidean_map: False
[INFO]    dem_gapfill_steps: 3
[INFO]    dem_resolution: 2.0
[INFO]    depthmap_resolution: 1000.0
[INFO]    dsm: True
[INFO]    dtm: True
[INFO]    end_with: odm_orthophoto
[INFO]    fast_orthophoto: False
[INFO]    force_gps: False
[INFO]    gcp: None
[INFO]    ignore_gsd: True
[INFO]    matcher_distance: 0
[INFO]    matcher_neighbors: 8
[INFO]    max_concurrency: 4
[INFO]    merge: all
[INFO]    mesh_octree_depth: 12
[INFO]    mesh_point_weight: 4
[INFO]    mesh_samples: 1.0
[INFO]    mesh_size: 300000
[INFO]    min_num_features: 14000
[INFO]    mve_confidence: 0.6
[INFO]    name: 9a3aa963-a5f9-4742-a589-907945610be5
[INFO]    opensfm_depthmap_method: PATCH_MATCH
[INFO]    opensfm_depthmap_min_consistent_views: 3
[INFO]    opensfm_depthmap_min_patch_sd: 1
[INFO]    orthophoto_compression: DEFLATE
[INFO]    orthophoto_cutline: False
[INFO]    orthophoto_no_tiled: False
[INFO]    orthophoto_resolution: 5
[INFO]    pc_classify: True
[INFO]    pc_csv: False
[INFO]    pc_ept: False
[INFO]    pc_filter: 2.5
[INFO]    pc_las: False
[INFO]    project_path: /www/data
[INFO]    rerun: None
[INFO]    rerun_all: False
[INFO]    rerun_from: None
[INFO]    resize_to: -1
[INFO]    skip_3dmodel: False
[INFO]    sm_cluster: True
[INFO]    smrf_scalar: 1.25
[INFO]    smrf_slope: 0.15
[INFO]    smrf_threshold: 0.5
[INFO]    smrf_window: 18.0
[INFO]    split: 100
[INFO]    split_overlap: 80.0
[INFO]    texturing_data_term: gmi
[INFO]    texturing_keep_unseen_faces: False
[INFO]    texturing_nadir_weight: 16
[INFO]    texturing_outlier_removal_type: gauss_clamping
[INFO]    texturing_skip_global_seam_leveling: False
[INFO]    texturing_skip_hole_filling: False
[INFO]    texturing_skip_local_seam_leveling: False
[INFO]    texturing_skip_visibility_test: False
[INFO]    texturing_tone_mapping: none
[INFO]    time: False
[INFO]    use_3dmesh: True
[INFO]    use_exif: False
[INFO]    use_fixed_camera_params: False
[INFO]    use_hybrid_bundle_adjustment: False
[INFO]    use_opensfm_dense: False
[INFO]    verbose: True
[INFO]    ==============
[INFO]    Running dataset stage
[INFO]    Loading dataset from: /www/data/9a3aa963-a5f9-4742-a589-907945610be5/images
[DEBUG]   /www/data/9a3aa963-a5f9-4742-a589-907945610be5/images
[INFO]    Loading 493 images


[INFO]    Wrote images database: /www/data/9a3aa963-a5f9-4742-a589-907945610be5/images.json
[INFO]    Found 493 usable images
[INFO]    Parsing SRS header: WGS84 UTM 51N
[INFO]    Finished dataset stage
[INFO]    Running split stage
[INFO]    Setting max-concurrency to 3 to better handle remote splits
[INFO]    Large dataset detected (493 photos) and split set at 100. Preparing split merge.
[INFO]    Altitude data detected, enabling it for GPS alignment
[INFO]    ['use_exif_size: no', 'feature_process_size: -1', 'feature_min_frames: 14000', 'processes: 3', 'matching_gps_neighbors: 8', 'matching_gps_distance: 0', 'depthmap_method: PATCH_MATCH', 'depthmap_resolution: 1000.0', 'depthmap_min_patch_sd: 1', 'depthmap_min_consistent_views: 3', 'optimize_camera_parameters: yes', 'undistorted_image_format: png', 'bundle_outlier_filtering_type: AUTO', 'align_orientation_prior: vertical', 'use_altitude_tag: yes', 'align_method: auto', 'local_bundle_radius: 0', 'submodels_relpath: ../submodels/opensfm', 'submodel_relpath_template: ../submodels/submodel_%04d/opensfm', 'submodel_images_relpath_template: ../submodels/submodel_%04d/images', 'submodel_size: 100', 'submodel_overlap: 80.0']
[INFO]    running /code/SuperBuild/src/opensfm/bin/opensfm extract_metadata "/www/data/9a3aa963-a5f9-4742-a589-907945610be5/opensfm"


[INFO]    running /code/SuperBuild/src/opensfm/bin/opensfm create_submodels "/www/data/9a3aa963-a5f9-4742-a589-907945610be5/opensfm"
[INFO]    Dataset has been split into 5 submodels. Reconstructing each submodel...
[INFO]    LRE: Initializing using cluster node 127.0.0.1:3001
[INFO]    LRE: Node is online and running odm version 0.9.1
[INFO]    LRE: Adding to queue /www/data/9a3aa963-a5f9-4742-a589-907945610be5/submodels/submodel_0000
[INFO]    LRE: Adding to queue /www/data/9a3aa963-a5f9-4742-a589-907945610be5/submodels/submodel_0001
[INFO]    LRE: Adding to queue /www/data/9a3aa963-a5f9-4742-a589-907945610be5/submodels/submodel_0002
[INFO]    LRE: Adding to queue /www/data/9a3aa963-a5f9-4742-a589-907945610be5/submodels/submodel_0003
[INFO]    LRE: Adding to queue /www/data/9a3aa963-a5f9-4742-a589-907945610be5/submodels/submodel_0004
[INFO]    LRE: About to process submodel_0000 locally
[INFO]    ==================================
[INFO]    LRE: About to process submodel_0001 remotely
[INFO]    Local Reconstruction submodel_0000
[INFO]    ==================================
[INFO]    running /code/SuperBuild/src/opensfm/bin/opensfm detect_features "/www/data/9a3aa963-a5f9-4742-a589-907945610be5/submodels/submodel_0000/opensfm"
[WARNING] LRE: submodel_0001 failed with: [{"name":"verbose","error":"Cannot convert 9a3aa963-a5f9-4742-a589-907945610be5 to boolean"}]
[INFO]    LRE: Re-queueing submodel_0001 (retries: 1)
[INFO]    LRE: About to process submodel_0002 remotely
[WARNING] LRE: submodel_0002 failed with: [{"name":"verbose","error":"Cannot convert 9a3aa963-a5f9-4742-a589-907945610be5 to boolean"}]
[INFO]    LRE: Re-queueing submodel_0002 (retries: 1)
[INFO]    LRE: About to process submodel_0003 remotely
[WARNING] LRE: submodel_0003 failed with: [{"name":"verbose","error":"Cannot convert 9a3aa963-a5f9-4742-a589-907945610be5 to boolean"}]
[INFO]    LRE: Re-queueing submodel_0003 (retries: 1)
[INFO]    LRE: About to process submodel_0004 remotely
[WARNING] LRE: submodel_0004 failed with: [{"name":"verbose","error":"Cannot convert 9a3aa963-a5f9-4742-a589-907945610be5 to boolean"}]
[INFO]    LRE: Re-queueing submodel_0004 (retries: 1)
[INFO]    LRE: About to process submodel_0001 remotely
[INFO]    LRE: Waiting 10 seconds before processing submodel_0001
2019-10-28 08:24:22,718 INFO: Extracting ROOT_HAHOG features for image 20181221_Miaoli_Shitan_ms2_117.JPG
2019-10-28 08:24:23,054 INFO: Extracting ROOT_HAHOG features for image 20181221_Miaoli_Shitan_ms1_117.JPG
2019-10-28 08:24:23,414 INFO: Extracting ROOT_HAHOG features for image 20181221_Miaoli_Shitan_ms2_055.JPG
2019-10-28 08:24:31,347 DEBUG: Found 18493 points in 8.16880202293s
2019-10-28 08:24:31,349 DEBUG: No segmentation for 20181221_Miaoli_Shitan_ms1_117.JPG, no features masked.
2019-10-28 08:24:31,611 INFO: Extracting ROOT_HAHOG features for image 20181221_Miaoli_Shitan_ms3_69.JPG
2019-10-28 08:24:31,876 DEBUG: Found 18735 points in 8.70158004761s
2019-10-28 08:24:31,880 DEBUG: No segmentation for 20181221_Miaoli_Shitan_ms2_117.JPG, no features masked.
[WARNING] LRE: submodel_0001 failed with: [{"name":"verbose","error":"Cannot convert 9a3aa963-a5f9-4742-a589-907945610be5 to boolean"}]
[INFO]    LRE: Re-queueing submodel_0001 (retries: 2)
[INFO]    LRE: About to process submodel_0002 remotely
[WARNING] LRE: submodel_0002 failed with: [{"name":"verbose","error":"Cannot convert 9a3aa963-a5f9-4742-a589-907945610be5 to boolean"}]
[INFO]    LRE: Re-queueing submodel_0002 (retries: 2)
[INFO]    LRE: About to process submodel_0003 remotely
2019-10-28 08:24:32,044 DEBUG: Found 18237 points in 8.51156592369s
2019-10-28 08:24:32,048 DEBUG: No segmentation for 20181221_Miaoli_Shitan_ms2_055.JPG, no features masked.
[WARNING] LRE: submodel_0003 failed with: [{"name":"verbose","error":"Cannot convert 9a3aa963-a5f9-4742-a589-907945610be5 to boolean"}]
[INFO]    LRE: Re-queueing submodel_0003 (retries: 2)
[INFO]    LRE: About to process submodel_0004 remotely
2019-10-28 08:24:32,166 INFO: Extracting ROOT_HAHOG features for image 20181221_Miaoli_Shitan_ms1_110.JPG
[WARNING] LRE: submodel_0004 failed with: [{"name":"verbose","error":"Cannot convert 9a3aa963-a5f9-4742-a589-907945610be5 to boolean"}]
[INFO]    LRE: Re-queueing submodel_0004 (retries: 2)
[INFO]    LRE: About to process submodel_0001 remotely
[INFO]    LRE: Waiting 20 seconds before processing submodel_0001
2019-10-28 08:24:32,344 INFO: Extracting ROOT_HAHOG features for image 20181221_Miaoli_Shitan_ms1_123.JPG
2019-10-28 08:24:40,034 DEBUG: Found 18739 points in 8.29558491707s
2019-10-28 08:24:40,037 DEBUG: No segmentation for 20181221_Miaoli_Shitan_ms3_69.JPG, no features masked.
2019-10-28 08:24:40,280 INFO: Extracting ROOT_HAHOG features for image 20181221_Miaoli_Shitan_ms3_31.JPG
2019-10-28 08:24:40,478 DEBUG: Found 18499 points in 8.00401902199s
2019-10-28 08:24:40,482 DEBUG: No segmentation for 20181221_Miaoli_Shitan_ms1_123.JPG, no features masked.
2019-10-28 08:24:40,619 DEBUG: Found 17747 points in 8.33730101585s
2019-10-28 08:24:40,621 DEBUG: No segmentation for 20181221_Miaoli_Shitan_ms1_110.JPG, no features masked.
2019-10-28 08:24:40,744 INFO: Extracting ROOT_HAHOG features for image 20181221_Miaoli_Shitan_ms2_010.JPG
2019-10-28 08:24:40,836 INFO: Extracting ROOT_HAHOG features for image 20181221_Miaoli_Shitan_ms3_05.JPG
2019-10-28 08:24:48,584 DEBUG: Found 18830 points in 8.18423914909s
2019-10-28 08:24:48,587 DEBUG: No segmentation for 20181221_Miaoli_Shitan_ms3_31.JPG, no features masked.
2019-10-28 08:24:48,880 INFO: Extracting ROOT_HAHOG features for image 20181221_Miaoli_Shitan_ms1_104.JPG
2019-10-28 08:24:48,982 DEBUG: Found 18861 points in 8.12652301788s
2019-10-28 08:24:48,985 DEBUG: No segmentation for 20181221_Miaoli_Shitan_ms2_010.JPG, no features masked.
2019-10-28 08:24:49,230 INFO: Extracting ROOT_HAHOG features for image 20181221_Miaoli_Shitan_ms1_145.JPG
2019-10-28 08:24:49,709 DEBUG: Found 18264 points in 8.7456908226s
2019-10-28 08:24:49,712 DEBUG: No segmentation for 20181221_Miaoli_Shitan_ms3_05.JPG, no features masked.
2019-10-28 08:24:49,956 INFO: Extracting ROOT_HAHOG features for image 20181221_Miaoli_Shitan_ms3_80.JPG
[WARNING] LRE: submodel_0001 failed with: [{"name":"verbose","error":"Cannot convert 9a3aa963-a5f9-4742-a589-907945610be5 to boolean"}]
[INFO]    LRE: Re-queueing submodel_0001 (retries: 3)
[INFO]    LRE: About to process submodel_0002 remotely
[WARNING] LRE: submodel_0002 failed with: [{"name":"verbose","error":"Cannot convert 9a3aa963-a5f9-4742-a589-907945610be5 to boolean"}]
[INFO]    LRE: Re-queueing submodel_0002 (retries: 3)
[INFO]    LRE: About to process submodel_0003 remotely
[WARNING] LRE: submodel_0003 failed with: [{"name":"verbose","error":"Cannot convert 9a3aa963-a5f9-4742-a589-907945610be5 to boolean"}]
[INFO]    LRE: Re-queueing submodel_0003 (retries: 3)
[INFO]    LRE: About to process submodel_0004 remotely
[WARNING] LRE: submodel_0004 failed with: [{"name":"verbose","error":"Cannot convert 9a3aa963-a5f9-4742-a589-907945610be5 to boolean"}]
[INFO]    LRE: Re-queueing submodel_0004 (retries: 3)
[INFO]    LRE: About to process submodel_0001 remotely
[INFO]    LRE: Waiting 30 seconds before processing submodel_0001
2019-10-28 08:24:57,154 DEBUG: Found 18115 points in 8.14896893501s
2019-10-28 08:24:57,156 DEBUG: No segmentation for 20181221_Miaoli_Shitan_ms1_104.JPG, no features masked.
2019-10-28 08:24:57,396 INFO: Extracting ROOT_HAHOG features for image 20181221_Miaoli_Shitan_ms1_141.JPG
2019-10-28 08:24:57,453 DEBUG: Found 18285 points in 8.08539485931s
2019-10-28 08:24:57,455 DEBUG: No segmentation for 20181221_Miaoli_Shitan_ms1_145.JPG, no features masked.
2019-10-28 08:24:57,602 INFO: Extracting ROOT_HAHOG features for image 20181221_Miaoli_Shitan_ms2_041.JPG
2019-10-28 08:24:59,278 DEBUG: Found 18541 points in 9.20740008354s
2019-10-28 08:24:59,281 DEBUG: No segmentation for 20181221_Miaoli_Shitan_ms3_80.JPG, no features masked.
2019-10-28 08:24:59,523 INFO: Extracting ROOT_HAHOG features for image 20181221_Miaoli_Shitan_ms4_002.JPG
2019-10-28 08:25:04,705 DEBUG: Found 18472 points in 7.18407392502s
2019-10-28 08:25:04,706 DEBUG: No segmentation for 20181221_Miaoli_Shitan_ms1_141.JPG, no features masked.
2019-10-28 08:25:04,936 INFO: Extracting ROOT_HAHOG features for image 20181221_Miaoli_Shitan_ms2_018.JPG
2019-10-28 08:25:06,485 DEBUG: Found 17667 points in 8.79993796349s
2019-10-28 08:25:06,488 DEBUG: No segmentation for 20181221_Miaoli_Shitan_ms2_041.JPG, no features masked.
2019-10-28 08:25:06,732 INFO: Extracting ROOT_HAHOG features for image 20181221_Miaoli_Shitan_ms2_054.JPG
2019-10-28 08:25:08,081 DEBUG: Found 18643 points in 8.44536280632s
2019-10-28 08:25:08,084 DEBUG: No segmentation for 20181221_Miaoli_Shitan_ms4_002.JPG, no features masked.
2019-10-28 08:25:08,311 INFO: Extracting ROOT_HAHOG features for image 20181221_Miaoli_Shitan_ms2_129.JPG
2019-10-28 08:25:12,981 DEBUG: Found 18594 points in 7.92231798172s
2019-10-28 08:25:12,984 DEBUG: No segmentation for 20181221_Miaoli_Shitan_ms2_018.JPG, no features masked.
2019-10-28 08:25:13,125 INFO: Extracting ROOT_HAHOG features for image 20181221_Miaoli_Shitan_ms2_108.JPG
2019-10-28 08:25:15,435 DEBUG: Found 18044 points in 8.58410215378s
2019-10-28 08:25:15,438 DEBUG: No segmentation for 20181221_Miaoli_Shitan_ms2_054.JPG, no features masked.
2019-10-28 08:25:15,576 INFO: Extracting ROOT_HAHOG features for image 20181221_Miaoli_Shitan_ms2_074.JPG
2019-10-28 08:25:17,093 DEBUG: Found 18503 points in 8.69408607483s
2019-10-28 08:25:17,096 DEBUG: No segmentation for 20181221_Miaoli_Shitan_ms2_129.JPG, no features masked.
2019-10-28 08:25:17,341 INFO: Extracting ROOT_HAHOG features for image 20181221_Miaoli_Shitan_ms3_71.JPG
2019-10-28 08:25:21,691 DEBUG: Found 17845 points in 8.48174595833s
2019-10-28 08:25:21,693 DEBUG: No segmentation for 20181221_Miaoli_Shitan_ms2_108.JPG, no features masked.
2019-10-28 08:25:21,826 INFO: Extracting ROOT_HAHOG features for image 20181221_Miaoli_Shitan_ms2_052.JPG
[WARNING] LRE: submodel_0001 failed with: [{"name":"verbose","error":"Cannot convert 9a3aa963-a5f9-4742-a589-907945610be5 to boolean"}]
[INFO]    LRE: Re-queueing submodel_0001 (retries: 4)
[INFO]    LRE: About to process submodel_0002 remotely
[WARNING] LRE: submodel_0002 failed with: [{"name":"verbose","error":"Cannot convert 9a3aa963-a5f9-4742-a589-907945610be5 to boolean"}]
[INFO]    LRE: Re-queueing submodel_0002 (retries: 4)
[INFO]    LRE: About to process submodel_0003 remotely
[WARNING] LRE: submodel_0003 failed with: [{"name":"verbose","error":"Cannot convert 9a3aa963-a5f9-4742-a589-907945610be5 to boolean"}]
[INFO]    LRE: Re-queueing submodel_0003 (retries: 4)
[INFO]    LRE: About to process submodel_0004 remotely
[WARNING] LRE: submodel_0004 failed with: [{"name":"verbose","error":"Cannot convert 9a3aa963-a5f9-4742-a589-907945610be5 to boolean"}]
[INFO]    LRE: Re-queueing submodel_0004 (retries: 4)
[INFO]    LRE: About to process submodel_0001 remotely
[INFO]    LRE: Waiting 40 seconds before processing submodel_0001
2019-10-28 08:25:23,617 DEBUG: Found 18620 points in 7.92439603806s
2019-10-28 08:25:23,620 DEBUG: No segmentation for 20181221_Miaoli_Shitan_ms2_074.JPG, no features masked.
2019-10-28 08:25:23,848 INFO: Extracting ROOT_HAHOG features for image 20181221_Miaoli_Shitan_ms1_111.JPG
2019-10-28 08:25:25,596 DEBUG: Found 18747 points in 8.14708900452s
2019-10-28 08:25:25,598 DEBUG: No segmentation for 20181221_Miaoli_Shitan_ms3_71.JPG, no features masked.
2019-10-28 08:25:25,740 INFO: Extracting ROOT_HAHOG features for image 20181221_Miaoli_Shitan_ms3_84.JPG
2019-10-28 08:25:30,005 DEBUG: Found 18434 points in 8.06343984604s
2019-10-28 08:25:30,006 DEBUG: No segmentation for 20181221_Miaoli_Shitan_ms2_052.JPG, no features masked.
2019-10-28 08:25:30,268 INFO: Extracting ROOT_HAHOG features for image 20181221_Miaoli_Shitan_ms2_113.JPG
2019-10-28 08:25:31,894 DEBUG: Found 17620 points in 7.92464804649s
2019-10-28 08:25:31,896 DEBUG: No segmentation for 20181221_Miaoli_Shitan_ms1_111.JPG, no features masked.
2019-10-28 08:25:32,140 INFO: Extracting ROOT_HAHOG features for image 20181221_Miaoli_Shitan_ms3_65.JPG

engine version 0.9.1
Api version 1.5.3

online
node

i am pretty sure that this is not network issue.

This could be a bug; could you open an issue in https://github.com/OpenDroneMap/ODM/issues ?

1 Like