WebODM Process Failure

I have been having trouble processing an image dataset.
Image data set is from a Photogrammetry mission run with the DJI Zenmuse L1, (Run as a Photogrammetry mission with the 20MP camera, with 80%/80% overlap.)

I have tried to process it multiple times and the error message is basically the same but references a different “no such file or directory” Full console output below, but the part that is different each time is this actual .npz file referenced.

If i process the images as Fast Ortho it works, but any other option always fails. As a last ditch I got a trial version of DJI Terra Pro and it was able to process the dataset so I am pretty sure that the photos themselves are ok.

FileNotFoundError: [Errno 2] No such file or directory: ‘D:\WebODM\resources\app\apps\NodeODM\data\a50cef40-90db-4d59-9c7c-5bbb1410c15f\opensfm\features\DJI_20220828133334_0181_Zenmuse-L1-mission.JPG.features.npz’
[INFO] running “D:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\bin\opensfm” create_tracks “D:\WebODM\resources\app\apps\NodeODM\data\a50cef40-90db-4d59-9c7c-5bbb1410c15f\opensfm”

[INFO] Initializing ODM 2.8.6 - Mon Aug 29 15:12:47 2022 [INFO] ============== [INFO] 3d_tiles: False [INFO] auto_boundary: True [INFO] boundary: {} [INFO] build_overviews: False [INFO] camera_lens: auto [INFO] cameras: {} [INFO] cog: True [INFO] copy_to: None [INFO] crop: 3 [INFO] debug: False [INFO] dem_decimation: 1 [INFO] dem_euclidean_map: False [INFO] dem_gapfill_steps: 3 [INFO] dem_resolution: 2.0 [INFO] depthmap_resolution: 640 [INFO] dsm: True [INFO] dtm: False [INFO] end_with: odm_postprocess [INFO] fast_orthophoto: False [INFO] feature_quality: high [INFO] feature_type: sift [INFO] force_gps: False [INFO] gcp: None [INFO] geo: None [INFO] gps_accuracy: 10 [INFO] ignore_gsd: False [INFO] matcher_neighbors: 0 [INFO] matcher_type: flann [INFO] max_concurrency: 32 [INFO] merge: all [INFO] mesh_octree_depth: 11 [INFO] mesh_size: 200000 [INFO] min_num_features: 10000 [INFO] name: a50cef40-90db-4d59-9c7c-5bbb1410c15f [INFO] no_gpu: False [INFO] optimize_disk_space: False [INFO] orthophoto_compression: DEFLATE [INFO] orthophoto_cutline: False [INFO] orthophoto_kmz: False [INFO] orthophoto_no_tiled: False [INFO] orthophoto_png: False [INFO] orthophoto_resolution: 2.0 [INFO] pc_classify: False [INFO] pc_copc: False [INFO] pc_csv: False [INFO] pc_ept: True [INFO] pc_filter: 2.5 [INFO] pc_geometric: False [INFO] pc_las: False [INFO] pc_quality: high [INFO] pc_rectify: False [INFO] pc_sample: 0 [INFO] pc_tile: False [INFO] primary_band: auto [INFO] project_path: D:\WebODM\resources\app\apps\NodeODM\data [INFO] radiometric_calibration: none [INFO] rerun: None [INFO] rerun_all: False [INFO] rerun_from: ['odm_postprocess'] [INFO] resize_to: 2048 [INFO] rolling_shutter: False [INFO] rolling_shutter_readout: 0 [INFO] sfm_algorithm: incremental [INFO] skip_3dmodel: False [INFO] skip_band_alignment: False [INFO] skip_orthophoto: False [INFO] skip_report: False [INFO] sm_cluster: None [INFO] smrf_scalar: 1.25 [INFO] smrf_slope: 0.15 [INFO] smrf_threshold: 0.5 [INFO] smrf_window: 18.0 [INFO] split: 999999 [INFO] split_image_groups: None [INFO] split_overlap: 150 [INFO] texturing_data_term: gmi [INFO] texturing_keep_unseen_faces: False [INFO] texturing_outlier_removal_type: gauss_clamping [INFO] texturing_skip_global_seam_leveling: False [INFO] texturing_skip_local_seam_leveling: False [INFO] texturing_tone_mapping: none [INFO] tiles: False [INFO] time: False [INFO] use_3dmesh: False [INFO] use_exif: False [INFO] use_fixed_camera_params: False [INFO] use_hybrid_bundle_adjustment: False [INFO] verbose: False [INFO] ============== [INFO] Running dataset stage [INFO] Loading dataset from: D:\WebODM\resources\app\apps\NodeODM\data\a50cef40-90db-4d59-9c7c-5bbb1410c15f\images [INFO] Loading images database: D:\WebODM\resources\app\apps\NodeODM\data\a50cef40-90db-4d59-9c7c-5bbb1410c15f\images.json [INFO] Found 601 usable images [INFO] Coordinates file already exist: D:\WebODM\resources\app\apps\NodeODM\data\a50cef40-90db-4d59-9c7c-5bbb1410c15f\odm_georeferencing\coords.txt [INFO] Model geo file already exist: D:\WebODM\resources\app\apps\NodeODM\data\a50cef40-90db-4d59-9c7c-5bbb1410c15f\odm_georeferencing\odm_georeferencing_model_geo.txt [INFO] Parsing SRS header: WGS84 UTM 19N [INFO] Finished dataset stage [INFO] Running split stage [INFO] Normal dataset, will process all at once. [INFO] Finished split stage [INFO] Running merge stage [INFO] Normal dataset, nothing to merge. [INFO] Finished merge stage [INFO] Running opensfm stage [WARNING] D:\WebODM\resources\app\apps\NodeODM\data\a50cef40-90db-4d59-9c7c-5bbb1410c15f\opensfm\image_list.txt already exists, not rerunning OpenSfM setup [WARNING] D:\WebODM\resources\app\apps\NodeODM\data\a50cef40-90db-4d59-9c7c-5bbb1410c15f\opensfm\exif already exists, not rerunning photo to metadata [WARNING] Detect features already done: D:\WebODM\resources\app\apps\NodeODM\data\a50cef40-90db-4d59-9c7c-5bbb1410c15f\opensfm\features exists [INFO] running "D:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\bin\opensfm" match_features "D:\WebODM\resources\app\apps\NodeODM\data\a50cef40-90db-4d59-9c7c-5bbb1410c15f\opensfm" 2022-08-29 15:12:50,855 INFO: Computing pair matching with 32 processes Traceback (most recent call last): File "D:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\bin\\opensfm_main.py", line 25, in commands.command_runner( File "D:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\opensfm\commands\command_runner.py", line 38, in command_runner command.run(data, args) File "D:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\opensfm\commands\command.py", line 13, in run self.run_impl(data, args) File "D:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\opensfm\commands\match_features.py", line 13, in run_impl match_features.run_dataset(dataset) File "D:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\opensfm\actions\match_features.py", line 14, in run_dataset pairs_matches, preport = matching.match_images(data, {}, images, images) File "D:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\opensfm\matching.py", line 56, in match_images match_images_with_pairs(data, config_override, exifs, pairs), File "D:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\opensfm\matching.py", line 80, in match_images_with_pairs matches = context.parallel_map(match_unwrap_args, args, processes, jobs_per_process) File "D:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\opensfm\context.py", line 58, in parallel_map res = Parallel(batch_size=batch_size)(delayed(func)(arg) for arg in args) File "D:\WebODM\resources\app\apps\ODM\venv\lib\site-packages\joblib\parallel.py", line 1061, in __call__ self.retrieve() File "D:\WebODM\resources\app\apps\ODM\venv\lib\site-packages\joblib\parallel.py", line 940, in retrieve self._output.extend(job.get(timeout=self.timeout)) File "multiprocessing\pool.py", line 768, in get File "multiprocessing\pool.py", line 125, in worker File "D:\WebODM\resources\app\apps\ODM\venv\lib\site-packages\joblib\_parallel_backends.py", line 595, in __call__ return self.func(*args, **kwargs) File "D:\WebODM\resources\app\apps\ODM\venv\lib\site-packages\joblib\parallel.py", line 262, in __call__ return [func(*args, **kwargs) File "D:\WebODM\resources\app\apps\ODM\venv\lib\site-packages\joblib\parallel.py", line 262, in return [func(*args, **kwargs) File "D:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\opensfm\matching.py", line 210, in match_unwrap_args matches = match(im1, im2, camera1, camera2, data, config_override, pose) File "D:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\opensfm\matching.py", line 579, in match p1, p2, matches, matcher_type = _match_descriptors_impl( File "D:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\opensfm\matching.py", line 353, in _match_descriptors_impl features_data1 = feature_loader.instance.load_all_data( File "D:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\opensfm\feature_loading.py", line 109, in load_all_data features_data = self._load_all_data_masked(data, image) File "D:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\opensfm\feature_loading.py", line 160, in _load_all_data_masked features_data = self._load_all_data_unmasked(data, image) File "D:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\opensfm\feature_loading.py", line 154, in _load_all_data_unmasked return self._load_features_nocache(data, image) File "D:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\opensfm\feature_loading.py", line 201, in _load_features_nocache features_data = data.load_features(image) File "D:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\opensfm\dataset.py", line 318, in load_features with self.io_handler.open(features_filepath, "rb") as f: File "D:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\opensfm\io.py", line 1462, in open return open(*args, **kwargs) FileNotFoundError: [Errno 2] No such file or directory: 'D:\\WebODM\\resources\\app\\apps\\NodeODM\\data\\a50cef40-90db-4d59-9c7c-5bbb1410c15f\\opensfm\\features\\DJI_20220828133334_0181_Zenmuse-L1-mission.JPG.features.npz' [INFO] running "D:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\bin\opensfm" create_tracks "D:\WebODM\resources\app\apps\NodeODM\data\a50cef40-90db-4d59-9c7c-5bbb1410c15f\opensfm" 2022-08-29 15:12:52,871 INFO: reading features Traceback (most recent call last): File "D:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\bin\\opensfm_main.py", line 25, in commands.command_runner( File "D:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\opensfm\commands\command_runner.py", line 38, in command_runner command.run(data, args) File "D:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\opensfm\commands\command.py", line 13, in run self.run_impl(data, args) File "D:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\opensfm\commands\create_tracks.py", line 13, in run_impl create_tracks.run_dataset(dataset) File "D:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\opensfm\actions\create_tracks.py", line 12, in run_dataset features, colors, segmentations, instances = tracking.load_features( File "D:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\opensfm\tracking.py", line 29, in load_features features_data = dataset.load_features(im) File "D:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\opensfm\dataset.py", line 319, in load_features return features.FeaturesData.from_file(f, self.config) File "D:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\opensfm\features.py", line 140, in from_file s = np.load(fileobject, allow_pickle=False) File "D:\WebODM\resources\app\apps\ODM\venv\lib\site-packages\numpy\lib\npyio.py", line 445, in load raise ValueError("Cannot load file containing pickled data " ValueError: Cannot load file containing pickled data when allow_pickle=False [INFO] running "D:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\bin\opensfm" reconstruct "D:\WebODM\resources\app\apps\NodeODM\data\a50cef40-90db-4d59-9c7c-5bbb1410c15f\opensfm" Traceback (most recent call last): File "D:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\bin\\opensfm_main.py", line 25, in commands.command_runner( File "D:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\opensfm\commands\command_runner.py", line 38, in command_runner command.run(data, args) File "D:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\opensfm\commands\command.py", line 13, in run self.run_impl(data, args) File "D:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\opensfm\commands\reconstruct.py", line 14, in run_impl reconstruct.run_dataset(dataset) File "D:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\opensfm\actions\reconstruct.py", line 9, in run_dataset tracks_manager = data.load_tracks_manager() File "D:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\opensfm\dataset.py", line 406, in load_tracks_manager return pymap.TracksManager.instanciate_from_file(self._tracks_manager_file(filename)) RuntimeError: Can't read tracks manager file [ERROR] The program could not process this dataset using the current settings. Check that the images have enough overlap, that there are enough recognizable features and that the images are in focus. You could also try to increase the --min-num-features parameter.The program will now exit.

That looks like the race condition error that often appears when using GPU for feature extraction. Were you using High feature quality and was GPU being used?

If so, try ultra feature extraction (depends on your graphics card VRAM as to whether that prevents GPU use) or set ‘no-gpu’ when editing processing options

Yes to both.
I have a Nvidia A6000, it has 48GB of VRAM.

I tried ultra and it also failed within 30seconds.

It is running now with no GPU. Could this be some quirk with my GPU?

It is not restricted to your GPU, it happens with various video cards, and appears to be related to things getting out of sync in the processing. It seems that feature (*.npz) files are called for matching before they have been written.

With 48GB VRAM and 20MP images, using ultra feature extraction wont force CPU feature extraction to be used, as it does with my NVIDIA GeForce GTX1650 Super, which only has 4GB VRAM.

Ah ok, thank you for the insight. Do you know how much this will effect my processing to force CPU? I have a 5950x with 128gb ram.

It’s not really possible to give an absolute answer for all datasets, but I recently did a test on a set of images to get some sort of idea, and using CPU only was not too detrimental on the time taken.
See this post: NVIDIA CUDA graphics card recommendation - #4 by Gordon

Thank you, I understand it is highly dependent on the dataset.

1 Like

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.