Error when GPU is enabled for processing

Hi ODM-Community,

I installed WebODM on my new laptop today. This new one has mainly a better GPU than my old one…
To test the performance I wanted to process some of my old data sets.
In advance: I have previously been able to process these images on my old laptop without any problems…

Well, on the new laptop, unfortunately, the processing does not work and I receive an error message after a short time. I tried the same dataset on another rather new laptop and found the same problem.

I compared the LOG files of my old laptop with the new ones and the only difference I found was the use of the GPU. Now I have deactivated the use of the GPU for processing and the processing runs without any problem also on the new laptops.

So, is this error known?
Are there GPU types that cause problems?
Is there any other way except disabling the use of the GPU?

Thank you very much for the support.
Sepp

1 Like

Can you post the details of the error from the console log?

1 Like

Hi Gordon,

here is the log:

[INFO] DTM is turned on, automatically turning on point cloud classification
[INFO] Initializing ODM 2.8.8 - Wed Aug 03 15:47:36 2022
[INFO] ==============
[INFO] 3d_tiles: False
[INFO] auto_boundary: True
[INFO] auto_boundary_distance: 0
[INFO] boundary: {}
[INFO] build_overviews: False
[INFO] camera_lens: auto
[INFO] cameras: {}
[INFO] cog: True
[INFO] copy_to: None
[INFO] crop: 3
[INFO] debug: False
[INFO] dem_decimation: 1
[INFO] dem_euclidean_map: False
[INFO] dem_gapfill_steps: 3
[INFO] dem_resolution: 5
[INFO] depthmap_resolution: 640
[INFO] dsm: True
[INFO] dtm: True
[INFO] end_with: odm_postprocess
[INFO] fast_orthophoto: False
[INFO] feature_quality: high
[INFO] feature_type: sift
[INFO] force_gps: False
[INFO] gcp: None
[INFO] geo: None
[INFO] gps_accuracy: 10
[INFO] ignore_gsd: False
[INFO] matcher_neighbors: 0
[INFO] matcher_type: flann
[INFO] max_concurrency: 8
[INFO] merge: all
[INFO] mesh_octree_depth: 11
[INFO] mesh_size: 200000
[INFO] min_num_features: 10000
[INFO] name: cd625ee8-3297-4188-bd05-5299fc01249f
[INFO] no_gpu: False
[INFO] optimize_disk_space: False
[INFO] orthophoto_compression: DEFLATE
[INFO] orthophoto_cutline: False
[INFO] orthophoto_kmz: False
[INFO] orthophoto_no_tiled: False
[INFO] orthophoto_png: False
[INFO] orthophoto_resolution: 5
[INFO] pc_classify: True
[INFO] pc_copc: False
[INFO] pc_csv: False
[INFO] pc_ept: True
[INFO] pc_filter: 2.5
[INFO] pc_geometric: False
[INFO] pc_las: False
[INFO] pc_quality: medium
[INFO] pc_rectify: False
[INFO] pc_sample: 0
[INFO] pc_tile: False
[INFO] primary_band: auto
[INFO] project_path: C:\WebODM\resources\app\apps\NodeODM\data
[INFO] radiometric_calibration: none
[INFO] rerun: None
[INFO] rerun_all: False
[INFO] rerun_from: None
[INFO] resize_to: 2048
[INFO] rolling_shutter: False
[INFO] rolling_shutter_readout: 0
[INFO] sfm_algorithm: incremental
[INFO] skip_3dmodel: False
[INFO] skip_band_alignment: False
[INFO] skip_orthophoto: False
[INFO] skip_report: False
[INFO] sky_removal: False
[INFO] sm_cluster: None
[INFO] smrf_scalar: 1.25
[INFO] smrf_slope: 0.15
[INFO] smrf_threshold: 0.5
[INFO] smrf_window: 18.0
[INFO] split: 999999
[INFO] split_image_groups: None
[INFO] split_overlap: 150
[INFO] texturing_data_term: gmi
[INFO] texturing_keep_unseen_faces: False
[INFO] texturing_outlier_removal_type: gauss_clamping
[INFO] texturing_skip_global_seam_leveling: False
[INFO] texturing_skip_local_seam_leveling: False
[INFO] texturing_tone_mapping: none
[INFO] tiles: False
[INFO] time: False
[INFO] use_3dmesh: False
[INFO] use_exif: False
[INFO] use_fixed_camera_params: False
[INFO] use_hybrid_bundle_adjustment: False
[INFO] verbose: False
[INFO] ==============
[INFO] Running dataset stage
[INFO] Loading dataset from: C:\WebODM\resources\app\apps\NodeODM\data\cd625ee8-3297-4188-bd05-5299fc01249f\images
[INFO] Loading 37 images
[INFO] Wrote images database: C:\WebODM\resources\app\apps\NodeODM\data\cd625ee8-3297-4188-bd05-5299fc01249f\images.json
[INFO] Found 37 usable images
[INFO] Parsing SRS header: WGS84 UTM 37N
[INFO] Finished dataset stage
[INFO] Running split stage
[INFO] Normal dataset, will process all at once.
[INFO] Finished split stage
[INFO] Running merge stage
[INFO] Normal dataset, nothing to merge.
[INFO] Finished merge stage
[INFO] Running opensfm stage
[INFO] Maximum photo dimensions: 5472px
[INFO] Photo dimensions for feature extraction: 2736px
[INFO] CUDA drivers detected
[INFO] Using GPU for extracting SIFT features
[INFO] Altitude data detected, enabling it for GPS alignment
[INFO] [‘use_exif_size: no’, ‘flann_algorithm: KDTREE’, ‘feature_process_size: 2736’, ‘feature_min_frames: 10000’, ‘processes: 8’, ‘matching_gps_neighbors: 0’, ‘matching_gps_distance: 0’, ‘matching_graph_rounds: 50’, ‘optimize_camera_parameters: yes’, ‘reconstruction_algorithm: incremental’, ‘undistorted_image_format: tif’, ‘bundle_outlier_filtering_type: AUTO’, ‘sift_peak_threshold: 0.066’, ‘align_orientation_prior: vertical’, ‘triangulation_type: ROBUST’, ‘retriangulation_ratio: 2’, ‘matcher_type: FLANN’, ‘feature_type: SIFT_GPU’, ‘use_altitude_tag: yes’, ‘align_method: auto’, ‘local_bundle_radius: 0’]
[INFO] Wrote reference_lla.json
[INFO] running “C:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\bin\opensfm” detect_features “C:\WebODM\resources\app\apps\NodeODM\data\cd625ee8-3297-4188-bd05-5299fc01249f\opensfm”
2022-08-03 15:47:42,209 INFO: Scale-space expected size of a single image : 170.66471099853516 MB
2022-08-03 15:47:42,209 INFO: Expecting to queue at most 63 images while parallel processing of 8 images.
2022-08-03 15:47:42,220 INFO: Reading data for image DJI_0541.JPG (queue-size=0
2022-08-03 15:47:42,220 INFO: Reading data for image DJI_0551.JPG (queue-size=0
2022-08-03 15:47:42,220 INFO: Reading data for image DJI_0572.JPG (queue-size=0
2022-08-03 15:47:42,220 INFO: Reading data for image DJI_0598.JPG (queue-size=0
2022-08-03 15:47:42,889 INFO: Reading data for image DJI_0573.JPG (queue-size=1
2022-08-03 15:47:42,892 INFO: Reading data for image DJI_0599.JPG (queue-size=1
2022-08-03 15:47:42,892 INFO: Extracting ROOT_SIFT_GPU features for image DJI_0572.JPG
2022-08-03 15:47:42,892 INFO: Extracting ROOT_SIFT_GPU features for image DJI_0598.JPG
2022-08-03 15:47:42,985 INFO: Reading data for image DJI_0542.JPG (queue-size=1
2022-08-03 15:47:42,986 INFO: Extracting ROOT_SIFT_GPU features for image DJI_0541.JPG
2022-08-03 15:47:43,020 INFO: Reading data for image DJI_0552.JPG (queue-size=1
2022-08-03 15:47:43,020 INFO: Extracting ROOT_SIFT_GPU features for image DJI_0551.JPG
d:\a\odm\odm\superbuild\build\pypopsift_deps\popsift-src\src\popsift\sift_octave.cu:360
Could not allocate 3D DoG array: out of memory
[INFO] running “C:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\bin\opensfm” match_features “C:\WebODM\resources\app\apps\NodeODM\data\cd625ee8-3297-4188-bd05-5299fc01249f\opensfm”
2022-08-03 15:47:44,708 INFO: Altitude is negative (-852315.5920482021) : viewing directions are probably divergent. Using default altitude of 1.0
2022-08-03 15:47:44,708 INFO: Altitude for orientation based matching 1.0
2022-08-03 15:47:44,770 INFO: Matching 324 image pairs
2022-08-03 15:47:44,770 INFO: Computing pair matching with 8 processes
Traceback (most recent call last):
File “C:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\bin\opensfm_main.py”, line 25, in
commands.command_runner(
File “C:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\opensfm\commands\command_runner.py”, line 38, in command_runner
command.run(data, args)
File “C:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\opensfm\commands\command.py”, line 13, in run
self.run_impl(data, args)
File “C:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\opensfm\commands\match_features.py”, line 13, in run_impl
match_features.run_dataset(dataset)
File “C:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\opensfm\actions\match_features.py”, line 14, in run_dataset
pairs_matches, preport = matching.match_images(data, {}, images, images)
File “C:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\opensfm\matching.py”, line 56, in match_images
match_images_with_pairs(data, config_override, exifs, pairs),
File “C:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\opensfm\matching.py”, line 80, in match_images_with_pairs
matches = context.parallel_map(match_unwrap_args, args, processes, jobs_per_process)
File “C:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\opensfm\context.py”, line 58, in parallel_map
res = Parallel(batch_size=batch_size)(delayed(func)(arg) for arg in args)
File “C:\WebODM\resources\app\apps\ODM\venv\lib\site-packages\joblib\parallel.py”, line 1061, in call
self.retrieve()
File “C:\WebODM\resources\app\apps\ODM\venv\lib\site-packages\joblib\parallel.py”, line 940, in retrieve
self._output.extend(job.get(timeout=self.timeout))
File “multiprocessing\pool.py”, line 768, in get
File “multiprocessing\pool.py”, line 125, in worker
File “C:\WebODM\resources\app\apps\ODM\venv\lib\site-packages\joblib_parallel_backends.py”, line 595, in call
return self.func(*args, **kwargs)
File “C:\WebODM\resources\app\apps\ODM\venv\lib\site-packages\joblib\parallel.py”, line 262, in call
return [func(*args, **kwargs)
File “C:\WebODM\resources\app\apps\ODM\venv\lib\site-packages\joblib\parallel.py”, line 262, in
return [func(*args, **kwargs)
File “C:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\opensfm\matching.py”, line 210, in match_unwrap_args
matches = match(im1, im2, camera1, camera2, data, config_override, pose)
File “C:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\opensfm\matching.py”, line 579, in match
p1, p2, matches, matcher_type = _match_descriptors_impl(
File “C:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\opensfm\matching.py”, line 353, in _match_descriptors_impl
features_data1 = feature_loader.instance.load_all_data(
File “C:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\opensfm\feature_loading.py”, line 109, in load_all_data
features_data = self._load_all_data_masked(data, image)
File “C:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\opensfm\feature_loading.py”, line 160, in _load_all_data_masked
features_data = self._load_all_data_unmasked(data, image)
File “C:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\opensfm\feature_loading.py”, line 154, in _load_all_data_unmasked
return self._load_features_nocache(data, image)
File “C:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\opensfm\feature_loading.py”, line 201, in _load_features_nocache
features_data = data.load_features(image)
File “C:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\opensfm\dataset.py”, line 318, in load_features
with self.io_handler.open(features_filepath, “rb”) as f:
File “C:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\opensfm\io.py”, line 1462, in open
return open(*args, **kwargs)
FileNotFoundError: [Errno 2] No such file or directory: ‘C:\WebODM\resources\app\apps\NodeODM\data\cd625ee8-3297-4188-bd05-5299fc01249f\opensfm\features\DJI_0603.JPG.features.npz’
[INFO] running “C:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\bin\opensfm” create_tracks “C:\WebODM\resources\app\apps\NodeODM\data\cd625ee8-3297-4188-bd05-5299fc01249f\opensfm”
2022-08-03 15:47:46,304 INFO: reading features
Traceback (most recent call last):
File “C:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\bin\opensfm_main.py”, line 25, in
commands.command_runner(
File “C:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\opensfm\commands\command_runner.py”, line 38, in command_runner
command.run(data, args)
File “C:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\opensfm\commands\command.py”, line 13, in run
self.run_impl(data, args)
File “C:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\opensfm\commands\create_tracks.py”, line 13, in run_impl
create_tracks.run_dataset(dataset)
File “C:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\opensfm\actions\create_tracks.py”, line 12, in run_dataset
features, colors, segmentations, instances = tracking.load_features(
File “C:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\opensfm\tracking.py”, line 29, in load_features
features_data = dataset.load_features(im)
File “C:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\opensfm\dataset.py”, line 318, in load_features
with self.io_handler.open(features_filepath, “rb”) as f:
File “C:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\opensfm\io.py”, line 1462, in open
return open(*args, **kwargs)
FileNotFoundError: [Errno 2] No such file or directory: ‘C:\WebODM\resources\app\apps\NodeODM\data\cd625ee8-3297-4188-bd05-5299fc01249f\opensfm\features\DJI_0541.JPG.features.npz’
[INFO] running “C:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\bin\opensfm” reconstruct “C:\WebODM\resources\app\apps\NodeODM\data\cd625ee8-3297-4188-bd05-5299fc01249f\opensfm”
Traceback (most recent call last):
File “C:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\bin\opensfm_main.py”, line 25, in
commands.command_runner(
File “C:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\opensfm\commands\command_runner.py”, line 38, in command_runner
command.run(data, args)
File “C:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\opensfm\commands\command.py”, line 13, in run
self.run_impl(data, args)
File “C:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\opensfm\commands\reconstruct.py”, line 14, in run_impl
reconstruct.run_dataset(dataset)
File “C:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\opensfm\actions\reconstruct.py”, line 9, in run_dataset
tracks_manager = data.load_tracks_manager()
File “C:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\opensfm\dataset.py”, line 406, in load_tracks_manager
return pymap.TracksManager.instanciate_from_file(self._tracks_manager_file(filename))
RuntimeError: Can’t read tracks manager file
[ERROR] The program could not process this dataset using the current settings. Check that the images have enough overlap, that there are enough recognizable features and that the images are in focus. You could also try to increase the --min-num-features parameter.The program will now exit.

This is the ‘race condition’ error, which happens with GPU feature extraction, although I’m not sure exactly why, as it happens to me on an apparently random basis.

See: Build51 problems

okay, not good…
what were your settings for the successful recontruction? (regarding your last Build51 problems post)

See the final post in the thread. Things have progressed since then with more recent builds, but the race condition problem still happens sometimes.