"cannot process dataset" - due to image quality - but image quality is actually good!

I attempted to process a large dataset and it failed on two occasions, exiting with “cannot process dataset”. The last few lines of the log are:


2021-10-02 05:05:40,927 DEBUG: Matching LPBMuangGowBlk06M2P0933.jpg and LPBMuangGowPousiS1M2P0283.jpg. Matcher: FLANN (symmetric) T-desc: 2.485 T-robust: 0.004 T-total: 2.489 Matches: 93 Robust: 70 Success: True
2021-10-02 05:05:40,929 DEBUG: Matching LPBMuangGowPousiS1M2P0198.jpg and LPBMuangGowBlk08M2P0779.jpg. Matcher: FLANN (symmetric) T-desc: 2.367 T-robust: 0.003 T-total: 2.371 Matches: 978 Robust: 945 Success: True
2021-10-02 05:05:40,988 INFO: Matched 43172 pairs (brown-brown: 43172) in 6759.5914055 seconds (0.1565735061150746 seconds/pair).
[INFO] running D:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\bin\opensfm create_tracks “D:\WebODM\resources\app\apps\NodeODM\data\563f5abe-8230-4550-b71f-b0d9f45096fe\opensfm”
2021-10-02 05:07:07,903 INFO: reading features
2021-10-02 05:09:03,178 DEBUG: Merging features onto tracks
2021-10-02 05:14:16,743 DEBUG: Good tracks: 9529735
[INFO] running D:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\bin\opensfm reconstruct “D:\WebODM\resources\app\apps\NodeODM\data\563f5abe-8230-4550-b71f-b0d9f45096fe\opensfm”
Traceback (most recent call last):
File “D:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\bin\opensfm_main.py”, line 15, in
commands.command_runner(
File “D:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\opensfm\commands\command_runner.py”, line 38, in command_runner
command.run(data, args)
File “D:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\opensfm\commands\command.py”, line 12, in run
self.run_impl(data, args)
File “D:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\opensfm\commands\reconstruct.py”, line 11, in run_impl
reconstruct.run_dataset(dataset)
File “D:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\opensfm\actions\reconstruct.py”, line 8, in run_dataset
tracks_manager = data.load_tracks_manager()
File “D:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\opensfm\dataset.py”, line 719, in load_tracks_manager
return pymap.TracksManager.instanciate_from_string(f.read())
ValueError: invalid stoi argument
[ERROR] The program could not process this dataset using the current settings. Check that the images have enough overlap, that there are enough recognizable features and that the images are in focus. You could also try to increase the --min-num-features parameter.The program will now exit.

The first failed run used sift detector and exited at ~53 hours. After looking at the log I changed the detector to hahog and it exited after only 02:39.
All the images in this large 10000+ dataset have been run successfully as subsets 2500, 3000+ with the same parameters or quite similar, so the suggestion that there is something wrong with the images is puzzling. Images are nadir / 85% front and side overlap / elev 60m. I am running winwebodm on an amd 5950x / 128gb ram / 2tb ssd / windows 10.

Processing parameters:
Options: build-overviews: true, camera-lens: brown, crop: 0, debug: true, dsm: true, dtm: true, feature-quality: ultra, mesh-size: 250000, min-num-features: 10000, orthophoto-kmz: true, orthophoto-png: true, pc-classify: true, pc-csv: true, pc-ept: true, pc-geometric: true, pc-las: true, pc-quality: ultra, pc-rectify: true, use-hybrid-bundle-adjustment: true, verbose: true

I have uploaded the full log and a few images - “No segmentation for LPBMuangGow~.jpg, no features masked.” as well as a couple of orthophotos from successful runs of subsets.
https://drive.google.com/drive/folders/1DGmKmF4sjQls-djqiJS7ygfZ0_RYULa9?usp=sharing

Please let me know your suggestions and what other information you would like to see.

Thanks

1 Like

I think one of our libraries is getting data too big for it :
https://en.cppreference.com/w/cpp/string/basic_string/stol

Yeah the error message is not to be followed.

The program is crashing due to some issue in OpenSfM.

It’s a large dataset, so probably related to its size. I don’t think 128GB of RAM will be sufficient to process 10000+ images anyhow, have you tried to use split/merge? Splitting Large Datasets — OpenDroneMap 2.6.3 documentation

1 Like

I havent tried split merge yet as I am trying to determine the limits for processing a single dataset with this configuration. But it seems I may have to go this route.

Just to clarify - Does the message “no segmentation for ?.jpg, no features masked” indicate that no or insufficient features were found in an image and that it will not be used in the processing pipeline? And if that is indeed the meaning, given that the majority of my images are of roads and rooflines and of good quality why do so many fail the detection criteria? And would lowering the threshold for number of features from 10000 to 8000 allow more images to meet the criteria and contribute to the success of the processing?
thanks

Also can you point me to guidance on use of sift vs hahog and flann vs bow?

This means a correctly formatted mask image was not provided. This is to be expected if you are not masking.

Broad strokes, sift is usually best. HAHOG can find more tie points in some situations, but is pretty slow in comparison.

Flann is a good matcher, but bow can be faster and more resource efficient at the cost of slightly less matching robustness.

2 Likes

Further to the above topic.

I successfully processed a 3700 image submodel of the ~10 000 image set in ~ 56 hours using the original parameters above. there were no issues.

I tried a split merge then of the whole image set - now 9900 images. Split was 3300, overlap 150 otherwise using the same parameters. Processing failed with the same message at the sfm stage. It doesnt look like even the the first submodel was reconstructed. I tried resuming from the sfm stage but it exited after a couple of minutes.

========================================

2021-10-09 21:53:20,273 INFO: Shots and/or GCPs are well-conditioned. Using naive 3D-3D alignment.
2021-10-09 22:12:37,996 DEBUG: Ceres Solver Report: Iterations: 11, Initial cost: 6.752372e+04, Final cost: 6.673224e+04, Termination: CONVERGENCE
2021-10-09 22:14:24,364 INFO: Removed outliers: 832
2021-10-09 22:15:04,862 INFO: Reconstruction 0: 3704 images, 4490331 points
2021-10-09 22:15:04,862 INFO: 1 partial reconstructions in total.
[INFO] Reconstructing D:\WebODM\resources\app\apps\NodeODM\data\4c7da010-c987-40bd-94dc-43685fffc6e1\submodels\submodel_0001\opensfm
[INFO] running D:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\bin\opensfm create_tracks “D:\WebODM\resources\app\apps\NodeODM\data\4c7da010-c987-40bd-94dc-43685fffc6e1\submodels\submodel_0001\opensfm”
2021-10-09 22:25:12,245 INFO: reading features
2021-10-09 22:28:51,084 DEBUG: Merging features onto tracks
2021-10-09 22:31:39,602 DEBUG: Good tracks: 6835310
[INFO] running D:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\bin\opensfm reconstruct “D:\WebODM\resources\app\apps\NodeODM\data\4c7da010-c987-40bd-94dc-43685fffc6e1\submodels\submodel_0001\opensfm”
Traceback (most recent call last):
File “D:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\bin\opensfm_main.py”, line 15, in
commands.command_runner(
File “D:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\opensfm\commands\command_runner.py”, line 38, in command_runner
command.run(data, args)
File “D:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\opensfm\commands\command.py”, line 12, in run
self.run_impl(data, args)
File “D:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\opensfm\commands\reconstruct.py”, line 11, in run_impl
reconstruct.run_dataset(dataset)
File “D:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\opensfm\actions\reconstruct.py”, line 8, in run_dataset
tracks_manager = data.load_tracks_manager()
File “D:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\opensfm\dataset.py”, line 719, in load_tracks_manager
return pymap.TracksManager.instanciate_from_string(f.read())
ValueError: invalid stoi argument
[ERROR] The program could not process this dataset using the current settings. Check that the images have enough overlap, that there are enough recognizable features and that the images are in focus. You could also try to increase the --min-num-features parameter.The program will now exit.

==========================================

Console log LPBMuangGowAllConsoleFail20211005 is at: LPBMuangGowAllConsoleFail20211005.txt - Google Drive

i have reduced the split to 2500. Will let you know what happens.

2 Likes

I’m over 8 hours into 9655 M2P (resized to 0.2X) images with
Options: dsm: true, mesh-size: 240000, min-num-features: 11000, optimize-disk-space: true, orthophoto-png: true, split: 2000, split-overlap: 200
I’ve previously had success with subsets of most of this area, so I hope it works.

Currently it is only using 15% CPU and 1/10 the available memory, so perhaps I should have made fewer, larger splits. The log hasn’t updated in over 20 mins, so I hope it is still working!

2021-10-10 18:25:05,512 INFO: -------------------------------------------------------
2021-10-10 18:25:05,549 INFO: DJI_0376.JPG resection inliers: 1077 / 1146

1 Like

A hour later…

2021-10-10 19:15:37,562 INFO: DJI_0242_5.JPG resection inliers: 11 / 36
2021-10-10 19:15:37,563 INFO: Adding DJI_0242_5.JPG to the reconstruction
2021-10-10 19:15:39,887 INFO: -------------------------------------------------------
2021-10-10 19:15:39,901 INFO: DJI_0241_5.JPG resection inliers: 44 / 73
2021-10-10 19:15:39,902 INFO: Adding DJI_0241_5.JPG to the reconstruction
2021-10-10 19:15:42,301 INFO: -------------------------------------------------------
2021-10-10 19:15:42,314 INFO: DJI_0240_6.JPG resection inliers: 19 / 49
2021-10-10 19:15:42,316 INFO: Adding DJI_0240_6.JPG to the reconstruction
2021-10-10 19:15:44,636 INFO: -------------------------------------------------------
2021-10-10 19:15:44,652 INFO: DJI_0239_6.JPG resection inliers: 11 / 48
2021-10-10 19:15:44,653 INFO: Adding DJI_0239_6.JPG to the reconstruction
2021-10-10 19:15:46,997 INFO: -------------------------------------------------------
2021-10-10 19:15:47,002 INFO: DJI_0541_11.JPG resection inliers: 11 / 14
2021-10-10 19:15:47,003 INFO: Adding DJI_0541_11.JPG to the reconstruction
2021-10-10 19:15:49,201 INFO: -------------------------------------------------------
2021-10-10 19:15:49,203 INFO: Some images can not be added
2021-10-10 19:15:49,204 INFO: -------------------------------------------------------
2021-10-10 19:15:49,239 INFO: Shots and/or GCPs are well-conditioned. Using naive 3D-3D alignment.
2021-10-10 19:33:57,368 DEBUG: Ceres Solver Report: Iterations: 101, Initial cost: 4.418946e+04, Final cost: 3.312729e+04, Termination: NO_CONVERGENCE
2021-10-10 19:34:43,750 INFO: Removed outliers: 26706
2021-10-10 19:35:00,125 INFO: Reconstruction 0: 3642 images, 1860230 points
2021-10-10 19:35:00,125 INFO: 1 partial reconstructions in total.
[INFO] Aligning submodels…

No errors listed, however

09:01:35       Processing failed (3221225477)

I guess I’ll try it all in one next.

How much RAM do you have?
If this run doesnt work for me I am considering lowering the mesh size to 200 000 just to see if that is successful.

96GB, with a 1TB SSD devoted to WebODM, plenty of free space

1 Like

Possibly related: Specific photo split-merge Aligning submodels Segmentation fault · Issue #1232 · OpenDroneMap/ODM · GitHub

3 Likes

36 hours into trying all images at once (scaled to 0.2X size), and 22 hours into:

[INFO] running E:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\bin\opensfm export_visualsfm --points “E:\WebODM\resources\app\apps\NodeODM\data\79751b08-fb14-4dec-9090-ff4ffd46b342\opensfm”

it is a bit disturbing to see WebODM is only using 1.3% processor time and <2% of RAM, so not exactly making full use of the available resources!
I’m not sure if that is normal behaviour, or it’s about to fail.

38h23m into it now, and WebODM CPU time down to 0.5%.

Is this expected, or is there some sort of slow process death happening?

For export_visualsfm that seems a bit long; usually that’s a relatively fast operation.

2 Likes

No errors showing, 41 hours now, still ~1% CPU usage, and nothing has changed in the log window.
With nearly 10000 images to work on, I would think rather more resources should be in use.
I guess I’ll have to kill it as no progress is being made, but is there anyway to find out what the issue might be, or what parameters I should change for the next attempt?

EDIT - I see the problem, my HD has 0 bytes available! This is a 1TB SSD drive with only WebODM on it. I guess I should have bought a 2TB drive!

I’ll free up 118GB by moving the files to load to another drive.

It seems Optimise disk space wasn’t sufficient.

Freeing up the extra disk space didn’t help, it hasn’t been utilised, and the CPU and RAM usage remains very low.
Next step is to cancel (which is taking a very long time) and resume to see if things will get moving again.

Gave up waiting, it was still stuck unable to finish cancelling after 40 mins. Can’t even delete it, have to crash out of this one. Restarting from scratch again
:disappointed:

After closing WebODM and restarting, this project allowed me to resume processing. CPU usage is high, but RAM still only very low.

After 3.5 more hours, it has failed,
Processing failed (4294967295)

Not sure if it was due to the re-start, as I’ve found they usually fail after restarts, but I’ll try again tomorrow.

Any suggestions for different settings?

here is the end of the log:

2021-10-12 18:09:01,644 DEBUG: No segmentation for DJI_0854_2.JPG, no features masked.
2021-10-12 18:09:01,662 DEBUG: No segmentation for DJI_0865.JPG, no features masked.
2021-10-12 18:09:01,684 DEBUG: Matching DJI_0450_4.JPG and DJI_0452_5.JPG. Matcher: FLANN (symmetric) T-desc: 0.361 T-robust: 0.001 T-total: 0.363 Matches: 738 Robust: 723 Success: True
2021-10-12 18:09:01,905 DEBUG: Matching DJI_0871.JPG and DJI_0850_2.JPG. Matcher: FLANN (symmetric) T-desc: 0.395 T-robust: 0.000 T-total: 0.395 Matches: 57 Robust: 49 Success: True
2021-10-12 18:09:01,919 DEBUG: Matching DJI_0883_4.JPG and DJI_0923_4.JPG. Matcher: FLANN (symmetric) T-desc: 0.398 T-robust: 0.001 T-total: 0.399 Matches: 808 Robust: 762 Success: True
2021-10-12 18:09:01,953 DEBUG: Matching DJI_0270_6.JPG and DJI_0295_6.JPG. Matcher: FLANN (symmetric) T-desc: 0.383 T-robust: 0.001 T-total: 0.384 Matches: 384 Robust: 373 Success: True
2021-10-12 18:09:01,960 DEBUG: Matching DJI_0617_9.JPG and DJI_0616_9.JPG. Matcher: FLANN (symmetric) T-desc: 0.403 T-robust: 0.002 T-total: 0.406 Matches: 1588 Robust: 1583 Success: True
2021-10-12 18:09:01,960 DEBUG: Matching DJI_0002_4.JPG and DJI_0573_11.JPG. Matcher: FLANN (symmetric) T-desc: 0.380 Matches: FAILED
2021-10-12 18:09:01,971 DEBUG: Matching DJI_0854_2.JPG and DJI_0865.JPG. Matcher: FLANN (symmetric) T-desc: 0.346 T-robust: 0.001 T-total: 0.347 Matches: 36 Robust: 28 Success: True
2021-10-12 18:09:02,037 INFO: Matched 43200 pairs (brown-brown: 43200) in 1954.3968936 seconds (0.045240668858796296 seconds/pair).
[INFO] running E:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\bin\opensfm create_tracks “E:\WebODM\resources\app\apps\NodeODM\data\79751b08-fb14-4dec-9090-ff4ffd46b342\opensfm”
2021-10-12 18:09:43,425 INFO: reading features
2021-10-12 18:12:00,576 DEBUG: Merging features onto tracks
2021-10-12 18:14:26,028 DEBUG: Good tracks: 5433154
[WARNING] Found a valid OpenSfM reconstruction file in: E:\WebODM\resources\app\apps\NodeODM\data\79751b08-fb14-4dec-9090-ff4ffd46b342\opensfm\reconstruction.json
[INFO] Already extracted cameras
[INFO] Export reconstruction stats
[WARNING] Found existing reconstruction stats E:\WebODM\resources\app\apps\NodeODM\data\79751b08-fb14-4dec-9090-ff4ffd46b342\opensfm\stats\stats.json
[WARNING] Will skip exporting E:\WebODM\resources\app\apps\NodeODM\data\79751b08-fb14-4dec-9090-ff4ffd46b342\opensfm\reconstruction.geocoords.json
[INFO] Undistorting E:\WebODM\resources\app\apps\NodeODM\data\79751b08-fb14-4dec-9090-ff4ffd46b342\opensfm …
[WARNING] Already undistorted (nominal)
[WARNING] Found a valid OpenSfM NVM reconstruction file in: E:\WebODM\resources\app\apps\NodeODM\data\79751b08-fb14-4dec-9090-ff4ffd46b342\opensfm\undistorted/reconstruction.nvm
[INFO] Finished opensfm stage
[INFO] Running openmvs stage
[WARNING] Found existing E:\WebODM\resources\app\apps\NodeODM\data\79751b08-fb14-4dec-9090-ff4ffd46b342\opensfm\undistorted\openmvs\scene.mvs
[INFO] Running dense reconstruction. This might take a while.
[INFO] Estimating depthmaps
[INFO] running E:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\OpenMVS\DensifyPointCloud “E:\WebODM\resources\app\apps\NodeODM\data\79751b08-fb14-4dec-9090-ff4ffd46b342\opensfm\undistorted\openmvs\scene.mvs” --resolution-level 2 --min-resolution 320 --max-resolution 1094 --max-threads 16 --number-views-fuse 2 -w “E:\WebODM\resources\app\apps\NodeODM\data\79751b08-fb14-4dec-9090-ff4ffd46b342\opensfm\undistorted\openmvs\depthmaps” -v 0 --geometric-iters 0
===== Dumping Info for Geeks (developers need this to fix bugs) =====
Child returned 3221226505
Traceback (most recent call last):
File “E:\WebODM\resources\app\apps\ODM\stages\odm_app.py”, line 94, in execute
self.first_stage.run()
File “E:\WebODM\resources\app\apps\ODM\opendm\types.py”, line 341, in run
self.next_stage.run(outputs)
File “E:\WebODM\resources\app\apps\ODM\opendm\types.py”, line 341, in run
self.next_stage.run(outputs)
File “E:\WebODM\resources\app\apps\ODM\opendm\types.py”, line 341, in run
self.next_stage.run(outputs)
[Previous line repeated 1 more time]
File “E:\WebODM\resources\app\apps\ODM\opendm\types.py”, line 322, in run
self.process(self.args, outputs)
File “E:\WebODM\resources\app\apps\ODM\stages\openmvs.py”, line 79, in process
system.run(’%s “%s” %s’ % (context.omvs_densify_path,
File “E:\WebODM\resources\app\apps\ODM\opendm\system.py”, line 106, in run
raise SubprocessException(“Child returned {}”.format(retcode), retcode)
opendm.system.SubprocessException: Child returned 3221226505

===== Done, human-readable information to follow… =====

[ERROR] The program exited with a strange error code. Please report it at https://community.opendronemap.org