Processing exited after submodel stage (3221225477)

WebODM
#1

Further to my previous topic “cannot process dataset” - due to image quality -

I have been attempting a ~10 000 image project on winwebodm to generate ortho, dsm, dtm and 3d model. I am running winwebodm on an amd 5950x / 128gb ram / 2tb ssd / windows 10.

Processing parameters: build-overviews: true, camera-lens: brown, crop: 0, debug: true, dsm: true, dtm: true, feature-quality: ultra, mesh-size: 250000, min-num-features: 10000, orthophoto-kmz: true, orthophoto-png: true, pc-classify: true, pc-csv: true, pc-ept: true, pc-geometric: true, pc-las: true, pc-quality: ultra, pc-rectify: true, rerun-from: opensfm, split: 2500, use-hybrid-bundle-adjustment: true, verbose: true

I was unsuccessful to process the whole set of images (see previous post) and began a new attempt splitting into sets of 2500 images. After 248 hours the process exited with only the message - “processing failed (3221225477)”.

See
https://drive.google.com/drive/folders/1uSWEKr1iGf_31EVcszB3HB6cfVocYWW1?usp=sharing
for console log and others.

What a terrible waste of processing effort. I otherwise could have mined a few bitcoin.

What strategy do you suggest to succeed?

Aside from the undue length of time, the cpu and memory didnt seem to be taxed too greatly, seldom exceeding 20% during matching and reconstructing. At times only 50 - 60 images processed per hour. Your comments on efficient use of resources?

one more thing - does the program need an internet connection at any stage of processing? I want to stop all internet\security related processes on windows

Cannot process dataset, fail after 235 hours
#2

You may need to adjust the --split-overlap parameter:
https://docs.opendronemap.org/large/
https://docs.opendronemap.org/arguments/split-overlap/

I may also suggest dropping --use-hybrid-budle-adjustment

If you’re processing locally, no, you don’t need connectivity outside the machine. However, it does use websockets to communicate between the various components of WebODM, so I’d be very wary about shutting down all networking services.

#3

Thanks.
what I understand you are saying is that I compose the groups for the submodels myself rather than let the process cluster them, and that I set the overlap to 0.

I had a project fail due to some connectivity issue on the docker version but wasnt sure about the Windows one.

I will try out your suggestions in a couple of weeks and let you know the outcome

#4

quick question on grouping images for sub models. Documentation says
image name / model for each line. What is the separator - space tab or comma ? i have generated a csv text file.

#5

Space delimited should be sufficient.

Have you had any success?

#6

I have been attempting for a couple of months now to process this data set of about 10 000 images, so far without success. I have previously been successful when breaking up the project into 3-4000 image blocks, so the images themselves dont appear to be a problem.

I pared down the number of images to 9400 and split the project into 3 submodels. This attempt failed even when I set processing parameters very low. The latest attempt to process 6000 images split into 2 submodels also failed.

Typically error is invalid stod argument after matching so could be something is out of bounds. Memory/storage dont seem to be a problem although one cant be certain there wasnt a spike.

I have uploaded console logs to: webodmItEnviro20211023 - Google Drive

Please let me know if you require additional information. Maybe up against webodm limitations on a single machine with my specs. Running out of options …

Thanks

2 Likes
#7

What version of ODM are you processing against?

I know Piero is tracking issues with the invalid stoi/stod argument error, so there might be some mitigations you’re missing if you’re really back-level.

1 Like
#8

latest version …

1 Like
#9

just fyi - maybe off topic.
using different machine, processing a large 229mb odm generated orthophoto >qgis>otb>error:TrainImagesClassifier: Caught std::exception during application execution: invalid stod argument
must be contagious

1 Like
#10

Have you copied over the old dataset.py file with the newer version Piero mentioned recently?

1 Like
#11

No. Thanks.
Hadnt seen that. I will get on it and post my progress or lack thereof …

2 Likes
#12

I have made another attempt with the newest dataset.py mentioned above but unfortunately it has been unsuccessful. To reiterate - processing 6000 images for orthophoto dsm dtm and 3d model.

  1. After 153 hours webodm 1.9.7 crashed, restarting the computer. This may not have been related to webodm.

  2. I resumed processing without issues and the program failed after a total of 173 hours giving the message:
    Processing failed (4294967295)
    running D:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\bin\opensfm export_visualsfm --points “D:\WebODM\resources\app\apps\NodeODM\data\e8478e5d-b381-4b2b-b6b8-0e81ac70efab\opensfm”

  3. I resumed processing again and webodm failed after another ~ 4 hours giving the message:
    Processing failed (4294967295)
    File “D:\WebODM\resources\app\apps\ODM\opendm\system.py”, line 106, in run
    raise SubprocessException(“Child returned {}”.format(retcode), retcode)
    opendm.system.SubprocessException: Child returned 3221226505
    ===== Done, human-readable information to follow… =====
    [ERROR] The program exited with a strange error code. Please report it at https://community.opendronemap.org

The logs are available at: FailNewDatasetPy.7z - Google Drive

I have not seen any indication there is a hardware limitation to the processing.

Would be interested to hear about successes processing large datasets (>4000 images ) with webodm on local machines.

In the meantime any suggestions for another attempt?

#13

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.