What is a stoi argument?

[Gordon]

My near 10000 image set keeps failing after about an hour and a half, the last 2 times with this:

ValueError: invalid stoi argument (previous time was: invalid stod argument)

[ERROR] The program could not process this dataset using the current settings. Check that the images have enough overlap, that there are enough recognizable features and that the images are in focus. You could also try to increase the --min-num-features parameter.The program will now exit.

The images do have enough overlap, are in focus, have tons of features, and I’ve been increasing the min num features, this time: Options: dsm: true, min-num-features: 15000, optimize-disk-space: true

but it doesn’t help.

Less than 5GB of the available 96GB is being used, but the processor is being utilised at up to near 100%.

Suggestions welcome

[machinefly]

Hi Gordon
on what os you have run ODM/WebODM?
whats the parameters of single picture? aspecialy weight.

dealing with 10k pictures on 100GB of RAM can’t be easy.

if you run ODM on Ubuntu - increase swap
from my experience - do not use “optimize-disk-space”

[Gordon]

WebODM on Win 10 Pro. M2Pro drone images scaled to 0.25, so dimensions only 1372 X 912 pixels for processing.
I managed 5000 of these images in 15 hours, but every time I try to do 10000, it fails, and not through lack of memory.

[Gordon]

Another couple of fails with the same error, no idea why it is using only ~2% of available memory most of the time though.

[Saijin_Naib]

Do you know what stage it is failing in? Do you have more of the log before what you posted above?

[Gordon]

Here are a couple of the console logs I saved

https://drive.google.com/drive/folders/1QwspBStsLXWuMSXwGlmJ9EJRGvrPWjlt?usp=sharing

[pierotofy]

It’s a strange bug. When processing fails, are you able to copy the tracks.csv file from the opensfm folder of the dataset that is being processed?

If you can upload that on Google Drive I can try to reproduce this and hopefully fix it.

p.s. stoi is a C++ function that converts strings to numbers. https://www.cplusplus.com/reference/string/stoi/ the error message is not very indicative of what the actual problem might be (perhaps some kind of overflow).

[Gordon]

@pierotofy

tracks.csv file

Unfortunately it appears to be gone from when I cleared out all the failed attempts last night. All I have is the console logs.
I’m trying a smaller subset of 6894 images currently, and it has made it past the 1.5 hour mark.
I’m using Options: dem-resolution: 2.0, dsm: true, dtm: true, mesh-size: 220000, min-num-features: 10000, orthophoto-resolution: 2.0, pc-quality: high
and have only reduced by 0.5 this time. I have also turned off the disk space saver, which maybe removing required files?

Currently 2h15 min in:
2021-10-14 14:26:29,251 DEBUG: No segmentation for DJI_0407_5.JPG, no features masked.
2021-10-14 14:26:29,302 DEBUG: Matching DJI_0515_4.JPG and DJI_0513_4.JPG. Matcher: FLANN (symmetric) T-desc: 29.967 T-robust: 0.016 T-total: 29.987 Matches: 2066 Robust: 2044 Success: True

only using 13%CPU and 3.3GB RAM, but I see there are 3 instances of WebODM running, is that normal?

@pierotofy

p.s. stoi is a C++ function

I did see stod instead of stoi one time when it failed around 1.5 hours too.

[pierotofy]

Ah, well; if it crashes again with the same error, keep a copy of the file and share it? It would help tremendously.

[Gordon]

OK, it has crashed again
6894 images after 26:33:29 6894 26:33:29 Cannot process dataset

I’ve uploaded tracks.csv, the reports folder, consolestoi2.txt. Anything else you need?

https://drive.google.com/drive/folders/1QwspBStsLXWuMSXwGlmJ9EJRGvrPWjlt?usp=sharing

[pierotofy]

Do you have an antivirus or some other security software running on your machine @Gordon? Just trying to narrow the cause of the issue. It looks as if the tracks.csv file got truncated at some point during reconstruction.

[pierotofy]

This is probably a limit of std::stringstream, which is truncating the serialization of the tracks manager in OpenSfM.

[pierotofy]

#include <iostream>
#include <limits>
#include <sstream>

int main() {
    std::cout << ::std::numeric_limits<::std::streamsize>::max() << std::endl;

    std::stringstream ss;
    for (int i = 0; i < 1000; i++) {
        ss << "0";
    }
    std::string pl = ss.str();

    std::stringstream sstream;

    for (size_t i = 0; i < 2275943; i++) {
        sstream << pl;
    }
    sstream << "YO!!";
    sstream.flush();
    std::string out = sstream.str();
    std::cout << out.length() << std::endl;
    std::cout << out.substr(out.length() - 4, std::string::npos);
    return 0;
}

Affects Windows only.

Windows run:

9223372036854775807
536870912
0000

(Wrong)

Linux:

9223372036854775807
2275943004
YO!!

(Correct)

[pierotofy]

Fixed with Read/write tracks using file streams · OpenDroneMap/[email protected] · GitHub

If you want to jump ahead and test before an official update is released for WebODM, you can simply replace this file C:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\opensfm\dataset.py with this one: https://gist.githubusercontent.com/pierotofy/28626fd81bd14650820d3d222aca18d0/raw/51bdeea7ae7c896fcdaf0298d6917fbb3299c436/dataset.py

And re-run the dataset. It should work (or at least, process further than where it’s currently failing).

[Gordon]

@pierotofy

Do you have an antivirus or some other security software running

Only the standard Windows Defender.

[Gordon]

Thanks pierotofy, I’ve just replaced that file, but wont have time to test for a few days as I’m heading back over to finish off the final few square km of the job today.

I’ll then have over 25000 images covering 15sqkm to work with