Error after over 91 hours

[Gordon]

Having a go at the dataset here:

Multiple ground "layers" in reconstruction

Hi, I cannot find another such case so here it is: The flight(s) have been flown with a matrice 300, p1 camera (40Mp - 3:2). While part of the model seems to be reconstructed properly, you can see in the “capture” screenshot of the 3d model that the result in the algorithm is three different ground “layers”. What causes such breakdown? Any ideas on what if any parameters I should tweak? running on a 5950x with 128GB RAM hence the 28 max-concurrency Also for the same reason I have to split t…

It failed for reasons not obvious too me (Check failed: num_nonzeros_ >= 0, and the reasons given do not apply), when adding images to the reconstruction -

[Κιλκίς - Σταθμός Μουριών - 17/11/2022]
171 images 91:24:03

Cannot process dataset

|Created on:|30/12/2022, 11:15:52|
|Processing Node:|node-odm-1 (manual)|
|Options:|auto-boundary: true, crop: 5, dem-resolution: 2.0, dtm: true, feature-quality: ultra, gps-accuracy: 6, min-num-features: 20000, orthophoto-resolution: 1.0, pc-quality: ultra, pc-rectify: true, use-3dmesh: true|

2023-01-03 06:11:18,125 INFO: DJI_20221117143611_0005.JPG resection inliers: 238569 / 239578
2023-01-03 06:11:21,918 INFO: Adding DJI_20221117143611_0005.JPG to the reconstruction
2023-01-03 06:11:59,716 INFO: -------------------------------------------------------
2023-01-03 06:12:08,233 INFO: DJI_20221117143351_0041.JPG resection inliers: 235431 / 236112
2023-01-03 06:12:12,123 INFO: Adding DJI_20221117143351_0041.JPG to the reconstruction
2023-01-03 06:12:54,861 INFO: -------------------------------------------------------
2023-01-03 06:13:05,863 INFO: DJI_20221117143724_0055.JPG resection inliers: 220102 / 220228
2023-01-03 06:13:12,166 INFO: Adding DJI_20221117143724_0055.JPG to the reconstruction
2023-01-03 06:13:31,133 INFO: Shots and/or GCPs are well-conditioned. Using naive 3D-3D alignment.
D:\a\ODM\ODM\SuperBuild\src\ceres\internal\ceres\block_sparse_matrix.cc:80 Check failed: num_nonzeros_ >= 0
[ERROR] The program could not process this dataset using the current settings. Check that the images have enough overlap, that there are enough recognizable features and that the images are in focus. You could also try to increase the --min-num-features parameter.The program will now exit.

---

There is plenty of overlap in the images, they are in focus, and there were Millions of features in each image, with up to 300,000 common features when matching.

389GB of memory available including virtual.

Console log available if useful.

Graph of final 24 hours resource use

[gabrieldeluca]

@Gordon

ceres\block_sparse_matrix.cc:80 Check failed: num_nonzeros_ >= 0

It could be that the limit number of elements in the Ceres solution matrix was exceeded. If you can try to limit in some way (by changing the feature type or manually adjusting the opensfm configuration if you dare) the number of features or matches to verify if that is the problem.

[Gordon]

Life is too short to try ultra settings again I’ve got other projects to process.

I did run at medium feature quality and it all came out at one level, which is what the OP had a problem with.

[SURV69]

When does one know, that WebODM, is finished, by virtue of poor data or errors.

I have 1200 photos, that worked well in WebODM 1.9.3(?), that’s been cooking for 20.5 hours, with the bar indicator stuck at the same spot(one space past "Processing ), which is always the typical location before the program begins it’s finishing processing.

My problem is, the last timed indicator is now about 11 hours ago, and the last 9 lines is [INFO], with no time designation, and no new line entries for at least 5 or so hours…

How can one tell that the program, has in fact finished or can not finish?

If there is no error indicator or hard crash, does this mean the computations are continuing, with no timed processing steps?

In the WebODM3, I used all defaults, including “High Resolution”, something I rarely use.

BTW, are measure points “scalable”, and how can one be very “certain”, that the points are/are not “hot”?

[Gordon]

@SURV69

How can one tell that the program, has in fact finished or can not finish?

In Windows you can check details in Task Manager to make sure CPU and memory usage, commits etc are changing. If they are not changing at, all it’s a good indication things have ground to a halt and you need to start again with
different settings.

Also, in another thread I posted this:

@Gordon

Progress bar was almost exactly half way at 27h 38m, and the job finished at 28h 04m. The 2nd half of the progress bar represented just 1.5% of the total time in this case.

The first half isn’t always 1.5%, but the 2nd half is always a lot less time than the 1st.

Also here, where no movement for over 18 hours:

Processing Stalled?

Is your memory and CPU usage varying at all? I’m currently 4 days into a task and there has been no update of the console or movement of the progress bar for almost 18 hours. However, something is happening, as can be determined by computer activity, even if WebODM doesn’t indicate anything. Red= SSD space, Brown = CPU use, Blue=RAM, 0:00 time = now [CPUwith-no-console-update]