Large dataset, need advice to try to make it finish processing

I’ve been trying very unsuccessfully to run a large dataset, just under 4300 images since July 5th. The dataset is from half a section, 320 acres of dryland wheat.

I’m throwing it at my beefiest system:
Processor: Intel® Xeon® CPU E5-2630 v2 @ 2.60GHz, 24 cores
Memory 190 GB
Hard drive space: 7.7 TB available.

I’ve tried everything I can to baby this dataset along. I’m not sure what else to try.

Processing Options: rerun-from: dataset, split-multitracks: true, orthophoto-resolution: 0.5, dtm: true, dem-resolution: 3, ignore-gsd: true, split-overlap: 150, split: 500, fast-orthophoto: true, crop: 3, max-concurrency: 10, camera-lens: brown, skip-3dmodel: true, verbose: true, use-hybrid-bundle-adjustment: true

2020-08-23 12:29:34,839 INFO: Shots and/or GCPs are well-conditionned. Using naive 3D-3D alignment.
2020-08-23 12:30:57,610 INFO: Shots and/or GCPs are well-conditionned. Using naive 3D-3D alignment.
2020-08-23 12:36:17,522 DEBUG: Ceres Solver Report: Iterations: 14, Initial cost: 1.336149e+04, Final cost: 1.299608e+04, Termination: CONVERGENCE
2020-08-23 12:37:06,812 INFO: Removed outliers: 3992
2020-08-23 12:37:24,279 INFO: {‘average_track_length’: 3.173559360817665, ‘points_count’: 1311050, ‘observations_count’: 4160695, ‘cameras_count’: 799, ‘average_track_length_notwo’: 4.864367495113097}
2020-08-23 12:37:24,286 INFO: Reconstruction 0: 799 images, 1311050 points
2020-08-23 12:37:24,286 INFO: 1 partial reconstructions in total.
[INFO] Reconstructing /var/www/data/c54f272c-9432-4644-bf3a-d879d4171f9b/submodels/submodel_0006/opensfm
[INFO] running /usr/bin/env python2 /code/SuperBuild/src/opensfm/bin/opensfm create_tracks “/var/www/data/c54f272c-9432-4644-bf3a-d879d4171f9b/submodels/submodel_0006/opensfm”
2020-08-23 12:39:56,802 INFO: reading features
2020-08-23 12:41:12,914 DEBUG: Merging features onto tracks
2020-08-23 12:42:55,278 DEBUG: Good tracks: 1906415

It always starts off strong, but then it stalls and hangs up. As you can see, the process had last updated 5 days ago, before I terminated it a few minutes ago.

I’d love to see this dataset process, and I’m open to suggestions of what else to try.

I’ve already processed this dataset in Agisoft Metashape (it took about 3 days), but I really hate getting defeated by a computer. I guess this is my unicorn and would like to see this finish.
Are there any tools that could allow me to split the images by quarter section, like what’s in the image below? I’m sure that if I could split this dataset into multiple logical chunks (and still maintain enough overlap to merge them together) I can get this to finish. I’m just not sure what to try from a WebODM stance.
image

1 Like

See this:
https://docs.opendronemap.org/large.html

Especially:
https://docs.opendronemap.org/large.html#local-split-merge

And ask lots of questions. I have a feeling the docs can be improved, but it will help to have someone trying this to know what needs added.

2 Likes

Hi Steve, as you can see from the variables I posted above, I’ve tried the split and split-overlap. I’ve even tried splitting the job into even smaller 300 image chunks too.

In hind site, I should have split the flights into at least three smaller chunks and made sure they overlapped appropriately. I could have then run them in separate tasks like the job I linked below.

This particular job is the reason I brought on the other systems. I’ve started to try to use them with ClusterODM, but my initial attempts didn’t seem to work. I have to revisit that again when I have more time.

As you can see, I’ve processed several other not-so-small jobs successfully and those finished without splitting the jobs. I did have to futz around with the max-concurrency setting, because if I read the documentation right that may be the biggest culprit for the systems running out of resources. Even though my systems are running dual 12 core Xeon processors, and 64 GB to 180 GB of RAM, they kept hanging up on the jobs until I brought I set max-concurency:10 from the default of 24.
image

--max-concurrency <positive integer>
                  The maximum number of processes to use in various
                  processes. Peak memory requirement is ~1GB per thread
                  and 2 megapixel image resolution.
                  Default: number of cores

These jobs consist of oblique crosshatch flights at 30 degrees and a single nadir pass. It is probably overkill, but man… Metashape has been churning out some beautiful models with this strategy.

This dataset finally processed successfully. I deleted the previous task, re-uploaded the images and ran it with:
Options: split: 500, dsm: true

After 84 hours of processing, it finally completed sucessfully.

https://aerial.tools/public/task/bdf3378d-1111-4895-ae6c-712c875ae33b/map/

4 Likes

Whoa! Sweet DSM:
image

5 Likes