Large Data Set - Time & Resources Required?

[coffee_bandit]

Hi all. Looking for a little guidance. I’m trying to run a fairly large data set, 5000 pictures (3MB each-ish, 4000 x 3000 pixels). I’m running WebODM through docker on Windows 10 on a Ryzen 3900X so 24 threads with 128GB RAM. I’ve set docker to use 115GB of RAM (anymore and I get errors in the processing, probably because Windows then doesn’t have enough RAM). Set my swap to 240GB. I’m running the GPU build of WebODM to speed things up a little with my RTX 2070 Super.

Running the dataset with these options:

auto-boundary:true, dsm:true, matcher-neighbors:20, mesh-size:300000, min-num-features:30000, verbose:true

I had increased the min-num-features because my overlap isn’t great, probably around 30%. I’m basically trying to map a large area (2,000 hectares).

The processing completes the opensfm phase and then starts the openmvs stage. I get:

[INFO]    Running dense reconstruction. This might take a while.
[INFO]    Estimating depthmaps
[INFO]    running "/code/SuperBuild/install/bin/OpenMVS/DensifyPointCloud" "/var/www/data/20dbdc88-89c7-461e-9bf5-cc88beb5def5/opensfm/undistorted/openmvs/scene.mvs"  --resolution-level 3 --min-resolution 500 --max-resolution 4000 --max-threads 24 --number-views-fuse 2 -w "/var/www/data/20dbdc88-89c7-461e-9bf5-cc88beb5def5/opensfm/undistorted/openmvs/depthmaps" -v 0 --geometric-iters 0

At which point I got to just over 400hrs (17 days) of total processing and then my virtual drive in docker maxes out at the 1.5TB I had set it to (never thinking it would use all that space). Docker crashes and I have to wipe it and start from scratch. My RAM usage is fine, the swap only ever hits 30GB.

So my question is, is it normal to take that much time (400hrs) and use that much space (1.5TB) just up to the openmvs stage? With my low overlap, is it worth doing a 3D model (or even possible) or just skipping it and setting fast-orthophoto True? The 3D model would be nice to have but not necessary.

I’m thinking of deleting every other picture to drop the total number down to 2,500 if that helps. Or maybe split the pictures into batches and then process them separately?

Open to any suggestions to speed this up and have it complete on me!

[Saijin_Naib]

You might actually need --fast-orthophoto with overlap that low. We’re typically expecting 65% or greater.

And yeah, that could be right for time and size, especially if swapping comes into play. It really tanks performance.

Do you have an estimate for your GSD? You could try passing --pc-sample GSD in cm (4.33 for instance) to keep the density sane and matched to the actual data quality.

[coffee_bandit]

My calculated GSD is about 14cm. WebODM should automatically calculate it and apply that as a limit, no?

--pc-sample GSD is in cm or m? The documentation says m.

Thanks for the ideas. I’ll try again with the --fast-orthophoto enabled.

[Gordon]

With only 30% overlap and a GSD=14cm, features for matching might be fairly sparse.

With you running out of disk space, I wonder if it’s similar to my case here:

What's eating up my HD space?

This is! E:\WebODM\resources\app\data\db\base\16384 This directory has over 500GB of files, some current and clearly in use for the current task, but many of the big files date back to well before I started this particular task. They generally have names like 51926.493 or 51928.2 etc, and are of size 1048576KB, ie ~1GB, and there are 500 of the big files in total. What are they, and are the ones not associated with the current task required? Need to know as HD space is again running short w…

[smathermather]

@Gordon

With you running out of disk space, I wonder if it’s similar to my case here:

Your case is the same as mine: What is eating up my hard disk part dieux (pssst! Spoiler: it's the database not being vacuumed))

This is (probably) different. Rather than losing a bit of space over time, this is during OpenMVS, which does take into the terabytes when processing thousands of images.