WebODM 1 Year experience with freezes, crashes (RAM), settings and hardware

Dear community,

it’s been a year now since I started using ODM and later WebODM and I learned a lot and want to share my experience.
I study geoinformatics and I build and design UAS.
Most of the time I use ODM for mapping in some cases multispectral analysis of vegetation.

There are still some things happening I don’t understand, but maybe one day I will be able to solve all issues :slight_smile:

Please comment on this with you own experience and correct me if something is wrong.

#1: Freezes

Sometimes a task freezes, can’t be canceld, or canceling takes for ever.
The timer resets every refresh and the log is stuck or shows nothing at all.

If you upload a file, that doesn’t belong there, this will happen most of the time.
In almost every other case it was a RAM issue for me, a mistake made in your RAM, and ECC isn’t a fix.

In my case, after a few month woking with ODM I build a new PC (1950x + 32GB 2933C14).
With this PC it happend sometimes, sometimes task went through sometimes they crashed. RAM was 99% stable, like 4 memtest86+ passes.
Later I upgraded to 64GB RAM (3200C15), now stange things happned, and this issue was very present.
Sometimes the RAM was stable, sometimes it wasn’t, stange thing indeed.
So I changed the RAM again to 128GB 3000C14 (Samsung B-Die, a Threadripper thingy) downclocked it to 2666 because this is the last official stable Threadripper OC and it almost solved the issue. I had one task fail this way ever since!

If this happend to me, I needed to power down the PC and back up again, restarting the pc or restarting docker didn’t help.

#2: Crashes

A crashed task was mostly running out of memory, back in the days I only had 32GB this happened alot. Every task above 300 images was very heavy on the RAM and I needed to adjust my settings.

I wrote about this here: Several questions regarding WebODM and RAM

With enough RAM there were crashes, but only with unstable RAM, since my RAM is stable and there is enough I didn’t had a single crash that was RAM related.
There are cases when a dataset isn’t optimized or the settings aren’t best, but this is very individually and there is no standart solution.

#3: Settings

I think we all know how much settings change a task and how important the correct settings for a given dataset are.
Most of the time I take images from a P4P with a 4864 x 3648 resolution, for me resize-to: 2432 worked best.
And because I almost always do vegetaion my other settings are: min-num-features: 24000, matcher-neighbors: 24
Often I just need the orthophoto, so I use: fast-orthophoto: true
With my P4P at an hight of 35m I use: orthophoto-resolution: 1
@50m: orthophoto-resolution: 3
@100m: orthophoto-resolution: 5
Sometimes opensfm-depthmap-method: BRUTE_FORCE helps to improve quality on flat areas, but trees are a problem.

There are some other settings I use in special cases but other than that that is all I do for now.
Be warned min-num-features: 24000 uses a lot of RAM, even for small datasets!

#4: Hardware

As said, most of the time I process pictures from a P4P, in other cases there are pictures from a Sony Alpha 5100 or MAPIR Survey3. But only with the P4P I am very experienced.

Everytime before I start mapping I take sample pictures, adjust the EV and other settings.
I use fixed ISO100 or 200 and a shutter above 400, a fixed WB, style all +1 and I prefer color set to D-Cinelike, it’s not oversaturated and works best for shadows. Sometimes I can only take pictures in the evening, and sharp shadows are bad for my photos :frowning: but with D-Cinelike it is still useable.
I take pictures in 4:3 JPEG, sometimes I take 5000+ images, and I don’t like to post process all from RAW.

For flightplaning I use Dronedeploy or Pix4D, I prefer Dronedeploy but sometimes I need to fly very low, and P4D has a lower minimal flight height.
I almost always use 75% / 75% overlap and adjust my flight height to my needs.

Right now I use a Threadripper 1950x on an Aorus x399 Gaming7 with 128GB 3000C14 @2666C14 and a 1080ti. Docker has it’s dedicated 400GB Intel Optane 900P, and other than WebODM I only use Portainer on this image. Docker for Windows uses 24CPUs and 96GB RAM +4GB SWAP :stuck_out_tongue:
This allowes me to do other work beside.
The CPU isn’t OC’ed but I increased the LLC to Normal, no problem because it’s watercooled.

I hope my experience might help someone, I surely could have used this information a year ago :smiley: