Out of memory issue with split parameter on 16GB RAM

[Vincenzo1]

Hi everyone,

I’m processing 1337 images with the following settings:

--split 150 --pc-quality high --feature-quality high --mesh-size 500000 \
--use-3dmesh true --orthophoto-resolution 2.5 --dsm true \
--dem-resolution 2.5 --radiometric-calibration none --skip-report true

From what I understood, the --split 150 option should divide the dataset into smaller sub-blocks to avoid out-of-memory errors. However, my machine still runs out of memory.

I also tried lowering all the settings and running only:

--split 150 --fast-orthophoto true --skip-report true

But I still get an out-of-memory error.

My machine has 16GB of RAM. Is this normal, or am I misunderstanding how the --split parameter works?
Also, I don’t fully understand how --skip and --skip-overlap interact. Could someone explain the difference and how they should be used?

Thanks in advance, and sorry if this is a basic question — I’m new to WebODM processing.
(I also have another open question in the forum ).

Thermal orthophotos

Hi everyone, I’m trying to generate thermal orthophotos of photovoltaic plants using WebODM, from images captured with a DJI Mavic 3T. The missions were planned with 80% side and 85% front overlap, flying at 12 m above the panels, with a resulting GSD of 1.59 cm according to the mission planner. The issue I’m facing is that the resulting orthophoto shows strong distortions, especially in rooftop PV installations, where the panels appear warped and the surfaces are not well represented. Here …

[yanben]

I use the custom .wslconfig file saved as ‪C:\Users\#your user folder\.wslconfig. The wslconfig allows to save the temporary processing files in a diskpart in other drive. Then, I plug in an external ssd drive instead all processing in my: C drive. My experience tells me do not use normal hard drive! You can find more information about the .\wslconfig from the community. Here is my setting based on my lab computer:

[wsl2]
memory=48GB
processors=12
localhostforwarding=true
swap=700GB
swapfile=E:\temp\wsl-swap.vhdx. I am not sure using the swap method would impact the processing quality or not. All best

[Vincenzo1]

Hi yanben,

thanks a lot for your detailed reply and for sharing your configuration — very useful!
You’re right, I could definitely try using an external SSD instead of my internal drive, that might help with the I/O bottleneck.

Just to clarify, in my case the “out of memory” error seems to be related to RAM. From what I understand, the actual limit is the amount of memory I assign to Docker through WSL. My current configuration looks like this:

[wsl2]
memory=14GB
processors=12
swap=4GB

Do you think the crashes I’m seeing are simply because 14 GB isn’t enough for a dataset of ~1300 images, and that I would need to increase the memory (or swap) here? Or could the external SSD setup also help with this type of error?

Thanks again for your help!

[yanben]

With my knowledge and experience, the memory =14 GB is the power of your computer to process your image. Out of memory is because of your computer drive size. I notice that by Webodm will create a huge disk as temp file in C:\Users\#your user folder\AppData\Local\Docker\wsl\disk. If you create a swapfile in your computer, the process will save the temp files in both folders to avoid crash in your :C drive.

[Vincenzo1]

Thanks again for the explanation! That makes sense and I’ll definitely try your suggestion with the swapfile and the external SSD.

Just to share, here’s a part of my log where the error happens:

[WARNING] LRE: submodel_0001 failed with: ...
opendm.system.SubprocessException: Child returned 137

...

[ERROR]   Whoops! You ran out of memory! Add more RAM to your computer, if you're using docker configure it to use more memory, for WSL2 make use of .wslconfig (https://docs.microsoft.com/en-us/windows/wsl/wsl-config#configure-global-options-with-wslconfig), resize your images, lower the quality settings or process the images using a cloud provider (e.g. https://webodm.net).⁠

So it looks like in my case the error is indeed related to RAM. But I’ll still try the swapfile + external SSD setup you suggested to see if that improves the stability.

Thanks a lot for your help

[smathermather]

Yes, RAM is the culprit, but you can add up to 2x your RAM as swap and increase the chances of it completing.