Any new tricks for memory management?

#1

Hello,

I know this topic has come up many times before and I’ve read through most, if not all, of the related posts, so please understand that I’ve tried to do my research. I am using the Docker version of ODM on an Ubuntu 18.04 machine in GCP. It is a 16 processor VM with 102GB of RAM. I have a very large set of data (tens of thousands of images) that were collected with a high-resolution metric camera. Image dimensions are about 13,000 by 8,000. Now, I know I can’t process these images in a single go, so have written a function that breaks my images up into chunks and tries to process those chunks. My goal is to be able to process about 100 images at a time. However, processing about 100 images at a time I can’t get ODM to run to completion. I continually get a memory allocation error after the line in the log that says
Sorting texture patches... done.

I know these are large images, however, I am using a machine with 102GB of RAM and still overrunning. The command I’m attempting to run is
docker run -ti --cpus=8 --rm -v /home/mydir:/datasets/code opendronemap/odm --project-path /datasets --fast-orthophoto --orthophoto-resolution 25 --ignore-gsd --skip-3dmodel --resize-to 1000 --dem-resolution 25 --orthophoto-compression JPEG --max-concurrency 8 --texturing-skip-global-seam-leveling --texturing-skip-local-seam-leveling --mesh-size 20000 --mesh-octree-depth 8
The native resolution of the data are about 7-10cm, so I’ve tried to resize to 1000 pixels and export both the DSM and ortho at reduced resolution (25cm) thinking that might save me, but alas no. Eventually, I need an ortho at native resolution.

Based on many failed runs, I have a few (hopefully) simple questions.

  1. Am I just out of luck and there is no solution but to increase RAM on my VM until it works? This seems like a bad, not to mention expensive, solution to me.
  2. I am going to try to go below 100 scenes per cluster to see if this helps. I’ve worked my way down from trying 250 images to 100, but I’m afraid if I go much smaller I won’t get enough coverage to matter.
  3. I’ve tried running at max-concurrency 1 and that did not help either.

I know memory management has been a big topic here and often the answer is simply “get more RAM”. I’m hoping that maybe there are some new tricks out there or perhaps I’m doing something wrong that someone can shed some light on. Could it be an issue with using either Ubuntu or Google Cloud? I can provide a full log trace if someone wants it.

Thank you very much for any help folks can provide.

Scott

#2

Small update on this question. I did two things that didn’t help. Reduced the number of images to about 85 and increased memory on my node to 204GB. Still overran the memory. I really feel like I’m doing something wrong here. Would appreciate any ideas others have. Thanks.

#3

Check out the --split and --split-overlap options. We’ve already written code for splitting large datasets. See https://docs.opendronemap.org/large.html

1 Like
#4

Hello Piero,

Thank you for the tip. I tried that option, but got a rather confused error message, so I wrote my own routine to chunk up my dataset and process the chunks. However, I still have a memory issue and I’m very confused. My dataset is about 6cm native resolution, and my files are large (approx 13,000 x 8,000). I realize these are larger than most people would use.

After a lot of experimentation and trial and error, I can process pretty large chunks of data (> 150 files) using --orthophoto-resolution 20 and --dem-resolution 20 and I utilize a reasonable amount of RAM, normally below 25GB max utilization. This is great and allows me to generate something. However, as soon as I go to 15cm resolution the same processing chain will grab more than 204GB of RAM and crash my machine. This doesn’t make any sense to me since I’ve increased my pixel density by less than a factor of 2. I would expect to be able to do that process for less than 50GB of RAM. Any thoughts as to why this might be happening?

Thanks so much for all your work on this software. It really is great and I’d love to use it for my processing chain if I can get some of these issues worked out.

Thank you,
Scott

#5

I forgot to add one thing. The grabbing of RAM is largely independent of the number of images. If I process 30 images to 20cm orthophoto/dem the process will use max RAM of about 11GB. However, if I take this same 30 images and try to process to 15cm, the process will grab over 204GB and crash my machine. This is very odd. Here is a screencap of my log of processing jobs.

image

#6

Mm, I’m not sure what the root problem would be, but the large images might be the culprit. Start with smaller inputs (less images), then progressively increase.