NodeODM log message - Capping memory usage

Johnny5 · 1 March 2022 07:22

I noticed messages in NodeODM logs about memory usage capping and I’d like to understand why this capping happens and how the software determines what cap to set.
In those logs, just before the “Reading data for image…” messages there is always two messages,
“Capping memory usage to nn MB” and “Expecting to process nn images”.

Below are some examples.

From a computer with 64GB RAM, 128GB swap, 2TB SSD, 3GB GPU:
------
2022-03-01 05:36:57,534 INFO: Capping memory usage to ~ 24766.759765625 MB
2022-03-01 05:36:57,534 INFO: Expecting to process 200 images.
------
2022-03-01 04:54:34,976 INFO: Capping memory usage to ~ 25630.984375 MB
2022-03-01 04:54:34,976 INFO: Expecting to process 200 images.
------
2022-03-01 03:08:31,023 INFO: Capping memory usage to ~ 29401.572265625 MB
2022-03-01 03:08:31,023 INFO: Expecting to process 200 images.
------
and an example from a computer with 16GB RAM, 32GB swap, 1TB SSD, 3GB GPU:
------
2022-02-15 07:10:30,155 INFO: Capping memory usage to ~ 4397.548828125 MB
2022-02-15 07:10:30,155 INFO: Expecting to process 57 images.
------

The cap appears to be set much lower than the available CPU RAM, e.g. about half or a quarter of available RAM.
Why does that memory usage cap get set & how is the value calculated?

Saijin_Naib · 1 March 2022 20:52

This is a great question!

I’ve searched the codebases of ODM, NodeODM, WebODM, OpenSFM, and OpenCV, and I can not find that string anywhere:

Capping memory usage to

I had hoped to elucidate things more, but my current understanding, taken from OpenSFM code, is this:
OpenSFM takes something like 90% of the free memory on Windows
Then, it sub-divides it into two halves, one for the parallel processing queue, and one for the processing task itself.

github.com

mapillary/OpenSfM/blob/21c22a9b3a506dcc6da4f4a0931a6b6b56019fb8/opensfm/features_processing.py#L23-L47

      
        
            mem_available = log.memory_available()
            processes = data.config["processes"]
            if mem_available:
                # Use 90% of available memory
                ratio_use = 0.9
                mem_available *= ratio_use
                logger.info(
                    f"Planning to use {mem_available} MB of RAM for both processing queue and parallel processing."
                )
            
            
    # 50% for the queue / 50% for parralel processing
                expected_mb = mem_available / 2
                expected_images = min(
                    max_queue_size, int(expected_mb / average_image_size(data))
                )
                processing_size = average_processing_size(data)
                logger.info(
                    f"Scale-space expected size of a single image : {processing_size} MB"
                )
                processes = min(max(1, int(expected_mb / processing_size)), processes)

This file has been truncated. show original

I will likely need a fact-check on this, however.

And I’d really love to know where that string is hiding

smathermather · 1 March 2022 21:45

It’s hiding in OpenMVS, I’m fairly sure.

Saijin_Naib · 2 March 2022 20:54

Our friend was introduced into OpenSFM’s opensfm/actions/detect_features.py on 2021-06-01:

github.com/mapillary/OpenSfM

feat: cap memory usage of feature detection

committed 12:31PM - 01 Jun 21 UTC

YanNoun

+54 -4

Summary: This Diff aims at limiting the memory usage of the feature extraction. …Since they're no way to explicitely control memory allocation/free in Python, we use the image size and queue size as a proxy for memory usage : producers reading new images to be put in the queue will block until the queue has been freed by any consumers. We use the current free memory / 2 as higher bound, capping by the arbitrary number of 200 images. Reviewed By: paulinus Differential Revision: D28744887 fbshipit-source-id: d62acc98429ddb2ac360bfe62d05730ca5df0d40

It was removed from OpenSFM’s opensfm/actions/detect_features.py on 2021-12-15:

and then added to OpenSFM’s opensfm/features_processing.py on 2021-12-15:

whereupon it was removed from OpenSFM’s opensfm/features_processing.py on 2022-02-21:

github.com/mapillary/OpenSfM

fix: robustness to large-frame camera

committed 07:51PM - 21 Feb 22 UTC

YanNoun

+142 -87

Summary: This Diff strenghten the behaviour of parrallel processing steps with i…mages, namely feature extraction and undistortion. We take into account the memory needed for processing a given image in both processes, in order to better pick a suitable number of // processes and queue size. We also refactor slightly PLY export, so that no intermediate string is created and instead, data is directly flushed to file handler. Reviewed By: mlopezantequera Differential Revision: D34302997 fbshipit-source-id: b988b29450e2036567892492ed78489ad11af3bc

That being said, it looks like the calculation has not changed much in the time since the log message was added and then removed. My concern is whether or not I understood it correctly.

smathermather · 2 March 2022 22:42

Woah! That was a roller coaster.

Johnny5 · 2 March 2022 23:01

Thanks guys!
So at a high level it sounds like the memory cap is to prevent the feature extraction processing consuming all the available RAM on the computer?

Saijin_Naib · 2 March 2022 23:25

Seems so!

The commit message indicates that this is a means to try to manage memory deterministically since Python doesn’t let you do so.

Function is still intact, but the log message changed over time.

git log -S and git log -L are very neat and I just discovered them

system · 1 April 2022 23:26

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.