Can't process 639 pictures with 32GB RAM 3080TI laptop

WebODM
#1

Hello guys,
Extremely new member here.
Tried to install WebODM using Git and Docker, gave it a try, and got some good results on small projects. Now I’m trying something a bit larger and it is not able to process because it lacks memory.
First I tried Ultra Quality, then High Resolution, now Default, and nothing works. Thought it could be that this computer couldn’t handle it at first but I’m starting to doubt it!
I’m currently processing it again and I saw this in the log:
09:25:08 [App] RAM: 15.46GB Physical Memory 4.00 GB Virtual Memory
This laptop has 32GB RAM and 16GB VRAM, I’m clearly doing something wrong.
Can anyone guide me here?

While I was writting this, it failed again, processing node out of memory:

Geometric-consistent estimated depth-maps 623 (97.50%, 9m54s, ETA 15s)…
Geometric-consistent estimated depth-maps 624 (97.65%, 9m54s, ETA 14s)…
Geometric-consistent estimated depth-maps 625 (97.81%, 9m55s, ETA 13s)…
Geometric-consistent estimated depth-maps 626 (97.97%, 9m56s, ETA 12s)…
Geometric-consistent estimated depth-maps 627 (98.12%, 9m57s, ETA 11s)…
Geometric-consistent estimated depth-maps 628 (98.28%, 9m58s, ETA 10s)…
Geometric-consistent estimated depth-maps 629 (98.44%, 9m59s, ETA 9s)…
Geometric-consistent estimated depth-maps 630 (98.59%, 10m, ETA 8s)…
Geometric-consistent estimated depth-maps 631 (98.75%, 10m1s, ETA 7s)…
Geometric-consistent estimated depth-maps 632 (98.90%, 10m2s, ETA 6s)…
Geometric-consistent estimated depth-maps 633 (99.06%, 10m3s, ETA 5s)…
Geometric-consistent estimated depth-maps 634 (99.22%, 10m4s, ETA 4s)…
Geometric-consistent estimated depth-maps 635 (99.37%, 10m5s, ETA 3s)…
Geometric-consistent estimated depth-maps 636 (99.53%, 10m6s, ETA 2s)…
Geometric-consistent estimated depth-maps 637 (99.69%, 10m7s, ETA 1s)…
Geometric-consistent estimated depth-maps 638 (99.84%, 10m7s, ETA 952ms)…
Geometric-consistent estimated depth-maps 639 (100.00%, 10m8s, ETA 0ms)…
Geometric-consistent estimated depth-maps 639 (100%, 10m8s761ms)

Filtered depth-maps 1 (0.16%, 18s, ETA 0ms)…
Filtered depth-maps 26 (4.07%, 18s, ETA 7m)…
Filtered depth-maps 48 (7.51%, 18s, ETA 3m)…
Filtered depth-maps 71 (11.11%, 18s, ETA 2m)…
Filtered depth-maps 96 (15.02%, 18s, ETA 1m)…
Filtered depth-maps 114 (17.84%, 18s, ETA 1m)…
Filtered depth-maps 130 (20.34%, 18s, ETA 1m)…
Filtered depth-maps 139 (21.75%, 18s, ETA 1m)…
Filtered depth-maps 156 (24.41%, 18s, ETA 58s)…
Filtered depth-maps 165 (25.82%, 19s, ETA 54s)…
Filtered depth-maps 174 (27.23%, 19s, ETA 51s)…
Filtered depth-maps 190 (29.73%, 19s, ETA 45s)…
Filtered depth-maps 202 (31.61%, 19s, ETA 42s)…
Filtered depth-maps 216 (33.80%, 19s, ETA 38s)…
Filtered depth-maps 224 (35.05%, 19s, ETA 36s)…
Filtered depth-maps 235 (36.78%, 19s, ETA 34s)…
Filtered depth-maps 252 (39.44%, 19s, ETA 30s)…
Filtered depth-maps 258 (40.38%, 20s, ETA 29s)…
Filtered depth-maps 274 (42.88%, 20s, ETA 26s)…
Filtered depth-maps 284 (44.44%, 20s, ETA 25s)…
Filtered depth-maps 292 (45.70%, 20s, ETA 24s)…
Filtered depth-maps 303 (47.42%, 20s, ETA 22s)…
Filtered depth-maps 313 (48.98%, 20s, ETA 21s)…
Filtered depth-maps 328 (51.33%, 20s, ETA 19s)…
Filtered depth-maps 337 (52.74%, 20s, ETA 18s)…
Filtered depth-maps 356 (55.71%, 20s, ETA 16s)…
Filtered depth-maps 363 (56.81%, 21s, ETA 16s)…
Filtered depth-maps 370 (57.90%, 21s, ETA 15s)…
Filtered depth-maps 378 (59.15%, 21s, ETA 14s)…
Filtered depth-maps 386 (60.41%, 21s, ETA 14s)…
Filtered depth-maps 395 (61.82%, 21s, ETA 13s)…
Filtered depth-maps 403 (63.07%, 21s, ETA 12s)…
Filtered depth-maps 405 (63.38%, 21s, ETA 12s)…
Filtered depth-maps 412 (64.48%, 21s, ETA 12s)…
Filtered depth-maps 429 (67.14%, 21s, ETA 10s)…
Filtered depth-maps 443 (69.33%, 22s, ETA 9s)…
Filtered depth-maps 456 (71.36%, 22s, ETA 8s)…
Filtered depth-maps 469 (73.40%, 22s, ETA 8s)…
Filtered depth-maps 474 (74.18%, 22s, ETA 7s)…
Filtered depth-maps 488 (76.37%, 22s, ETA 6s)…
Filtered depth-maps 503 (78.72%, 22s, ETA 6s)…
Filtered depth-maps 512 (80.13%, 22s, ETA 5s)…
Filtered depth-maps 526 (82.32%, 22s, ETA 4s)…
Filtered depth-maps 538 (84.19%, 22s, ETA 4s)…
Filtered depth-maps 542 (84.82%, 23s, ETA 4s)…
Filtered depth-maps 556 (87.01%, 23s, ETA 3s)…
Filtered depth-maps 569 (89.05%, 23s, ETA 2s)…
Filtered depth-maps 576 (90.14%, 23s, ETA 2s)…
Filtered depth-maps 590 (92.33%, 23s, ETA 1s)…
Filtered depth-maps 605 (94.68%, 23s, ETA 1s)…
Filtered depth-maps 622 (97.34%, 23s, ETA 653ms)…
Filtered depth-maps 633 (99.06%, 24s, ETA 228ms)…
Filtered depth-maps 639 (100%, 24s137ms)

Fused depth-maps 0 (0.00%, 573ms, ETA 0ms)…
Fused depth-maps 1 (0.16%, 1s, ETA 0ms)…
Fused depth-maps 2 (0.31%, 1s, ETA 0ms)…
Fused depth-maps 3 (0.47%, 1s, ETA 0ms)…
Fused depth-maps 4 (0.63%, 1s, ETA 0ms)…
Fused depth-maps 5 (0.78%, 2s, ETA 0ms)…
Fused depth-maps 7 (1.10%, 2s, ETA 3m)…
Fused depth-maps 8 (1.25%, 2s, ETA 3m)…
Fused depth-maps 9 (1.41%, 3s, ETA 4m)…
Fused depth-maps 10 (1.56%, 3s, ETA 3m)…
Fused depth-maps 12 (1.88%, 4s, ETA 3m)…
Fused depth-maps 14 (2.19%, 4s, ETA 3m)…
Fused depth-maps 16 (2.50%, 4s, ETA 2m)…
Fused depth-maps 18 (2.82%, 5s, ETA 2m)…
Fused depth-maps 20 (3.13%, 5s, ETA 2m)…
Fused depth-maps 21 (3.29%, 5s, ETA 2m)…
Fused depth-maps 22 (3.44%, 5s, ETA 2m)…
Fused depth-maps 24 (3.76%, 6s, ETA 2m)…
Fused depth-maps 25 (3.91%, 6s, ETA 2m)…
Fused depth-maps 27 (4.23%, 6s, ETA 2m)…
Fused depth-maps 30 (4.69%, 6s, ETA 2m)…
Fused depth-maps 32 (5.01%, 7s, ETA 2m)…
Fused depth-maps 33 (5.16%, 7s, ETA 2m)…
Fused depth-maps 34 (5.32%, 7s, ETA 2m)…
Fused depth-maps 35 (5.48%, 7s, ETA 2m)…
Fused depth-maps 37 (5.79%, 7s, ETA 2m)…
Fused depth-maps 39 (6.10%, 7s, ETA 2m)…
Fused depth-maps 41 (6.42%, 8s, ETA 2m)…
Fused depth-maps 42 (6.57%, 8s, ETA 2m)…
Fused depth-maps 45 (7.04%, 9s, ETA 2m)…
Fused depth-maps 46 (7.20%, 9s, ETA 2m)…
Fused depth-maps 47 (7.36%, 10s, ETA 2m)…
Fused depth-maps 48 (7.51%, 10s, ETA 2m)…
Fused depth-maps 51 (7.98%, 10s, ETA 2m)…
Fused depth-maps 52 (8.14%, 10s, ETA 2m)…
Fused depth-maps 55 (8.61%, 11s, ETA 1m)…
Fused depth-maps 58 (9.08%, 12s, ETA 2m)…
Fused depth-maps 61 (9.55%, 12s, ETA 2m)…
Fused depth-maps 65 (10.17%, 12s, ETA 1m)…
Fused depth-maps 66 (10.33%, 13s, ETA 1m)…
Fused depth-maps 69 (10.80%, 13s, ETA 1m)…
Fused depth-maps 72 (11.27%, 13s, ETA 1m)…
Fused depth-maps 73 (11.42%, 13s, ETA 1m)…
Fused depth-maps 75 (11.74%, 13s, ETA 1m)…
Fused depth-maps 78 (12.21%, 14s, ETA 1m)…
Fused depth-maps 79 (12.36%, 14s, ETA 1m)…
Fused depth-maps 82 (12.83%, 15s, ETA 1m)…
Fused depth-maps 83 (12.99%, 15s, ETA 1m)…
Fused depth-maps 84 (13.15%, 16s, ETA 1m)…
Fused depth-maps 85 (13.30%, 16s, ETA 1m)…
Fused depth-maps 86 (13.46%, 16s, ETA 1m)…
Fused depth-maps 88 (13.77%, 16s, ETA 1m)…
Fused depth-maps 89 (13.93%, 17s, ETA 1m)…
Fused depth-maps 91 (14.24%, 17s, ETA 1m)…
Fused depth-maps 92 (14.40%, 18s, ETA 1m)…Killed
[WARNING] Sub-scene /var/www/data/8d5c6ac6-2aec-42dd-a9e5-b85130dbc6e2/opensfm/undistorted/openmvs/scene_0000.mvs could not be reconstructed, skipping…
[WARNING] Could not compute PLY for subscene /var/www/data/8d5c6ac6-2aec-42dd-a9e5-b85130dbc6e2/opensfm/undistorted/openmvs/scene_0000.mvs
[INFO] Merging 0 scene files
[ERROR] Could not compute dense point cloud (no PLY files available).

#2

Have you allocated enough memory for docker?

#3

I tried to create a .wslconfig file with the following configuration:

# Settings apply across all Linux distros running on WSL 2
[wsl2]

# Limits VM memory to use no more than 4 GB, this can be set as whole numbers using GB or MB
memory=24GB 

# Sets the VM to use two virtual processors
processors=6

# Specify a custom Linux kernel to use with your installed distros. The default kernel used can be found at https://github.com/microsoft/WSL2-Linux-Kernel
kernel=C:\\temp\\myCustomKernel

# Sets additional kernel parameters, in this case enabling older Linux base images such as Centos 6
kernelCommandLine = vsyscall=emulate

# Sets amount of swap storage space to 8GB, default is 25% of available RAM
swap=8GB

# Sets swapfile path location, default is %USERPROFILE%\AppData\Local\Temp\swap.vhdx
swapfile=C:\\temp\\wsl-swap.vhdx

# Disable page reporting so WSL retains all allocated memory claimed from Windows and releases none back when free
pageReporting=false

# Turn off default connection to bind WSL 2 localhost to Windows localhost
localhostforwarding=true

# Disables nested virtualization
nestedVirtualization=false

# Turns on output console showing contents of dmesg when opening a WSL 2 distro for debugging
debugConsole=true

But it didn’t work

1 Like
#4

Hi Rosendo,

Have you check the diagnostic webpage of WebODM? Does the memory setting is as same as your .wslconfig?

If not, it means your docker didn’t use the setting from .wslconfig. In most causes, the reason is the subsystem running your Linux distribution is still running (restart docker doesn’t work).

Use following code in powershell to check:

wsl --list --running

If yes, you can use this to shutdown the subsystem, note it will shut down all running distributions(not only WebODM):

wsl --shutdown

Then you can restart docker and check diagnostic page again.If the memory setting is correct, you can try to reprocess the dataset.
ref: Advanced settings configuration in WSL | Microsoft Learn

The additional option is using -split flag to split the dataset into small subsets. But I’m not very sure it’ll be helpful in this case.
ref: split — OpenDroneMap 3.0.2 documentation

#5

Hi Alfred,
Thanks for the reply.

Tried those codes but nothing seemed to change.
The diagnostic page shows: Free 208,2 GB | Used: 30,0 GB | Total: 251,0 GB and Free: 11,0 GB | Used: 4,5 GB | Total: 15,5 GB, it was already like that.
While processing, it is still 15,46GB physical memory and 4GB Virtual Memory.
Since I made my .wlsconfig file I can see that Docker uses 15GB and 50% CPU

Made that split to 213 now and the processing is stuck on “Queue” showing nothing on task results

#6

===== Dumping Info for Geeks (developers need this to fix bugs) =====
Child returned 1
Traceback (most recent call last):
File “/code/stages/odm_app.py”, line 81, in execute
self.first_stage.run()
File “/code/opendm/types.py”, line 386, in run
self.next_stage.run(outputs)
File “/code/opendm/types.py”, line 365, in run
self.process(self.args, outputs)
File “/code/stages/splitmerge.py”, line 192, in process
system.run(" ".join(map(double_quote, map(str, argv))), env_vars=os.environ.copy())
File “/code/opendm/system.py”, line 110, in run
raise SubprocessException(“Child returned {}”.format(retcode), retcode)
opendm.system.SubprocessException: Child returned 1

===== Done, human-readable information to follow… =====

[ERROR] Uh oh! Processing stopped because of strange values in the reconstruction. This is often a sign that the input data has some issues or the software cannot deal with it. Have you followed best practices for data acquisition? See Flying Tips — OpenDroneMap 3.0.2 documentation

This is the new info that I got. Processing works with Pix4D, is the problem really the input data?

#7

Hi Rosendo,

Could you bump Docker’s memory allocation up to 24GB or 28GB and change --feature-quality to a lower level and try again?

Is your dataset multispectral?

#8

It’s probably the memory, though it would say that in the error messages :thinking:

Yet for 639 images in 12-20 megapixel resolution, be prepared to have at least 50 Gbyte of memory available. That is physical and virtual combined. I can not remember that 20 Gbyte memory (the 16 Gbyte physical + 4 Gbyte virtual you mention) would be enough for anything …

If things fail, I usually go back to use “default” settings. If I then get errors I increase feature-quality and then min_num_features to values like 24000 and then 48000 and then 96000.

1 Like
#9

Have you rebooted the machine or forecefully restarted the WSL/Linux Subservice host process to have it read the modified .wslconfig?

Is your .wslconfig in the root of your User Directory?
So:

C:\Users\RosendoWRK\.wslconfig

would be the path to the file.

1 Like
#10

Hi Alfred,
No success, I make changes in my .wslconfig, save it, make sure Linux distribution isnt running, start Docker, and in nowhere in the process or diagnostic page shows more than 15GB ram. Lowering feature quality didnt work either. Dataset is not multispectral.

1 Like
#11

Nothing works, maybe the problem is my machine after all

1 Like
#12

All yes! No success

1 Like
#13

Can you post the full contents of the .wslconfig file using the forum’s code block syntax? So, prepend three grave marks before the text, and append three grave marks after. (grave is the ` mark).

1 Like
#14

So the dataset you are trying is first “bigger” dataset?
It could just be the dataset that you are trying.

Maybe download and try one of the following datasets, which are known to work:

There are some datasets with ~400 pics. If they process, you know that it is not the installation or your hardware.
If you are willing to share your dataset, I might give it a try to see if it actually works.

2 Likes