Hi - I am running WebODM on a machine with 128GB RAM (103GB allocated to Docker) and WebODM Diagnostics is showing approx. 5GB being used. I am running 600-1500 image data sets. Any thoughts as to why the full amount of RAM is not being utilized? Thank you in advance.
Amount of Ram is usually related to thread on every software.
One thread = one usage of ram to simplify. So if you don’t allocate vCPU, amount of ram could not grow up … How many vCPU had you assign to webODM ?
Thank you! Yes, I finally discovered max-concurrency! I had plenty allocated in docker, however this parameter seems to limit thread and/or RAM utilization.
Unfortunately now I am getting 100% “Process exited with code 1”…even re-runs that had previously processed. Not sure what’s going on…
Now you are running out of RAM.
Reduce your concurrency by half and see if it completes.
I am curious: how many cores do you have and what are your other settings? With 100GB of RAM, you should safely be able to run datasets of 3000+ under ordinary circumstances.
16-core, 32-thread, 128gb (not fully allocated to Docker). I don’t think I am running out of memory, see attached screen shot…but something is off for sure…
Probably a new topic, but I see my local disk only has 48gb remaining. Is it odd that the WebODM diagnostics is showing so much more (this is an allocation from the same drive)? I think I am running out of storage…but am not able to locate the files. Could they be WebODM deleted runs, etc?
I ended up doing a full Docker ‘purge’ to quickly get back up and running. I lost everything not downloaded and had to download WebODM again, but I was able to get back up and running fairly quickly.
Unfortunately, I think I have discovered that “deleting” in WebODM is not removing items from the hard disk and/or the virtual environment. See two images attached (note when I disable Docker, free space on the C: drive increases from 322GB to 435GB (slightly larger that the 99.5GB used shown in diagnostics…but there is still hundreds of GB somewhere). Note, other than the OS and a few small programs, this drive does not have much else on it. Are there any tips on an easier way to find the files / do file cleanup?
Thanks again for all the help so far!
it’s more a docker knowledge issue IMO. You need to know docker (some examples) :
## info docker images # list docker images docker ps # list running container docker ps -a # list all containers even stopped docker volume ls # list volumes docker network ls # list attached network docker system df # docker disk usage ## Cleaning docker rmi image_name # remove docker image docker stop container_name # stop docker container docker rm container_name # remove stopped container docker system prune -a # WARNING! This will remove: # - all stopped containers # - all networks not used by at least one container # - all dangling images # - all build cache docker system prune -a --volumes # same as above and remove volumes docker volume rm volume_name # remove volume_name docker volume prune # remove all volumes docker network rm prune # remove all network
note that a volume could me mounted as external files, so even with removing volume files will remain present. Watch your docker compose yaml to understand which volume is physically mounted or docker run command options
Thank you for the feedback and I agree that Docker knowledge / commands seem to be the key. I was just wondering if there were alternate (GUI / WebODM) solutions out there to help with this type of file management / cleanup? Thanks again.