Docker.raw fills disk

[alanterra]

I am still wandering in the darkness, and occasionally falling into holes. Luckily I haven’t broken any bones yet…

I got Linux Mint running on my 2018 Mac Mini, then installed Docker.io and webodm. After a bunch of problems, I finally started stitching photos, and left the process running overnight (over 1,000 images).

I got back to find the web process wedged and my screen full of error messages that Celery had “write error saving DB to disk”, and a warning from Linux that I had zero bytes free on my disk. It turned out that ~/.docker/desktop/vms/0/data/Docker.raw was 1.1 TB in size.

The good news (it’s a miracle!) is that after deleting Docker.raw, rebooting, and reinstalling webodm from github, my results were there!

Any advice on why Docker.raw grew to such size, and how to stop that from happening in the future?

[smathermather]

It’s always quite the journey. Glad all the bones are still in place. There are two options: tune your NodeODM instance (good for multi-user or cases where you don’t want to modify flags for each run, but a bit more complicated to do) or just add the optimize-disk-space flag to each bigger process to ensure cleanup of assets as the processing happens.

Storage is roughly 8-10x whatever you upload, at least until the intermediate products expire, but using optimize-disk-space ensures that as much on-the-go cleanup happens to minimize that, and at the end, you just have the images you uploaded + the products created, without any intermediate products. This comes at the cost of all re-runs of the toolchain having to start from the beginning.

Take a look also at conversations here:

Docker WebODM node consuming huge amounts of data

We haven’t had this question in a while, but it’s a valid and interesting one. For safe measure, I usually tell folks to have 10x the storage of their input data, so 8x makes sense as an actual value you’re seeing. You’ve got a couple options, but the simplest is to add the optimize-disk-space flag: optimize-disk-space Delete heavy intermediate files to optimize disk space usage. This affects the ability to restart the pipeline from an intermediate stage, but allows datasets to be processe…

And here:

Out of Memory - frozen system - ubuntu/docker

Production deployments of OpenDroneMap do require tuning. Defaults often work fine, but your workflow, user base, datasets will drive the tuning of the deployment. In short, see the third answer in the search here: https://community.opendronemap.org/search?q=space%20order%3Alatest Direct link as follows: You can also set up your NodeODM deployment to clean up more often, which allows you to centrally control the cleanup schedule and doesn’t require your users to remember to use the correc…

[alanterra]

Thank you for pointing out some maps I can use.

I forgot to set optimize-disk-space for this run, but I usually do. I sort of thought that 1 TB of free space would be fine for 12 GB of photos, but I did see somewhere in the console output that intermediate results are saved as TIFFs…

Interestingly, whatever caused Docker.raw to blow up happened after the results were complete. I’ll continue to work on this.

[smathermather]

1TB should be enough for 12GB, but it is very close – if it’s closer to 10x for your data, for whatever reason, that would exceed 1TB.

An operating theory could be processing was complete and under the threshold and then it blew up on zipping the final products, but I’m not clear on when/how the blowup happened.

[alanterra]

Just a quick followup on my post. I suspect that the file ~/.docker/desktop/vms/0/data/Docker.raw was left over from when I was running Docker Desktop for Linux, and wasn’t deleted when I switched to Docker.io (Docker service).
Desultory searches suggest that this file is a sparse file system. And I have no idea when this file grew to 1 TB in size. Since I have deleted this file, Docker has not recreated it.
So, in summary, my guess is that the problem reported here is not related to webodm at all, and is only due to the individual history of my computer.
If the powers that be want to delete this thread, it’s fine with me.

[smathermather]

I think I speak for most when I say we’ve all run out of space to annoying and unpredictable results.

No worries! Leaving it is useful for others if they encounter something similar. Glad you figured it out!