My SSD E drive is 1TB in size, and it only has WebODM on it, and only the current task’s files.
Can I remove the resized images in the above directory when the images are also stored in the split directories and the undistorting is 12 hours in since any activity shown in the console log?
E:\ is down to 1GB of free space, so I need to do something ASAP.
TIA
EDIT: I’ve had to do it anyway, as I was down to 850MB and that was decreasing quite quickly.
So it would appear that 1TB is not really a big enough drive to accommodate a ~9400 image task split-merged into 3 submodels.
Hmm, hard to know what is going on, as this line in the console:
2021-12-01 00:20:55,335 DEBUG: Undistorting image DJI_0743_7.JPG
appeared nearly 14 hours ago. Surely when most images are apparently undistorting in fractions of a second, the final image doesn’t take over half a day?
Task Manager was already indicating not much was happening in WebODM when I moved the reduced images to another drive, minimal CPU activity or memory use, perhaps related to the rapidly diminishing available memory?
If the images I moved were still in use, can I move all the images in:
E:\WebODM\resources\app\apps\NodeODM\data\5202fe49-c38a-4215-b5e7-c8b409128bb6\submodels\submodel_0000\images etc
I would have thought these directories of images would be the ones in use for undistorting?
Why does there need to be multiple copies of the same images, using up valuable disk space?
That’s pretty good for the already compressed JPEG format, but doesn’t solve the problem of WebODM storing the same set of ~9400 reduced size images (~47GB each in this case) in multiple places-
If I’m understanding the purpose/intent properly, it is likely that they’re copies because MacOS/Linux/Windows all put symlinks behind admin/elevation and that’s not something we want when just processing data.
Since we can’t just have one copy and symlink it everywhere it needs to be, we have to copy them if they’re to be in the various sub-directories where they need to be.
As for why we don’t just keep the one copy from import and change all the various sub-processess to reference that data as needed, I assume that is because that’s a whole nasty house of cards and a lot of patches to keep in our source trees. Don’t take my word on this, however.
Definitely seems like something that might be possible to optimize in the future, pending no horrible constraints that prevent it. Less file space on disk is lovely, and less copy/write means faster job time in total and less wear over time. I like.
It surprises me that a 1TB drive with nothing but WebODM on it can’t handle a 9400 image dataset without running out of space, even after I have removed all the other tasks.
I haven’t had this issue. I have 4 TB allocated and ran 90,000 images through. But I am running Linux, so maybe there’s a difference in how data are handled on disks that may or may not handle file links on mac and windows.
I was about to say I think I’ll have to give up on this, as posted earlier nothing much is happening and it’s been over 16 hours since the console updated. Now about 52 hours into this task, although the timer reckons it is only 38:55*, but I just noticed the available space on the drive changed by 2GB, so maybe something is happening, just verrry verrry slowwwwwwwly.
I did copy some of the images back to E:, where there is currently 5.9GB available, and slowly increasing.
Time didn’t change even after ‘force reload’ or running diagnostic.
Admittedly not knowing how it is all done, I would have thought the copy used for undistorting would be the copy in the resources\ submodel directory, or perhaps the \images directory, rather than all the way back up to \media\project\2?
EDIT - Just checked and there are about the same number of images listed as ‘undistorting’ as there are in submodel_0000, so I assume it is using images from the \submodel directories, which I didn’t move.
