I’m having some strange issues with the ODM split-merge workflow. I am running ODM with Docker on a machine with 32 CPUs and 48 GBs of RAM. My dataset is relatively small (850 images), but they are taken from a Phantom 4 drone flying at 10 m and have a very low GSD. I am trying to build a very high resolution orthophoto, and I’ve found that splitting into three clusters is a good way to do this without running out of RAM.
However, the behavior I’m seeing is that the split step works just fine, but then it hangs during the orthophoto merge step. Looking in
top, I just see the Python process sitting at ~100% CPU usage. The RAM usage increases for the first 10 minutes or so, and then levels off somewhere around 8 GB. After that, it does nothing. There is no output in the terminal. It does create an output merged orthophoto file on the disk, but this grows excruciatingly slowly (a few hundred KBs every 10 minutes or so).
I haven’t been able to find a lot of similar issues, other than issue #988 on Github (sorry, it won’t let me add a link), which appears to have been solved a long time ago. I think I’ve traced where it’s getting stuck for me to this section in the code, based on the behavior and the output that I’m seeing. I’m going to try running this manually on my dev machine to see if I can a) reproduce the issue, and b) come up with a firmer diagnosis.
This is the command I’m running:
singularity run docker://opendronemap/odm \ --project-path /lscratch/odm_jeevan_field \ --max-concurrency 32 \ --orthophoto-resolution 0.27 \ --fast-orthophoto \ --skip-3dmodel \ --matcher-distance 30 \ --split 400 \ --split-overlap 10 \ --rerun-all \ --texturing-skip-local-seam-leveling \ --merge orthophoto \ --orthophoto-compression JPEG \ 2021_08_09_p4_ortho
Unfortunately, I cannot share all of my images for confidentiality reasons, but I have provided a few examples. I’ve also included the output from ODM in that same folder.
Has anyone else had this problem before? I’d appreciate any insights you can provide.