Hi,
I realise there are lots of topics on large datasets already, but I haven’t found anything recent covering a similar problem.
I have a NodeODM instance running on Kubernetes in a cloud environment. The machine is pretty powerful (62 CPUs and ~800 GB of memory), but no “swap”. (As far as I’m aware, Kubernetes does not have any concept of “swap”, but please correct me if this is wrong and you’ve managed to configure “swap” for ODM on Kubernetes).
I am using pyodm to submit jobs to NodeODM. I’m dealing with data from aerial drones, both RGB and Multispectral. It’s working nicely so far, and I’ve been very impressed by the combination of pyodm + NodeODM. Thanks to everyone who has worked on this 
However, I’m currently struggling with appropriate settings for large missions. For example, I currently have a dataset with 6200 RGB images, each of which is 24 megapixels and typically about 10 MB in size. My goal is to generate the best quality orthophoto possible; the DSM and 3D model are also interesting, but less of a priority right now).
My first attempt used the following options:
default_options = {
"dsm": True,
"dtm": True,
"cog": True,
"orthophoto-compression": "LZW",
"orthophoto-resolution": 0.1, # cm/pixel. If set very small, output will be auto-limited by data to max sensible value (I think?)
"dem-resolution": 0.1, # cm/pixel. If set very small, output will be auto-limited by data to max sensible value (I think?)
"max-concurrency": 60,
"auto-boundary": True,
"use-3dmesh": True,
"fast-orthophoto": False,
"feature-quality": "high", # ultra | high | medium | low | lowest
"pc-quality": "high", # ultra | high | medium | low | lowest
}
Over a period of about 24 hours, this progressed fairly steadily to about 73% complete, but then it stopped for 5 days with no further progress running a pdal command. It didn’t actually fail or produce an error - just seemed to grind to a halt. However, I’m hoping to find a way to process these missions faster than this, so I cancelled the task and tried some different options.
My second attempt used the same options, but with "split": 2000 added. This seems to be running OK (I can see that the submodels have been created correctly) and over a period of about 2 days it has reached 84% complete. However, it has now been stuck for another two days at the same point, with the last message saying [INFO] Adjusted bounds. Looking more closely at the NodeODM working directory, it looks as though the orthophoto actually completed processing about 2 days ago (there is a 20 GB geotiff named odm_orthophtoto.original.tif). Since then, it seems NodeODM has spent another 2 days slowly writing data to dsm.tif: according to the OS, this file is constantly being modified, but yesterday it was about 5.5 GB and 24 hours later it’s only 7.7 GB, so it’s being written quite slowly.
Does anyone have any suggestions for optimisation, please?
I am reluctant to use fast-orthophoto: True, because I have read in OpenDroneMap: The missing guide that this will produce a lower quality orthophoto. I am basically looking for a combination of options that will maximise processing speed, but without sacrificing orthophoto quality. For example, I am wondering if the process currently writing to dsm.tif is slow due to the size of the point cloud, in which case maybe the dem-decimation option might help?
From the docs, it sounds as though setting "dsm": False and "dtm": False should have no effect on orthophoto quality, but "use-3dmesh": False will make the orthophoto worse. Does that sound correct?
I will continue to explore various options, but since each run takes 2 to 5 days I’d like to narrow down a sensible number of parameter combinations first. Any tips for settings used when processing large datasets (especially in a Kubernetes environment) would be greatly appreciated.
Thank you!
On the other hand, my options for optimisation may be limited, because turning off “optional” components seems to cause the entire workflow to fail.