Testing distributed cluster processing on Intel NUCs

Testing cluster processing of a 3,927 image dataset on different numbers of Intel NUCs (installed via POSM). Parameters adjusted: min-num-features: 9000, split: 500, split-overlap: 120

03%20PM
The log output has a timestamp when it starts extracting EXIF (after uploading and resizing) and after the merge stage (before postprocessing generation of base and overview tiles). This seems to be shown in the WebODM dashboard listing a longer duration than what is calculated with the timestamps. However, for the 4x POSM dataset the difference is over 10 hours. Additionally the WebODM dashboard duration for the 4x is longer than for the 2x.

Logs here:
1 x NUC8i7BEH1
2 x NUC8i7BEH1
4 x NUC6i7KYK
6 x NUC6i7KYK


  • What might be responsible for the outlier 10 hour difference and out of order processing time?
  • I will probably rerun the 1x and 2x tests on NUC6i7KYK to remove the variation due to NUC model. Although it’s my understanding that the two should be comparable when it comes to running ODM? The specs for the the NUC8i7BEH1 and the NUC6i7KYK. All have 32GB RAM installed.
  • Separately, would it be possible to make the products ready before postprocessing (i.e. the TIFF) available for download as soon as they are ready?

Cool benchmark! Really nice to see split-merge being timed.

  • The 4x run seems like an outlier and I would trust the log export time more than the WebODM dashboard time; the log export counts the time for processing (without including post processing), but I can’t imagine why post processing would take an astonishing 10 hours, unless this is a bug of some sort or the reconstruction came out larger than expected. Was there anything strange in the output (for example, an abnormally large DSM or orthophoto file size)? Or did the NUC hosting WebODM go asleep at any time during processing?
  • CPU impacts runtime, so there might be differences between the two models in terms of CPU speed.
  • In theory, yes, but this would complicate the processing logic quite a bit (and wouldn’t be too straightforward to implement either). Happy to merge a PR if anyone wants to implement it.

11%20PM

  • Looks like definitely better performance on the NUC8i7BEH1.
  • Reran the 4 box test and oddly enough it still had the 10 hour difference. The download assets dropdown for the task is missing the “Point Cloud (LAZ)” option. The other test runs have all options (GeoTIFF, MBTiles, Tiles, LAZ, All Assets). Checking the [INFO] logged at the start of 4x and the 6x test, the parameters all match. The 6x test includes all the hardware that ran the 4x test so if there was a problem with one of the boxes, I would think the problem would appear in both the 4x and 6x tests.