Processing Time Estimate of Large Dataset

I have around 13.2k photos covering around 10 sq km. I am using i3.16xlarge AWS EC2 instance (488 GiB of memory, 64 vCPUs). I have allocated 1 tb as hard disk.

It’s has been around 24 hours since the process has started ( + 2 more hours for Resizing). I have been monitoring the CPU usage and it is still pretty low.

cpu_utilization

Please find the attached logs too.

  1. Can anyone estimate how much more time it would take to process the data?
  2. why it is taking so long to reconstruct even though I feel RAM and CPUs are sufficient enough?
  3. CPU utilization seems low. Are all the cores are being used or some kind of single threaded process is going?
1 Like

Welcome!

Yes, some parts of the processing pipeline are single-threaded. Not every workflow can be successfully multithreaded.

It is close to 41 hours and still re-construction process is going on. I wonder what could be the best way to faster this re-construction process?

Otherwise, there would not be much significance to deploy such huge machine if there is no strategy/way but to wait.

1 Like

You could try using --auto-boundary to help reconstrain reconstruction, maybe try --feature-type ORB (at least 5x as fast as SIFT), and possibly investigate the split/merge pipeline to run multiple sub-reconstructions in parallel, which can help keep the single-threaded parts of the reconstruction moving in parallel most the time (though the final compositing is going to be in one task).

Thank you for the suggestion. What should I choose as --merge and --split parameter?
I have around 13.2k files and it is over 9.5 sq km area

1 Like

Hmm, maybe try splitting it into 4 queues, so --split 3300 and --split-overlap 150 (default) should do to start.

I’m not sure if this is too many parallel queues and might slow things down since you’ll only have 16 cores working per queue now…

How big is your dataset? I’m setting up a cluster with multiple 256gb server nodes & would be happy to test processing on a large dataset like this.
darren at nextechit dot com dot au & I’ll send you the link to upload.

1 Like