When does split-merge/parallel computing make sense?

Hello,

I am trying to find information on when a split-merge is starting to see a significant upside on lowering the computing time of the scheduled job.

Do we have any test done or data on this? I thought I saw a post on this but couldn’t find it!

Best Regards,
Rasmus

1 Like

Oh that’s a great question!

I’m not sure if we have profiled split/merge or collected metrics for benchmarking like we do for standard processing.

Corey will likely have better visibility into this.

1 Like

I’m currently working on a 6894 image (scaled by 0.5X) set. With no split-merge it would take an infinite amount of time, ie it failed after >115 hours.
Currently running it again with split-merge 2300/150, which will hopefully complete later this week.
So working near the limits it could be the difference between success and failure to complete a job.

1 Like

Okay interesting, I am looking to make algorithm for this. For example:

if images > 500:
      *Do splitmerge with ratio x*

Does anyone know how lightning ODM does it?

Best Regards,
Rasmus

1 Like

I think it would be wise to split depending on how large the images are and how mush ram you got.

It’s seems that way.

So if an image is 0.02gig and ram is 32gig, 32gig/0.02gig = 1600, then maybe that in some way can give you the right split value. I don’t know how much memory is needed for an image in WebODM.

3 Likes

Lightning does not support the split/merge pipeline at the moment.

It does however, have a lot of RAM and a lot of swap.

2 Likes

Tack för svaret Andreas!

Yeah indeed, I will spin up machines for optimal performance so ram can be adapted to the workload.

So I would like to know the most performant/time-efficient way of processing a workload.

3 Likes

I’m searching for the same formula, it turns out I need a better computer.

2 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.