How to load balance queue?

I want processing to happen in every node but currently I see

# | Node | Status | Queue | Engine | API | Flags
-- | -- | -- | -- | -- | -- | --

1 | 192.168.0.247:3000 | Online | 1/4 | odm 2.4.3 | 2.1.4 |  
2 | 192.168.0.248:3000 | Online | 0/4 | odm 2.4.3 | 2.1.4 |  
3 | 192.168.0.246:3000 | Online | 0/4 | odm 2.4.3 | 2.1.4 |  

One task takes up only 1 queue. But I would like it to queue on all node so processing will be very fast? Is there any scheduling provision that we can use to schedule it across all the queues on multiple nodes?

I do see split_merge options but I don’t see any documentation etc. ClusterODM/config.js at master · OpenDroneMap/ClusterODM · GitHub

Thank You.

1 Like

Welcome!

I’m not very familiar with the split/merge process, but have you checked out the Documentation here?
https://docs.opendronemap.org/large.html#distributed-split-merge

1 Like

Thanks.

After reading docs I think I am not looking for split merge as its not working and still runs on 1 queue?

Just to be clear what I want to do is:
Run 1 task and that task should run across multiple node (or multiple ips)

What I have done till now is?

  • Made cluster odm to work
  • Added 3 nodes to cluster odm
  • Connected web odm to Cluster odm
  • Added Split and split overflow on settings from edit option on task
  • Ran it

But it still only work on 1 node. Other 2 nodes are idle. I am abit confused if I am going on correct direction.

I am sorry as I am complete new to ODM.

Thanks.

ClusterODM doesn’t have a round-robin mode, which seems counter-intuitive when you own the hardware, but makes lots of sense if you are renting the hardware.

The logic for the sorting of nodes is here:

I’ve added an issue to track round-robin as a feature request here:

Contributions welcome, but it’s also useful to highlight other use cases. My workaround has been to give all nodes the same weight and to only let them take one job each, but a round robin or weighted round robin would be a super useful addition.

2 Likes

Nice to chat on the call today! Just as an FYI, you can tune your nodes a bit with the following flags:

To update other readers on the resolution, @shirshak55 wasn’t seeing multiple nodes used in a cluster. The nodes only activated once --split was set to a sufficiently low value and --split-overlap ensured that more than one submodel was being created. Changing split-overlap is generally only needed when refining how submodels blend together. In this case, it needed to be set low to ensure that a smaller test dataset was split, and wouldn’t have been needed with a much larger dataset.

3 Likes

So it’s useless to just use the --sm-cluster option, right?

I’ve had success adding the cluster as a processing node in WebODM AND populating the --sm-cluster option for the same task.

Haven’t looked around or tested a mix and match approach since it’s simple enough just to add it.

So it’s useless to just use the --sm-cluster option, right?