How to load balance queue?

shirshak55 · 23 February 2021 17:08

I want processing to happen in every node but currently I see

# | Node | Status | Queue | Engine | API | Flags
-- | -- | -- | -- | -- | -- | --

1 | 192.168.0.247:3000 | Online | 1/4 | odm 2.4.3 | 2.1.4 |  
2 | 192.168.0.248:3000 | Online | 0/4 | odm 2.4.3 | 2.1.4 |  
3 | 192.168.0.246:3000 | Online | 0/4 | odm 2.4.3 | 2.1.4 |

One task takes up only 1 queue. But I would like it to queue on all node so processing will be very fast? Is there any scheduling provision that we can use to schedule it across all the queues on multiple nodes?

I do see split_merge options but I don’t see any documentation etc. ClusterODM/config.js at master · OpenDroneMap/ClusterODM · GitHub

Thank You.

Saijin_Naib · 23 February 2021 17:11

Welcome!

I’m not very familiar with the split/merge process, but have you checked out the Documentation here?
https://docs.opendronemap.org/large.html#distributed-split-merge

shirshak55 · 23 February 2021 18:35

Thanks.

After reading docs I think I am not looking for split merge as its not working and still runs on 1 queue?

Just to be clear what I want to do is:
Run 1 task and that task should run across multiple node (or multiple ips)

What I have done till now is?

Made cluster odm to work
Added 3 nodes to cluster odm
Connected web odm to Cluster odm
Added Split and split overflow on settings from edit option on task
Ran it

But it still only work on 1 node. Other 2 nodes are idle. I am abit confused if I am going on correct direction.

I am sorry as I am complete new to ODM.

Thanks.

smathermather-cm · 24 February 2021 03:06

ClusterODM doesn’t have a round-robin mode, which seems counter-intuitive when you own the hardware, but makes lots of sense if you are renting the hardware.

The logic for the sorting of nodes is here:

github.com

OpenDroneMap/ClusterODM/blob/master/libs/nodes.js#L163


        maxImages: n.getInfo().maxImages ? n.getInfo().maxImages : 999999999,
        slots: Math.max(0, n.getInfo().maxParallelTasks - n.getInfo().taskQueueCount),
        queueCount: n.getInfo().taskQueueCount
    };
});
// Sort by node with smallest maxImages value
// tie break by most available slots
// and further by least queue count
// and further by turn number
sorted.sort((a, b) => {
    if (a.maxImages < b.maxImages) return -1;
    else if (a.maxImages > b.maxImages) return 1;
    else if (a.slots > b.slots) return -1;
    else if (a.slots < b.slots) return 1;
    else if (a.queueCount < b.queueCount) return -1;
    else if (a.queueCount > b.queueCount) return 1;
    else if (a.node.turn < b.node.turn) return -1;
    else if (a.node.turn > b.node.turn) return 1;
    else return 1;
});

I’ve added an issue to track round-robin as a feature request here:

Contributions welcome, but it’s also useful to highlight other use cases. My workaround has been to give all nodes the same weight and to only let them take one job each, but a round robin or weighted round robin would be a super useful addition.

smathermather-cm · 25 February 2021 19:22

Nice to chat on the call today! Just as an FYI, you can tune your nodes a bit with the following flags:

github.com

OpenDroneMap/NodeODM/blob/master/config.js#L29


let fs = require('fs');
let argv = require('minimist')(process.argv.slice(2));
let utils = require('./libs/utils');
const spawnSync = require('child_process').spawnSync;
if (argv.help){
	console.log(`
Usage: node index.js [options]
Options:
	--config <path>	Path to the configuration file (default: config-default.json)	
	-p, --port <number> 	Port to bind the server to (default: 3000)
	--odm_path <path>	Path to OpenDroneMap's code	(default: /code)
	--log_level <logLevel>	Set log level verbosity (default: info)
	-d, --daemon 	Set process to run as a deamon
	-q, --parallel_queue_processing <number> Number of simultaneous processing tasks (default: 2)
	--cleanup_tasks_after <number> Number of minutes that elapse before deleting finished and canceled tasks (default: 2880) 
	--cleanup_uploads_after <number> Number of minutes that elapse before deleting unfinished uploads. Set this value to the maximum time you expect a dataset to be uploaded. (default: 2880) 
	--test Enable test mode. In test mode, no commands are sent to OpenDroneMap. This can be useful during development or testing (default: false)
	--test_skip_orthophotos	If test mode is enabled, skip orthophoto results when generating assets. (default: false)

To update other readers on the resolution, @shirshak55 wasn’t seeing multiple nodes used in a cluster. The nodes only activated once --split was set to a sufficiently low value and --split-overlap ensured that more than one submodel was being created. Changing split-overlap is generally only needed when refining how submodels blend together. In this case, it needed to be set low to ensure that a smaller test dataset was split, and wouldn’t have been needed with a much larger dataset.

JOJOforshaun · 27 March 2021 05:40

So it’s useless to just use the --sm-cluster option, right?

and-viceversa · 26 March 2021 21:12

I’ve had success adding the cluster as a processing node in WebODM AND populating the --sm-cluster option for the same task.

Haven’t looked around or tested a mix and match approach since it’s simple enough just to add it.

JOJOforshaun · 27 March 2021 05:40

So it’s useless to just use the --sm-cluster option, right?